Exadata for Dummies

™

g Easier! Making Everythin cial Edition DLT Solutions Spe Consolidate into high performance and high availability architecture with Exadata Managing a mass of disjointed database servers, storage devices, and networks is no way to run a database shop. Consolidating your database and storage into a single database machine simplifies management and drives down costs. Exadata provides Capacity on Demand computing through high performance and reliable Oracle RAC databases and optimized storage server architecture. Stop struggling to meet requirements and start excelling with Exadata! • Understand Exadata architecture — find out how Oracle RAC database servers integrate with storage servers to provide speed and reliability

Open the book and find: • What exactly Exadata is and how it works • How Exadata is configured and tuned for performance and high availability • What flash, compression, and processing optimizations reduce storage and increase performance • Why Exadata systems make your IT environment faster and more manageable

• Scale systems to meet your needs — add Quarter, Half, and Full Racks of database and storage capacity to support any database processing environment • Implement a shared services solution — discover how Exadata allows you to consolidate existing database environments into a costeffective, Capacity on Demand environment

a t a d Exa

®

Learn to: • Understand Exadata Database Machine architecture

Go to Dummies.com® for videos, step-by-step examples, how-to articles, or to shop!

• Maximize database and storage server capabilities • Consolidate complex systems onto a single database machine Brought to you by

ISBN 978-1-118-14780-1 Book not for resale

Michael Wessler, OCP & CISSP

About Oracle Oracle provides the world’s most complete, open, and integrated business software and hardware systems, with more than 370,000 customers — including 100 of the Fortune 100 — representing a variety of sizes and industries in more than 145 countries around the globe. Oracle’s product strategy provides flexibility and choice to our customers across their IT infrastructure. Now, with Sun server, storage, operating-system, and virtualization technology, Oracle is the only vendor able to offer a complete technology stack in which every layer is integrated to work together as a single system. In addition, Oracle’s open architecture and multiple operating-system options give our customers unmatched benefits from industry-leading products, including excellent system availability, scalability, energy efficiency, powerful performance, and low total cost of ownership.

About DLT Solutions Through the partnership with Oracle, DLT Solutions provides all the key components — database, middleware, and applications — necessary to transform the operations of government organizations. With Oracle and DLT, government organizations can consolidate information, streamline operations, and provide excellent constituent services. As one of Oracle’s largest government resellers, DLT offers Oracle’s entire information technology portfolio. For twenty years, DLT Solutions has been the IT software and solutions provider of choice for federal, state, and local government and education customers. By hand selecting its manufacturer partners, including Autodesk, Google, NetApp, Oracle, Quest Software, Red Hat, Solarwinds, and Symantec, DLT fulfills its mission to be a value-added reseller of only the best software and hardware products and services, as well as a premier provider of technical support, through its many government contracts. The company specializes in carefully selected solution areas — Cloud Computing, Data Center Consolidation, Geospatial Data Systems, and Computer-Aided Design for Transportation, Utilities, and Manufacturing. For more information, contact DLT Solutions at 800-262-4358, e-mail [email protected], or visit www.dlt.com.

Runs Oracle 10x Faster *

The World’s Fastest Database Machine • Hardware by Sun • Software by Oracle

* But you have to be willing to spend 50% less on hardware

10x faster based on comparing Oracle data warehouses on customer systems vs. Oracle Exadata Database Machines. Potential savings based on total hardware costs. Oracle Database and options licenses not included. Actual results and savings may vary.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved . Oracle and Java registered trademarks of Oracle and/or its affiliates

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Exadata FOR

DUMmIES

‰

DLT SOLUTIONS SPECIAL EDITION

by Michael Wessler, OCP & CISSP

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Exadata For Dummies®, DLT Solutions Special Edition Published by John Wiley & Sons, Inc. 111 River Street Hoboken, NJ 07030-5774 www.wiley.com Copyright © 2011 by John Wiley & Sons, Inc., Hoboken, New Jersey Published by John Wiley & Sons, Inc., Hoboken, New Jersey No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without the prior written permission of the Publisher. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions. Trademarks: Wiley, the Wiley logo, For Dummies, the Dummies Man logo, A Reference for the Rest of Us!, The Dummies Way, Dummies.com, Making Everything Easier, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates in the United States and other countries, and may not be used without written permission. Oracle is a registered trademark of Oracle International Corporation, and DLT Solutions is a trademark of DLT Solutions, LLC. Under the laws of the United States these trademarks may only be used with express written permission from Oracle and DLT Solutions, LLC respectively. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc., is not associated with any product or vendor mentioned in this book. LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: THE PUBLISHER AND THE AUTHOR MAKE NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS WORK AND SPECIFICALLY DISCLAIM ALL WARRANTIES, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE. NO WARRANTY MAY BE CREATED OR EXTENDED BY SALES OR PROMOTIONAL MATERIALS. THE ADVICE AND STRATEGIES CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY SITUATION. THIS WORK IS SOLD WITH THE UNDERSTANDING THAT THE PUBLISHER IS NOT ENGAGED IN RENDERING LEGAL, ACCOUNTING, OR OTHER PROFESSIONAL SERVICES. IF PROFESSIONAL ASSISTANCE IS REQUIRED, THE SERVICES OF A COMPETENT PROFESSIONAL PERSON SHOULD BE SOUGHT. NEITHER THE PUBLISHER NOR THE AUTHOR SHALL BE LIABLE FOR DAMAGES ARISING HEREFROM. THE FACT THAT AN ORGANIZATION OR WEBSITE IS REFERRED TO IN THIS WORK AS A CITATION AND/OR A POTENTIAL SOURCE OF FURTHER INFORMATION DOES NOT MEAN THAT THE AUTHOR OR THE PUBLISHER ENDORSES THE INFORMATION THE ORGANIZATION OR WEBSITE MAY PROVIDE OR RECOMMENDATIONS IT MAY MAKE. FURTHER, READERS SHOULD BE AWARE THAT INTERNET WEBSITES LISTED IN THIS WORK MAY HAVE CHANGED OR DISAPPEARED BETWEEN WHEN THIS WORK WAS WRITTEN AND WHEN IT IS READ. For general information on our other products and services, please contact our Business Development Department in the U.S. at 317-572-3205. For details on how to create a custom For Dummies book for your business or organization, contact [email protected]. For information about licensing the For Dummies brand for products or services, contact [email protected]. ISBN: 978-1-118-14780-1 Manufactured in the United States of America 10 9 8 7 6 5 4 3 2 1

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Table of Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 About This Book......................................................................... 1 Icons Used in This Book............................................................. 2 Where to Go from Here.............................................................. 2

Chapter 1: Today’s Database Challenges. . . . . . . . . . . . . 3 Current Database Architecture................................................. 3 IT System Challenges.................................................................. 8 Need a Better Solution?............................................................ 14

Chapter 2: Exadata’s Unique Architecture. . . . . . . . . . . 15 What Is Exadata?....................................................................... 15 The Exadata Database Machines............................................ 20 Exadata Storage Server............................................................ 25

Chapter 3: Exadata’s OLTP and Data Warehousing Capabilities . . . . . . . . . . . . . . . . . . . . . . 29 OLTP Processing Requirements............................................. 29 Data Warehouse Processing.................................................... 30 Enterprise Manager Grid Control........................................... 31 Exadata Smart Flash Cache..................................................... 32 Exadata Smart Scan.................................................................. 34 Exadata Hybrid Columnar Compression............................... 34 High Availability, Backup, and Recovery............................... 35

Chapter 4: Benefiting from Shared Services and Server Consolidation . . . . . . . . . . . . . . . . . . . . . . . 37 Why Server Consolidation?..................................................... 37 How Exadata Supports Server Consolidation....................... 38 Exadata I/O Resource Manager and Quality of Service Management........................................... 40

Chapter 5: Ten (Okay, Five) Things to Look For in an Exadata Solution. . . . . . . . . . . . . . . . . . . . . . . . . . 43 Unified Hardware and Software.............................................. 43 Preconfigured Components..................................................... 43 Scaling for Large Applications................................................ 44 Reducing Infrastructure Costs through Consolidation........ 44 Agile Infrastructure to Meet Changing Requirements.......... 44 These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Publisher’s Acknowledgments We’re proud of this book and of the people who worked on it. For details on how to create a custom For Dummies book for your business or organization, contact info@ dummies.biz. For details on licensing the For Dummies brand for products or services, contact [email protected]. Some of the people who helped bring this book to market include the following: Acquisitions, Editorial, and Vertical Websites Senior Project Editor: Zoë Wykes Editorial Manager: Rev Mengle Business Development Representative: Sue Blessing Custom Publishing Project Specialist: Michael Sullivan

Composition Services Senior Project Coordinator: Kristie Rees Layout and Graphics: Claudia Bell, Carrie A. Cesavice Proofreader: Dwight Ramsey

Publishing and Editorial for Technology Dummies Richard Swadley, Vice President and Executive Group Publisher Andy Cummings, Vice President and Publisher Mary Bednarek, Executive Director, Acquisitions Mary C. Corder, Editorial Director Publishing and Editorial for Consumer Dummies Kathleen Nebenhaus, Vice President and Executive Publisher Composition Services Debbie Stailey, Director of Composition Services Business Development Lisa Coleman, Director, New Market and Brand Development

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Introduction

D

atabase applications are the heart of any IT system, and it is critical that they run at an optimum level. Unfortunately, we often see IT shops littered with database servers, storage networks, and network infrastructure — all running on a hodgepodge of hardware, software, and configuration settings. These systems struggle to provide the performance and high availability necessary.

Oracle Exadata takes that mash of database servers, storage, and network infrastructure and places it in a single database machine. By running Oracle RAC databases in a tuned and optimized high availability configuration, your database performance will soar. Integrated Oracle storage servers using high-performance technologies support your largest databases while reducing storage requirements. You can achieve a consolidated, high-performance database and storage server infrastructure without breaking your IT budget.

About This Book This book consists of five short chapters, each written as a stand-alone chapter, so feel free to start reading anywhere and skip around throughout the book! Chapter 1: Today’s Database Challenges. We look at how database systems are currently architected. Then we look at the technologies and techniques currently in use and why they aren’t sufficient for the future. Chapter 2: Exadata’s Unique Architecture. Here, we get into the nuts and bolts of Exadata’s architecture, how it works, and why Exadata is a better solution. Chapter 3: Exadata’s OLTP and Data Warehousing Compatibilities. This chapter looks at Exadata’s extensive features and optimizations for OLTP and data warehousing systems. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

2

Exadata For Dummies, DLT Solutions Special Edition Chapter 4: Benefiting from Shared Services and Server Consolidation. We delve into how Exadata supports shared services and drives down costs via server consolidation. Chapter 5: Ten (Okay, Five) Things to Look For in an Exadata Solution. Here, in that famous For Dummies style, we give you the “Part of Tens” detailing benefits you will experience by implementing Oracle Exadata in your organization.

Icons Used in This Book Throughout this book, we occasionally use icons to call attention to important information that is particularly worth noting. Here’s what to expect.



If you see an icon that says remember, you may want to, uh, remember the information. We won’t have deep meaning-oflife stuff, but it may be good to know for later.

Sometimes you just have to know a term or technical details to understand a larger topic. Or, it could just be that we want to throw in cool terms to try to impress people. Either way, this icon identifies techie stuff.



This icon usually denotes something the author wishes someone had told him before he learned it the hard way! Keep these items in mind to make life easier.

Where to Go from Here Hey, it’s your book so feel free to jump around wherever you’d like, or simply turn the page and just start reading.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 1

Today’s Database Challenges In This Chapter ▶ Modeling current database architectures ▶ Identifying database challenges ▶ Finding a better solution

T

oday’s business challenges dictate fast, agile, and reliable IT solutions, but those solutions must be secure and within the budget. At the core of any IT solution is the data, and that data is stored within the database. It stands to reason, therefore, that the database is the most important part of any IT solution. Database technology has evolved over several generations of architectures due to both technological advancements (hardware and software) and business drivers. Today, it is often the needs of business that determine the direction of database technology and architecture to meet those needs. In this chapter, we examine today’s database architectures and the challenges they face.

Current Database Architecture To understand the role of a database and what the key factors for a successful database are, you need to understand the role of the database in an IT system. Essentially, a computer application provides a user with the ability to perform some task

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

4

Exadata For Dummies, DLT Solutions Special Edition (such as ordering a product or service). All the data used to support that task is stored in the database. A modern IT system is generally composed of the following components:

✓ User. Very often a human sitting in front of a keyboard interface using a computer system. That user may be a customer or an employee. The interface may be a client side program or it may be a web browser, but that isn’t always the case with the growth of smartphone technology and similar devices. Increasingly, however, the “user” is another computer system interfacing with the application. It is not uncommon to have automated system-to-system interfaces such as web services. ✓ Application. The program or “application” that executes logic to present the data to the user and implements the business logic to manipulate that data. The application can execute on the client’s workstation or device, on a dedicated application or web application server, or on the database server. Often the application is distributed between components with a visual component at the user interface level, business processing on the application server, and heavy data-manipulation functions at the database server level. ✓ Database. The software component that stores and manages the system’s data. Data is organized in logical containers called tables. Each table is mapped to a physical storage device, which is often a disk. User accounts are created with security and business rule logic implemented to dictate what data a user can see and how that data is created, updated, and deleted. Database processing is often very intensive because enormous amounts of data are processed as part of each application request. Because of this, database software itself is often a specialized component that exists on one or more dedicated database servers. ✓ Storage. Data physically stored on a storage device and accessed by the database. The storage device itself is often disk, but it can also be a solid state flash drive for fast access or tape storage for archival access.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 1: Today’s Database Challenges

“In the Cloud” A recent architecture gaining visibility is cloud computing. Cloud computing is an approach in which instead of the client accessing a specific set of servers to perform a specific application function, that function is accessed as a service on the Internet or intranet. The “cloud” is the network to which the user sends an application request and from which the user receives a response.

Cloud computing is an advanced architecture that promises to lower costs and improve service because the same application is shared by everyone in the “cloud”; software becomes a shared service. Oracle is highly interested in cloud computing, and in later chapters we discuss how Oracle’s products support this architecture.

Storage devices are frequently attached to the database server over a network rather than being an internal disk within a server. This is because storage requirements exceed what a single server can hold internally, or the storage is shared to support multiple database servers. Storage devices themselves are often “smart devices” that have their own memory components, management tools, and processors to assist in performance and reliability. ✓ Network. Communication between the user and application server is the first network component. Within the system itself, there are many network connections between components. Between the application server and the database server is another connection. From the database server to the disk storage array is another network connection. Other connections between components and subsystems are common. The technical details for a network connection depend on which components are communicating. For example, the connection between a client’s web browser to the web application server is different than between a database server to its disk storage. However, common network requirements exist regardless of the network type; the network needs to be fast, secure, and reliable.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

5

6

Exadata For Dummies, DLT Solutions Special Edition Database systems fall into three general categories: OLTP, data warehousing, and hybrids.

On Line Transaction Processing On Line Transaction Processing (OLTP) systems are used to process transactions and perform work in a real-time, online manner. These are the most common database systems and are what often do the “work” for a business. OLTP means thousands of users simultaneously executing small, quick transactions. A prime example of an OLTP system is a call center application where customers call or order online a company’s product or service. This system may entail order entry, billing, customer data, shipping, and inventory components. Characteristics important to an OLTP system include ✓ Availability. If the system isn’t available, the company won’t generate revenue and will lose customers. System uptime is very important to OLTP systems; every minute of downtime has a dollar value of lost revenue associated with it. Architectures that allow the OLTP system to remain available even during computer failures are highly desirable. ✓ Performance. If customers have to wait excessively for processing to occur or if their process fails due to a timeout error, they’ll go to a competitor. At a minimum, fewer customer orders will be processed in a given time period. Poor performance leads to frustrated customers and as a result, high throughput performance and low latency is critical for OLTP systems. ✓ Agility. New business opportunities or regulatory requirements must be met in a rapid manner. Failure to anticipate and react to changing needs can be disastrous. A computer system must be able to be modified to meet changing requirements in an agile manner. ✓ Security. OLTP systems are prime targets for thieves and competitors due to the nature of the data they contain. Security breaches are costly beyond just what is stolen; loss of customers, damaged public image, fines, and legal action are likely. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 1: Today’s Database Challenges

7

OLTP systems are the focal point for IT organizations because they are critical to the success of the business. The system availability, performance, agility, and security are key characteristics to be considered when designing and evaluating OLTP database systems.

Data warehousing While OLTP systems deal in real-time, active data, data warehouses store archival data to be used in data mining, trend analysis, and meeting audit and regulatory requirements. As data in OLTP systems grows older, it is used less often. That older data is often moved into a data warehouse where specialized processing that requires a history of data occurs. Data warehouses are commonly used to ask business-related questions, the answers to which require the ability to collect years of data. A business analyst may ask, “How many cases of product X are sold on Super Bowl Sunday and can you determine the growth rate over the last 10 years?” — both of which are the types of questions reserved for data warehouses. Characteristics important to a data warehouse include ✓ Storage. The size of data warehouses is huge! Terabytes of data are common, and specialized processing and storage techniques to optimize performance are required. Rather than using high performance disk, data warehouses use capacity disk or tape storage because it would be cost prohibitive to store all data on high performance storage devices. ✓ Processing capability. OLTP systems support thousands of users running small quick transactions, but data warehouses support a handful of users executing very long running queries. It is possible for a data warehouse query to run many hours or even days to get an answer. Business Intelligence (BI) tools allow management and data analysts to perform “what-if” scenarios against historical data. This allows business trends to be identified and complex relationships between data to be seen that otherwise may go unnoticed. This technology has occurred in situations ranging from increasing company profits to catching criminals and scientific research. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

8

Exadata For Dummies, DLT Solutions Special Edition Data warehouses are fewer than OLTP systems, but by their very nature, they are much longer. Their design characteristics are different than OLTP systems as well, because they process long running queries rather than multiple fast queries.

Hybrid systems The hybrid is a combination of OLTP and data warehousing systems. Often, hybrid systems start as OLTP systems, which over time introduce data warehousing requirements. The ideal design characteristics of OLTP versus data warehouses are often at odds for each other. For example, an OLTP system is optimized for fast data access by multiple users whereas data warehouses are optimized for fewer users with long running queries. Different database design considerations and parameters — which are often mutually exclusive — are used for both systems. One compromise is to run a hybrid database in OLTP mode during the day and data warehousing mode at night. However, as systems are increasingly used at all hours, that compromise becomes less effective. Be sure to evaluate your capability to support both OLTP and warehousing requirements.

IT System Challenges Regardless of industry, IT organizations face similar challenges. Supporting complex systems with increasing requirements is at odds with never having enough time or resources. The following sections discuss common issues faced by IT management.

High availability Few systems are only open during business hours; most need to be available 24x7. Given that any component itself will eventually fail, a system needs to be engineered to support fault tolerance and eliminate single points of failure.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 1: Today’s Database Challenges

9

Fault tolerance is the ability of a system to suffer a component failure, yet still continue processing and remain available for the users. Any individual component will eventually fail, but the overall system must continue to function. Designing a system to have redundant servers so when one fails the other servers will continue processing is an example of fault tolerant design. Having only a single component (which will eventually fail) to perform a function is referred to as a single point of failure, which is something to avoid. If your system has a single point of failure, it’s simply a matter of when, not if, it will fail. Figure 1-1 shows how multiple database servers provide both increased processing capabilities and fault tolerance. If one server fails, processing continues on the remaining server. You also see how the single web application server is a single point of failure, making the web application server the most likely weak link in this system. Be sure to identify the fault tolerant components and single points of failure in any system or product you are evaluating.

Security Security is not a just a product you buy once or something that is added at the end of program development. Security needs to be an attitude and mindset that occurs at all levels of IT and management from the beginning of a project until a project is finally decommissioned. Products and techniques used to implement security are merely means to an end. Unfortunately, security is often poorly implemented and managed and is deemed a burden to IT systems. Managing configuration settings, access controls, and monitoring audit logs does require resources. Testing and applying security patches also require staff time and is usually after hours. Keeping a handle on security issues and coordinating with management, customers, and technical folks is a never-ending process, but it is a critical process.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Disk

Disk

Disk

Disk

Customer Database

Disk Storage Array

Disk Disk

Web Browser

Web Browser

Web Browser

Network Cloud

Web Application Server

Database Server

Disk Disk

Disk Storage Array

Customer Database

Storage Area Network

Exadata For Dummies, DLT Solutions Special Edition

Database Server

10

Figure 1-1: Fault tolerance and single point of failure.

Increasing storage requirements An undeniable direction in IT is the requirement to store more and more data, often growing at an accelerated rate. Business requirements force the storage of more data to support customers or develop data warehouses. Changing technology or social media storing images, maps, and audio/video also

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 1: Today’s Database Challenges

11

increase storage requirements. Finally, government or legal data retention requirements force data to be stored for years. Many choices regarding storage are available: ✓ Internal or attached storage ✓ Storage Area Network (SAN) or Network Attached Storage (NAS) ✓ Media options such as flash, performance disk, capacity disk, and tape ✓ Unified or tiered storage ✓ Redundant Array of Inexpensive Disks (RAID) levels and replication Despite improvements in storage technology, effectively managing storage is often a full-time job for one or more trained professionals.

Multiple, complex subsystems A real-world IT system is a complex environment composed of multiple subsystems or components. User management, application servers, web servers, database servers, networking, and storage are just some common components with specialized needs. Within each subsystem, further components exist. For example, the application itself may have credit card billing, inventory control, and auditing functions. Each of these functions has dependencies and subcomponents with further components as you drill down into the details. Managing these varied components is a challenge. In large systems, just being aware of where the vital subsystems exist is often difficult. Interoperability with these subsystems needs to be established and maintained.

Multiple vendors With every hardware device in a system, every software product installed, and every service or interface partner, there is another vendor or group to work with. Over time, the list of These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

12

Exadata For Dummies, DLT Solutions Special Edition vendors to evaluate, purchase support and license agreements with, and work patch and upgrade issues with is daunting. Best of breed is an approach in which you purchase the very best technology product. It allows you to have the industryleading product for a specific hardware or software component, which is sometimes a great benefit. The downside is having more vendors to manage and interoperability between components is often an issue as well. In a worst-case scenario, a customer is caught between two vendors who will not work together to solve a customer’s problem; finger-pointing then becomes a frustrating problem. Single source is an alternative approach in which all products for a system or subsystem are purchased by a single vendor. You may not have the “best” individual product, but you gain not having to manage interoperability issues between components because your vendor has already done that. Issues with patches and upgrades between subcomponents are already identified and resolved before they go to the customer. Finally, although you may not have the best individual components, the components you do have are optimized to work together in a more efficient manner than a series of bestof-breed components operating inefficiently.

Server consolidation Every time a new database server or storage server is added, the cost for hardware, software, licensing, and administrative support increases. While a large server room is visually impressive, it is also excessively expensive! Where possible, IT organizations are consolidating as many components to as few servers as possible in an effort to reduce costs. For example, simply combining development and test applications onto one server is server consolidation. Reducing from several test and development servers into a larger single server is a great way to reduce costs. Technology such as virtualization is a great tool in the server consolidation initiative. Virtualization allows a single server to be partitioned or subdivided into smaller servers that appear as one server to the user. Placing systems into shared environments, but appearing to be on a dedicated, self-contained environment is the goal of virtualization. Adding resources on These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 1: Today’s Database Challenges

13

demand as systems grow with minimal downtime is an additional benefit. As you review hardware and software, be sure to determine their support for virtualization.

Sustaining growth Sooner or later, if a system is growing, the existing server will need more processing capabilities to handle the increased workload. Few things are worse than having a server too small to support your customers. Often, either CPU or memory needs to be increased to support the processing demands of the application. A server can grow up (vertical) or it can grow out (horizontal) to increase processing capability. Vertical scaling is making an existing server “bigger” so that it can support increased processing requirements. Starting with a smaller, expandable server and adding more CPUs, memory, and storage as needed is a common, economical practice. One potential issue with vertical scaling is whether you need downtime to add more resources. An even larger issue is that eventually your server will hit its maximum number of CPUs and memory; vertical scaling does have an upper limit for how large a server can grow. Horizontal scaling is adding more servers to a computing environment to increase processing capability. Adding more servers to support increased workload is a common solution; it improves fault tolerance and is theoretically unlimited in the number you can add. The downside is that you have more servers to manage, which adds to hardware costs, software and licensing costs, and administrative overhead.

Time constraints Because business requirements move fast, so must support structures such as IT. Seemingly, there is never enough time to set up the perfect system. The problem is increased when the IT system itself is complex with many customized components and subsystems from different vendors. Adding in security, documentation, testing, and performance concerns, it is easy to see why IT systems struggle to keep up. IT systems inherently have complex components, but where possible they should be simplified and streamlined. The fewer These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

14

Exadata For Dummies, DLT Solutions Special Edition unnecessary components in a system, the more agile a system will be — and the faster it can react to changing requirements. The key to having an agile system is being able to add or modify components rapidly.

Need a Better Solution? It seems that every system is growing rapidly with changing business processing requirements and increased system availability demands. Unfortunately, IT budgets are not growing to at the same rate as these demands. Complex database applications can be designed to accomplish almost any task given enough time and resources; the problem is that organizations do not have infinite time and resources. A better solution would have the following characteristics: ✓ High availability stressing fault tolerance and void of single points of failure ✓ High-performance, flexible and expandable storage capabilities ✓ Security implemented seamlessly across all components ✓ The ability to increase processing capability both vertically and horizontally ✓ Virtualization capabilities to support consolidated systems and provide resources on demand ✓ Support for OLTP, data warehousing, or hybrid systems ✓ An architecture that can support cloud computing ✓ Simplified configuration with fewer unnecessary components and subsystems ✓ Optimized configuration between components and single source integration, patching, and upgrade issues ✓ Agile design and configuration to allow for rapid stand-up of new systems and reconfiguration of existing systems as requirements change It is the Oracle Exadata database and storage product that promises to be the better solution to the challenges of IT. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 2

Exadata’s Unique Architecture In This Chapter ▶ Understanding Exadata’s architecture and components ▶ Unveiling the Exadata Database Machines ▶ Detailing the components of Exadata Storage Server

B

reaking the mold of traditional database server environments is the key to expanding performance while reducing cost and complexity. Legacy database systems’ architecture lacks the agility, scalability, and performance to meet business needs. The solution lies with what is offered in the Oracle Exadata Database Machine and how it is fast, agile, and scalable. In this chapter, we examine how the Exadata Database Machine and Exadata Storage Server are architected in terms of hardware and software components. We discuss the various server configurations that are used to support small, medium, and large applications. We also discuss how servers can be expanded to meet increased processing requirements. Finally, we take a look at the relationship between the Exadata Database Machine and the Exadata Storage Server.

What Is Exadata? Enclosed in a single physical cabinet, Oracle’s Exadata Database Machine is a complete package of preconfigured and integrated database servers, storage servers, network infrastructure, and all supporting software. As a “database These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

16

Exadata For Dummies, DLT Solutions Special Edition machine,” the intent is for a customer’s databases to be hosted entirely within the Exadata Database Machine without the need for additional, external storage, networking, or software components. Truly a “database in a box,” Exadata contains all components to host a customer’s database needs for OLTP, data warehousing, and database consolidation.

Core Exadata components A single Oracle Exadata appliance is composed of the following components: ✓ Exadata Database Machine ✓ Exadata Storage Server ✓ InfiniBand Network ✓ All necessary database, storage, and management software

Exadata Database Machine At the heart of each Exadata appliance are multiple (2 to 8, depending on configuration) database servers. Preconfigured with Oracle 11g R2 database software, the software comes in an Oracle Real Application Cluster (RAC) configuration. Oracle RAC is an advanced database architecture where multiple database instances simultaneously operate against a single physical database. Oracle RAC configuration provides for high availability and horizontal scaleable growth to increase performance.

Exadata Storage Server Storage responsibilities are intelligently handled by the Exadata Storage Server. Composed of multiple (3 to 14, depending on configuration) storage servers, Oracle has taken storage beyond simply being a disk array. Each storage server is a multi-CPU device with memory, flash storage, and either capacity or high-performance disk. Intelligent Oracle storage software offloads data intensive SQL processing from the database server to the storage server to improve performance and reduce data transfer. That’s right; your SQL processing takes place at the storage (not at the database) server!

InfiniBand Network Network communication between the Exadata Database Machine and the Exadata Storage Servers is handled by dual These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 2: Exadata’s Unique Architecture

17

40 Gigabit InfiniBand links that provide very fast and reliable connectivity. InfiniBand is also used when adding additional Exadata appliances to increase capacity. Redundant and reliable, Oracle uses an interconnect protocol to move data directly to database buffers reducing CPU overhead.

Software All the software needed to run Exadata is installed, optimized, and configured. Two operating system choices are available at installation time for the Exadata Database Machine: Oracle Linux or Solaris 11 Express. Exadata Storage Server uses Oracle Linux. Oracle 11g R2 Enterprise Edition with Oracle RAC is the standard for database software. All database features and software options remain available. Key database management products including Enterprise Manager (EM) Grid Control and Recovery Manager (RMAN) are also available.

Architecture It is important to understand that Exadata is an appliance that contains the database and storage servers supported by the network and software. Figure 2-1 shows the architecture of a single Exadata appliance. As you can see in Figure 2-1, within a single Exadata Database Machine X2-2, there are four database servers in an Oracle RAC configuration connected via the InfiniBand network to seven storage servers. The database servers form the database server component connected via InfiniBand to seven storage servers to form the storage server component. Just as Oracle Exadata supports database technology, Oracle Exalogic supports the application server tier. Oracle Fusion Middleware and Oracle Fusion Applications are key components within Exalogic and Oracle’s Exalogic Elastic Cloud environment.

Configured, optimized, and ready to use A common problem in IT is that it takes too long to implement new hardware and software solutions, but Oracle makes it easy to jump aboard the Exadata train by offering the following features to help the customer get started quickly. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Storage Server Storage Server Storage Server Storage Server Storage Server Storage Server

InfiniBand Network

DB RAC Instance DB RAC Instance DB RAC Instance

DB RAC Instance

DB Server DB Server

DB Server

DB Server

Storage Server

Exadata For Dummies, DLT Solutions Special Edition

Exadata Database Machine X2-2 Half Rack

18

Figure 2-1: Exadata architecture.

Install standard, optimized configurations All software is installed with a standard, optimized configuration. The customer doesn’t have to guess what features to configure or how to set up software; configuration and tuning settings are predetermined. Oracle’s top tuning experts have determined and tested the optimal settings for the hardware and software components

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 2: Exadata’s Unique Architecture

19

within Exadata. These configurations prevent potentially inefficient configuration settings from being imported by systems coming from non-Exadata environments. This practice allows for faster, less error prone configuration. Furthermore, because it is a standardized build, Oracle Support Services are already familiar with the environment.

Implement common systems without changes The most common database environments are OLTP, data warehouses, or consolidated database systems. Oracle realizes this and has designed Exadata to allow these systems to be deployed directly to the Exadata server. No special Exadata product certifications are needed; the idea is to allow the conversion to Exadata in days/weeks not months/years.

Leverage existing Oracle expertise The learning curve for IT administrative staff is reduced because they’re still using familiar software. Oracle 11g R2 Enterprise Edition is already commonly used in non-Exadata environments. Key database and server functions are managed by Oracle Enterprise Manager Grid Control, which is standard in most database environments. Oracle software such as partitioning, Data Guard, Automatic Storage Management (ASM), and Recovery Manager (RMAN) are still used as they were in non-Exadata environments. At the Operating System (OS) level, enterprise-grade Oracle Linux and Oracle Solaris use all the features and utilities of those OSs and are already familiar to Database Administrators (DBAs) and System Administrators (SAs) alike.

Working with Oracle experts Exadata Start-Up Pack and Support Services take an active role in ensuring that the migration to Oracle Exadata is a success and provide support after the migration is complete. Several packages are available with Oracle Exadata Start-Up Packs: ✓ Oracle Exadata Start-Up Advisory Service. Oracle engineers are in contact with the customer to identify needs, objectives, and provide recommendations and an action plan to achieve the desired result.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

20

Exadata For Dummies, DLT Solutions Special Edition

✓ Oracle Exadata Production Support Readiness. Oracle experts work with the customer to develop an implementation plan and work with the customer through the plan to ensure success. Oracle takes an active role in working with the customer during the installation, migration, and post-migration project phases. ✓ Oracle Exadata Installation Service. Oracle engineers install Exadata components and validate that all hardware, software, and operating system functionality is correct and documented. ✓ Oracle Exadata Configuration Service. In accordance with Oracle best practices, Oracle engineers configure operating system parameters, database software, network settings, and storage. All settings are validated and documented for the customer. ✓ Oracle Exadata Quarterly Patch Deployment Service. Quarterly patch requirements are identified for the customer and are applied by Oracle. High-level system checks are performed to ensure that the Exadata server is still compliant with Oracle best practices. Additionally, Oracle Support Services provides support for the multiple components of Exadata. The hardware has a warranty and support for operating systems, devices, and configurations; upgrades are available in addition to the database support that’s so well known. Oracle Support Services are available 24x7, and Auto Service Requests (ASR) automate the creation of trouble tickets.

The Exadata Database Machines Exadata Database Machines are available on two models: the X2-2 and X2-8. Common to both machines is the choice of either Oracle Linux or Oracle Solaris 11 Express. The same database software options are common to both server types. Additionally, the same Exadata Storage Server architecture and version is used although the number of storage servers and capacity vary. The primary difference between the Exadata X2-2 and X2-8 is related to capacity in terms of processor type, number of processors, memory, and underlying server type. These equate to different processing capacities and types of workload that each server best supports. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 2: Exadata’s Unique Architecture

21

Exadata Database Machine X2-2 The Exadata Database Machine X2-2 is based on the Sun Fire X4170 M2 and is designed for OLTP and data warehouse implementations of different sizes. The X2-2 comes in three different capacity configurations to support small, medium, and large computing capacity; these are referred to as Quarter Rack, Half Rack, and Full Rack. Each rack differs only in the number of database and storage servers within the Exadata Database Machine: ✓ Quarter Rack: 2 database servers with 3 Exadata Storage Servers ✓ Half Rack: 4 database servers with 7 Exadata Storage Servers ✓ Full Rack: 8 database servers with 14 Exadata Storage Servers

Quarter Rack The entry-level Exadata Database Machine is the Quarter Rack configuration composed of 2 database servers and 3 Exadata Storage Servers. Each server has 2 Intel Xeon X5670 6 core processors at 2.93 GHz. Total processing capacity is ✓ 24 CPU cores (12 per server) ✓ 192 GB memory (96 GB per server) ✓ 4 x 10GbE ports (2 per server) ✓ 2 InfiniBand 36 port switches ✓ 3 Exadata Storage Servers ✓ 1.1 TB Exadata Smart Flash Cache ✓ 21 TB raw disk on high-performance disk ✓ 72 TB raw disk on high-capacity disk ✓ Up to 5.4 GB/second of uncompressed raw disk bandwidth on high-performance disk ✓ Up to 3 GB/second of uncompressed raw disk bandwidth on high-capacity disk

Half Rack The mid-level Exadata Database Machine is the Half Rack configuration composed of 4 database servers and 7 Exadata These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

22

Exadata For Dummies, DLT Solutions Special Edition Storage Servers. Each server has 2 Intel Xeon X5670 6 core processors at 2.93 GHz. Total processing capacity is

✓ 48 CPU cores (12 per server) ✓ 384 GB memory (96 GB per server) ✓ 8 x 10GbE ports (2 per server) ✓ 3 InfiniBand 36 port switches ✓ 7 Exadata Storage Servers ✓ 2.6 TB Exadata Smart Flash Cache ✓ 50 TB raw disk on high-performance disk ✓ 168 TB raw disk on high-capacity disk ✓ Up to 12.5 GB/second of uncompressed raw disk bandwidth on high-performance disk ✓ Up to 7 GB/second of uncompressed raw disk bandwidth on high-capacity disk

Full Rack The fully loaded Exadata Database Machine is the Full Rack configuration composed of 8 database servers and 14 Exadata Storage Servers. Each server has 2 Intel Xeon X5670 6 core processors at 2.93 GHz. Total processing capacity is ✓ 96 CPU cores (12 per server) ✓ 768 GB memory (96 GB per server) ✓ 16 x 10GbE ports (2 per server) ✓ 3 InfiniBand 36 port switches ✓ 14 Exadata Storage Servers ✓ 5.3 TB Exadata Smart Flash Cache ✓ 100 TB raw disk on high-performance disk ✓ 336 TB raw disk on high-capacity disk ✓ Up to 25 GB/second of uncompressed raw disk bandwidth on high-performance disk ✓ Up to 14 GB/second of uncompressed raw disk bandwidth on high-capacity disk

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 2: Exadata’s Unique Architecture

23

As the need for capacity grows, the X2-2 can be scaled vertically to either Half Rack or Full Rack configuration at the customer site to reduce downtime. This scalability allows customers to grow into a larger Exadata configuration as their needs dictate. The ability to support capacity on demand is a key to supporting customers in growing environments. Exadata capacity can also be expanded by adding memory and disk-only expansion racks. Exadata Expansion Racks can add from 4 to 18 additional Storage severs to any Exadata Database Machine. The Exadata Memory Expansion kit allows you to increase the memory on X2-2 machines. Additional Quarter Rack, Half Rack, and Full Rack X2-2 appliances can be added to scale a system horizontally, as shown in Figure 2-2. InfiniBand networking allows for the connection of up to eight Exadata or Exalogic appliances to allow maximum computing capability. Such a configuration would be capable of supporting very large systems! Technical hardware specifics will change over time. To see most up-to-date specifics on the X2-2 and X2-8, be sure to review the Oracle Data Sheets at www.oracle.com/ technetwork/database/exadata/dbmachinex2-2-datasheet-175280.pdf and www.oracle. com/technetwork/database/exadata/dbmachinex2-8-datasheet-173705.pdf.

Exadata Database Machine X2-8 The Exadata Database Machine X2-8 is based on the Sun Fire X4800 and is designed for database server consolidation and the largest, most intensive OLTP and data warehouse implementations. The X2-8 comes in only one configuration, which is a Full Rack of 2 database servers and 14 Exadata Storage Servers.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Full Rack 8 DB Servers 14 Storage Servers 100 TB Performance 336 TB Capacity

Full Rack 8 DB Servers 14 Storage Servers 100 TB Performance 336 TB Capacity

Full Rack 8 DB Servers 14 Storage Servers 100 TB Performance 336 TB Capacity

Exadata X2-2

Exadata X2-2 Full Rack 8 DB Servers 14 Storage Servers 100 TB Performance 336 TB Capacity

Exadata X2-2

Full Rack 8 DB Servers 14 Storage Servers 100 TB Performance 336 TB Capacity

Full Rack 8 DB Servers 14 Storage Servers 100 TB Performance 336 TB Capacity

Exadata X2-2

InfiniBand Network

Exadata X2-2

Exadata X2-2

Full Rack 8 DB Servers 14 Storage Servers 100 TB Performance 336 TB Capacity

Exadata X2-2

Full Rack 8 DB Servers 14 Storage Servers 100 TB Performance 336 TB Capacity

Exadata X2-2

24 Exadata For Dummies, DLT Solutions Special Edition

Figure 2-2: Scaling horizontally via multi racks.

Each database server is a Sun Fire X4800 with 8 Intel Xeon X7560 8 core processors at 2.26 GHz (giving 64 cores) and 1 TB of memory. Total processing capacity is

✓ 128 CPU cores (64 per server)

✓ 2 TB memory (1 TB per server)

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 2: Exadata’s Unique Architecture

25

✓ 16 x 10GbE ports (8 per server) ✓ 3 InfiniBand 36 port switches ✓ 14 Exadata Storage Servers ✓ 5.3 TB Exadata Smart Flash Cache ✓ 100 TB raw disk on high-performance disk ✓ 336 TB raw disk on high-capacity disk ✓ Up to 25 GB/second of uncompressed raw disk bandwidth on high-performance disk ✓ Up to 14 GB/second of uncompressed raw disk bandwidth on high-capacity disk Because the X2-8 is already at a Full Rack configuration, more hardware resources cannot be added to an individual machine. However, as with all Exadata Database Machines, the X2-8 can be scaled horizontally to where additional Exadata machines are connected via the InfiniBand network to increase capacity.

Exadata Storage Server Disk and flash storage are intelligently managed by the Exadata Storage Server. More than just a simple disk array, each Exadata Storage Server is itself a server running Oracle Linux with CPU, memory, and management software. The “intelligence” of the storage server comes in the Oracle database specific optimizations that occur to improve performance and management capabilities. All software and configuration items are pre-installed and optimized to manage Oracle databases. Management software is also included. Administrators use the already familiar Oracle Enterprise Manager tool with System Monitoring plugin monitor and manage storage servers. Fault tolerance is provided by dual port InfiniBand connections and redundant, hot-swappable power supplies. Disks are also hot swappable in addition to mirroring, which is implemented by the volume management software. These features allow a system to tolerate failures more gracefully and result in higher availability.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

26

Exadata For Dummies, DLT Solutions Special Edition Several key features of Exadata Storage Server include

✓ Oracle Automatic Storage Manager (ASM) software handles storage management for Oracle databases and is a long-used product to manage storage for Oracle RAC databases. ASM provides volume management, data striping, and disk mirroring for storage volumes. It also manages the data distribution across available storage. ✓ Database files can be encrypted for maximum security as an additional security option.



Encrypting data at rest, which is data stored on a device such as disk, is a hot requirement these days. Many government agencies fear loss of sensitive data and mandate that steps be taken to encrypt data at rest.

✓ Processing of many database operations occurs on the processors of the storage server. Specific database queries are processed at the storage level so that only the data requested is returned to the database server. This enhancement results in greatly reduced network traffic and offloads CPU processing requirements from the database server to the storage server. The result is an overall boost in performance by distributing processing and reducing data transfer and traffic. ✓ Additional Exadata Storage Server technologies such as Smart Scan, Smart Flash Cache, and Hybrid Columnar Compression are covered in Chapter 3 to show how they are used for specific environments. Like Exadata Database Machines, the Exadata Storage Server is composed of multiple, individual servers providing a unit of capacity. A single Exadata Storage Server is composed of: ✓ Two 6 core Intel Xeon L5640 processors at 2.26 GHz ✓ 24 GB system memory ✓ 384 GB Exadata Smart Flash Cache ✓ 12 x 600 GB 15,000 RPM High Performance SAS or 12 x 2 TB 7,200 RPM High Capacity SAS ✓ HBA disk controller with 512 MB battery backed write cache ✓ Dual-port 40 Gb/s InfiniBand Host Channel Adapter

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 2: Exadata’s Unique Architecture

27

As database servers are used, more storage servers are used, which results in capacity increases. Scalability of storage solutions is important as storage requirements increase at a rapid rate. Disk storage is available in two categories: High Performance Serial Attached SCSI (SAS) and High Capacity Serial Attached SCSI (SAS). High Performance SAS runs at 15,000 RPM while High Capacity SAS runs at a slower 7,200 RPM. Before selecting a type of disk, determine whether your database applications are large enough to warrant slower — but higher capacity — storage, or whether the databases are smaller and can take advantage of faster disk. Be sure to also consider if your application requires faster disks, which is a common requirement in many environments. Flash Cache is very high-speed solid-state storage. Because it lacks moving parts, Flash Cache is very fast and reliable. Generally too expensive to store an entire database, Exadata leverages technologies to keep the most-critical, frequently accessed data on Flash Cache. Different configurations of Exadata Database Machines are supported by groups of 14, 7, or 3 Exadata Storage Servers corresponding to Full, Half, or Quarter Rack configurations. Each configuration yields different storage capacity, as shown in Table 2-1.

Table 2-1

Raw Storage Server Capacity

Storage Type

Full Rack

Half Rack

Quarter Rack

Performance Disk

100 TB

50 TB

21 TB

Capacity Disk

336 TB

168 TB

72 TB

Flash Cache

5.3 TB

2.6 TB

1.1 TB

Table 2-1 shows raw storage capacity, which is the amount of storage physically provided to the storage server. However, in actual use, not all of that storage is usable. Some of the raw storage will be lost due to overhead, and much more will be lost due to mirroring.



Mirroring is making redundant copies of data on disks. The benefit is that if one disk is damaged or lost, the data copy isn’t lost and is therefore still available.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

28

Exadata For Dummies, DLT Solutions Special Edition The amount of raw disk provided is not the amount of disk you’ll actually have available to store data.



Automatic Storage Management controls the level of mirroring of storage on disk. Under a normal redundancy model where a single mirrored copy is used, you can expect your usable data to be cut in half, as shown in Table 2-2.

Table 2-2

Mirrored Storage Server Capacity

Storage Type

Full Rack

Half Rack

Quarter Rack

Performance Disk

45 TB

22.5 TB

9.25 TB

Capacity Disk

150 TB

75 TB

31.5 TB

Flash Cache

5.3 TB

2.6 TB

1.1 TB

Note: Flash Cache is not impacted by mirroring. The most fault tolerant level of mirroring providing the highest level of redundancy is high-redundancy mirroring. As seen in Table 2-3, high redundancy uses triple mirroring to reduce usable raw disk to approximate one-third.

Table 2-3 High-Redundancy Storage Server Capacity Storage Type

Full Rack

Half Rack

Quarter Rack

Performance Disk

30 TB

15 TB

6.25 TB

Capacity Disk

100 TB

50 TB

21.5 TB

Flash Cache

5.3 TB

2.6 TB

1.1 TB

High redundancy may not be necessary for many environments. By understanding the requirements of the specific application, you then can intelligently determine the mirroring level. Exadata Storage Server is a scalable, intelligent component in the Exadata Database Machine. Beyond only storing data, each storage server uses hardware and software to reduce CPU processing on the database server and reduce the amount of data transferred over the network by executing some database processing at the storage level. Familiar Oracle tools such as Enterprise Manager and ASM combined with Oracle Linux allow for easy Oracle Exadata Storage Server management. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 3

Exadata’s OLTP and Data Warehousing Capabilities In This Chapter ▶ Understanding OLTP and data warehousing environments ▶ Examining tools for managing OLTP and data warehouse systems ▶ Improving performance with Exadata Smart Flash Cache and

Smart Scan

▶ Reducing storage with Exadata Hybrid Columnar Compression ▶ Providing High Availability and backup and recovery solutions

O

LTP database applications are very common, yet to be successful, they have special requirements. Many best practices have been developed for OLTP systems. Oracle has implemented these best practices with their database products for years and has a very successful track record for OLTP systems. Data warehousing systems have different processing characteristics. Instead of supporting many concurrent users, the system allows for fewer users, but they are users executing exhaustive, long-running queries. In this chapter, we see how OLTP and data warehouse systems are optimized and supported on the Exadata Database Machine.

OLTP Processing Requirements OLTP systems are the systems people login to and use to generate revenue. Often someone is buying a product or a service These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

30

Exadata For Dummies, DLT Solutions Special Edition via a database application. If the transaction is fast and occurs without errors, the company makes money, and the customer likely returns for future business. On the other hand, if the transaction processing is slow or fails or if the system itself is simply not available, you can lose revenue. Businesses place a price tag on lost revenue based on the number of hours an OLTP system is down. It is not uncommon for the price tag to be many thousands of dollars per hour if a system isn’t available. For these reasons, OLTP systems need to be available, fast, and execute without errors. These characteristics are common requirements for OLTP systems:

✓ High concurrency. Many people use an OLTP system at the same time. Locking or contention or system bottle necks are not acceptable. ✓ Fast performance. The more sales transactions processed per hour, the more revenue generated. Therefore, even a small-percentage increase in processing capability can yield tangible benefits. Furthermore, if a system runs so slow that it is a burden to the customer, that customer may opt to take their business elsewhere. ✓ High Availability (HA). If the system is down, revenue cannot be generated. Furthermore, frustrated customers will take their business to competitors. A simple cost-benefit analysis shows how investing in a High Availability architecture saves an organization money over time because unplanned downtime is eliminated. For an OLTP system to be successful, it must support many users simultaneously, provide fast performance, and be available when needed. Fortunately, Exadata effectively supports those requirements.

Data Warehouse Processing Data warehouses (DW) are the largest databases used to support long-running queries to identify patterns, trends, and “what if” analysis for the decision makers of an organization. While not as common as OLTP systems, DWs have processing requirements which need to be met. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 3: Exadata’s OLTP and Data Warehousing Capabilities

31

Data warehouses are in many ways the opposite of OLTP systems. To contrast against OLTP, a DW has the following characteristics: ✓ Low concurrency. Only a few people will use a DW at a given time. Because these systems are used for analysis and “what if” analysis, the average user won’t access it. However, the users in a DW are often “power users” who are analysts and managers working to identify new business opportunities. ✓ Long-running queries. While an OLTP transaction needs to be complete within seconds, DW queries can run hours or days. This is due to processing large volumes of data that involves complex processing logic. SQL tuning, efficient execution plans, and increased processing power are critical to DW performance. Any opportunities to reduce unnecessary processing need to be taken. ✓ Large size. DW’s are often built by data feeds from OLTP systems. That means that a DW often dwarfs the OLTP systems it was spawned from. Expect enormous and constantly expanding storage requirements. Although a DW is in many ways the opposite of an OLTP system, Oracle Exadata is well suited to meet the unique challenges of DW processing.

Enterprise Manager Grid Control Enterprise Manager (EM) Grid Control is an administrative tool used by administrators to intelligently manage and monitor components in an Oracle environment. More than just managing Oracle databases, EM also can manage and monitor application servers and monitor operating system resources. Plug-ins are available to allow EM to monitor non-Oracle components as well. Additional management and diagnostic packs provide additional EM capabilities. In the Exadata environment, not only are databases, database servers, and operating systems monitored, but so are Exadata Storage Servers. EM is a tool very well known to DBAs; many would have a difficult time doing their job without it. As part of Grid Computing, EM has “agents” installed on all the servers throughout an These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

32

Exadata For Dummies, DLT Solutions Special Edition enterprise that collect data and monitor components. This information is sent back to a management server where it is stored for analysis. The DBA can monitor in real time or can set up automatic alerts that are triggered when an event occurs (such as a full disk or down database). This “lights out” management capability is critical to running in a 24x7 environment. EM offers the following powerful features (among many others):

✓ Monitor real-time database activity and identify and troubleshoot database sessions causing issues. ✓ Identify and tune SQL queries running long or causing performance bottlenecks. ✓ Modify or create new database tables, indexes, code, or users. ✓ Back up and recover databases via Recovery Manager (RMAN). ✓ Monitor OS and database health for potential issues. EM automatically sends alerts or takes corrective action when issues are detected. ✓ Upgrade databases and apply patches. EM provides hooks into the Exadata environment beyond the standard database and OS capabilities it already has; EM can also monitor and manage storage servers. The greater benefit of what EM provides is the capability to better manage OLTP databases in an automated, error-free manner to promote performance, concurrency, and increased availability.

Exadata Smart Flash Cache The fastest storage media is solid state technology, also known as flash storage. Even the fastest disk will never be close to the speed of flash. Because flash storage is essentially nonvolatile memory, it has no moving parts, which makes it very fast and reliable. On the other hand, disk drives have moving parts that, due to the laws of physics, will always be slower and less reliable. Additional issues with disk storage are I/O bottlenecks that can occur in the disk subsystem.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 3: Exadata’s OLTP and Data Warehousing Capabilities

33

Flash storage is historically too expensive to store entire databases on as a disk replacement. This is especially the case as disk storage requirements continue to grow. However, it is a wise strategy to store your most critical and frequently used data on flash technology and leave your remaining data on disk storage. Given the large amount of flash storage available in some configurations, having an entire database in flash is not inconceivable. Oracle Exadata Smart Flash Cache leverages the flash that is available on each Exadata Storage Server for database caching of high-use objects. As Table 3-1 shows, the amount of flash available is dependent on your configuration.

Table 3-1

Raw Storage Server Capacity

Storage Type

Full Rack

Half Rack

Quarter Rack

Flash Cache

5.3 TB

2.6 TB

1.1 TB

Within the Exadata Storage Server, Oracle uses Sun Flash Accelerator F20 PCIe cards to provide flash. Each card provides 96 GB of flash, and each storage server has 4 cards. Because these are flash cards, they avoid I/O bottlenecks of disk controllers. You can use several methods to implement flash storage: ✓ Oracle Database and Exadata Storage Server software, integrated to automatically place data that is likely to benefit from caching in flash storage. For example, you would not cache large table scans but you would cache random reads against tables and indexes in flash storage. ✓ Database storage parameters are available to pin objects (such as tables) in cache in flash storage. The database administrator can modify storage clauses for database objects to be stored in flash cache. The database administrator and developers would be responsible for knowing which objects benefit the most, but tools such as EM could make the job easier.



Database administrators using Oracle database 11g R2 should consider Oracle Database Smart Cache. This is different from Exadata Smart Flash Cache.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

34

Exadata For Dummies, DLT Solutions Special Edition By increasing the speed for database transactions with tools like Exadata Smart Flash Cache, you can process more transactions per hour, resulting in greater revenue.

Exadata Smart Scan In traditional database architectures, a full table-scan query returns all data blocks for a table from disk to the database server for processing. In most cases, much of that data isn’t what was requested. Regardless, the database server still must use its CPU to process the data to find what it actually wants. Furthermore, network resources are wasted by moving data that’s usually not wanted by the user. Exadata Smart Scan offloads processing of queries from the database server to the storage server. Processors on the Exadata Storage Server process the data on behalf of the database SQL query. Only the data requested in the query is returned to the database server. Thus, the “heavy lifting” occurs at the Exadata Storage Server, which results in reduced processing on the database server and less network traffic. Using Exadata Smart Scan is transparent to the application and requires no SQL changes. Some metrics for data warehouse environments indicate a 10X or more reduction of data sent to the database server over the network. Combined with the reduction of CPU processing on the database server, Exadata Smart Scan is especially valuable to data warehouse systems.

Exadata Hybrid Columnar Compression Compressing data to reduce the amount of storage required is especially useful for data warehouse implementations where reducing storage reduces costs. In other cases, it can actually increase performance by reducing the size of data transferred over the network. Further savings are realized when you consider the reduced space needed for backups, standby systems, and production-sized test environments. Exadata Hybrid Columnar Compression (EHCC) is a technique where tables are organized based on similar columns and These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 3: Exadata’s OLTP and Data Warehousing Capabilities

35

then compressed. The key is organizing data based on column values; this keeps similar data together resulting in greater compression factors. Oracle Exadata Hybrid Columnar Compression provides two compression modes: Query Mode for data warehouses and Archive Mode for seldom-used archive data. Oracle reports a 10X compression factor for Query Mode and a 15X to 50X compression factor for Archive Mode data. Compressed tables are still accessed via standard SQL commands with SELECT, INSERT, and DELETE — however, an UPDATE of EHCC data does impact performance. The data itself is not uncompressed during processing; only the data to be returned to the user needs to be uncompressed. The decompression is executed at the storage server rather than at the database server to enhance performance.

High Availability, Backup, and Recovery Impact on a business or organization is severe when any critical systems are not available, but OLTP system downtime often has a large price tag attached. Because of this, Oracle Exadata Database Machine has many features to provide High Availability (HA) and enhance backup and recovery mechanisms. Knowing that any piece of hardware is destined to fail at some point, the key questions are how to enhance fault tolerance and eliminate single points of failure. These are areas where the Exadata database and storage servers excel. Although a hardware failure is a common cause for downtime, it’s not the only cause. Software errors, logical data corruption, human errors, or just taking a component down for patching for routine maintenance are other common causes for downtime. HA and backup and recovery are provided by: ✓ Multiple, redundant database servers available in every Exadata database server configuration. If one server fails or needs to be shutdown for maintenance, the other server is still available. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

36

Exadata For Dummies, DLT Solutions Special Edition

✓ Oracle Real Application Clusters (RAC) is installed by default on Oracle Exadata Database Machines. Oracle RAC allows for multiple database instances on separate servers to access the same physical database. ✓ Data Guard and Standby Databases allow a database to be the primary database for OLTP activity but keep another standby database located on a different server. The standby database is kept in sync to match the content of the primary so if the primary goes down, the standby can be used immediately. ✓ Enterprise Manager Grid Control promotes HA by allowing the DBA to effectively monitor and manage the database and support infrastructure. Many potential issues are automatically detected and fixed before they become serious problems. Using EM also reduces some human errors by administrators, which can cause downtime. ✓ Recovery Manager is used via EM to schedule database backup jobs and recover databases. Additionally, RMAN can be used to move or clone databases to new server environments. ✓ Automatic Storage Management (ASM) manages the storage volumes supporting the database. ASM allows the administrator to create disk groups, enforcing standard disk mirroring and also triple disk mirroring. This provides fault tolerance in the event of a disk failure. ✓ Redundant hardware components are throughout the Exadata Database Machine. Redundant network components, disk and disk subsystems, and servers provide fault tolerance for when a hardware failure does occur. HA and backup and recovery are critical to OLTP systems because a failure equates to very expensive losses for a company. The Exadata Database Machine and Storage Server are architected and support software to enhance HA for the OLTP system. Data warehouse users also gain from the HA benefits because the system will be available for their reporting processes.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 4

Benefiting from Shared Services and Server Consolidation In This Chapter ▶ Identifying the benefits of server consolidation ▶ Understanding how Exadata supports shared services and

consolidation

M

anaging a hodgepodge of disjointed database servers, storage devices, and network infrastructure is difficult, expensive, and inefficient. Smart IT organizations are pushing to consolidate into common, shared services architecture. By providing capacity on demand as a shared service, the IT cost goes down while performance, security, and ease of management go up. This chapter takes a take a look at how shared services and server consolidation benefit organizations.

Why Server Consolidation? Walk into most established IT shops and you’ll see a parade of different database servers, storage devices, and network configurations supporting the current database environments. The result is having different versions of server hardware, operating systems, Oracle database software, and configurations supporting very similar systems. Trying to manage this environment is a constant struggle, and here’s why:

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

38

Exadata For Dummies, DLT Solutions Special Edition

✓ Every server needs to be managed individually at the hardware, operating system, and database level. ✓ Differences in hardware and software versions require a customized approach to every system, even though different systems are themselves relatively similar. ✓ Enforcing configuration and optimization parameters are at best a manual, labor-intensive task. Best practices are difficult to uniformly implement across environments. Worse yet, poor configurations may exist undetected. ✓ Patching and upgrading heterogeneous customized servers and software is labor intensive and error prone. ✓ Adding new capacity is a challenge because you must “fit in” the new capacity into the current disjointed environment. ✓ Servers and storage are often either over utilized or under utilized in terms of capacity and performance. Either you’re over stressed or you’re wasting resources. Want to show your management a tangible way you are working to cut costs? Start your server consolidation initiative and show the cost savings in terms of the few computing environ ments you’ll need to support. All this extra customized work results in increased costs and complexity, but provides no real benefit to anyone.

How Exadata Supports Server Consolidation Using Exadata to consolidate your existing environments provides the following benefits: ✓ Standardized, homogeneous environment. Oracle software on Oracle hardware eliminates differences in hardware, operating systems, and database software. Servicing one type of system is easier than servicing many and allows your staff to become experts in a technology. ✓ Consistent hardware and software configurations. Exadata hardware and software are the same version and have the same base configuration settings. This makes management and maintenance much easier, faster, and less error prone. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 4: Benefiting from Shared Services and Server Consolidation

39

✓ Optimized configuration settings. Exadata comes preconfigured based on Oracle best practices. Starting with a clean configuration, you don’t have to worry about antiquated settings carrying forward into your new environment. ✓ Segmentation of applications. Virtualization technologies and techniques allow applications to still operate as if they were in their own dedicated environments. ✓ Support for OLTP, data warehouse, and mixed workloads. Different applications have different processing requirements. Exadata has optimizations for OLTP, data warehouse, and mixed workload environments. ✓ Easy and rapid implementation. Applications can move to Exadata with little or no customization; Exadata is designed for rapid implementation. Oracle Exadata Start-Up packs provide Oracle staff to assist with migration and support. ✓ Capacity on Demand (COD) computing. Once on Exadata, applications are configured to only use the capacity they need. Quarter Racks, Half Racks, Full Racks, and multiple servers are options. As more capacity is needed, more resources are allocated. ✓ Increased security. Software patching and configuration is easier and more uniform. Removing undocumented configuration settings and enforcing a consistency promotes security.



If your IT shop bills business units for support of services, Exadata shared services and Capacity on Demand computing makes this process easier and less expensive. In Figure 4-1, you see a before-and-after view of how Exadata can change a database environment. As you see, the original environment had multiple database servers, networks, and storage servers. With Exadata, you now have a single database machine supporting the same workload in a more efficient, highly available configuration.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Storage Server #2 Storage Server #1

InfiniBand Network

Storage Server #3

Application #4 Application #4 Application #1 Application #2 Application #3 Application #1 Application #2 Application #3

RAC Database Servers

After Exadata

RAC Database Servers

Exadata Database Server Machine

Network Attached Storage #2 Network Attached Storage #1 Internal Disk Storage

Application #1 OLTP DB Server

Application #2 OLTP DB Server

Application #3 Hybrid DB Server

Application #4 Data Warehouse

Exadata For Dummies, DLT Solutions Special Edition

Before Exadata

40

Figure 4-1: Exadata server consolidation.

Exadata I/O Resource Manager and Quality of Service Management A valid concern of server consolidation is how workload is managed on a shared environment. Often, mixed workloads prove to be difficult to manage. Exadata has several components to ensure processing workload is properly managed. These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 4: Benefiting from Shared Services and Server Consolidation

41

Exadata I/O Resource Manager Exadata I/O Resource Manager allows the administrator to create resource plans to prioritize how different I/O requests are processed. This allows different databases to have appropriate levels of I/O processing priority. For example, you can give a higher priority to a high visibility OLTP application while less visible data warehouse processing receives a lesser priority. Exadata I/O Resource Manager is beneficial in consolidated environments where different, conflicting workloads exist. For example, one feature is the ability to reserve flash storage for specific databases, ensuring that the most critical applications have the fastest storage. Another example is where development, test, and production databases coexist. High throughput for your production database can be ensured by giving development and test a lower relative I/O priority and giving production a higher relative I/O priority. This relative allocation is implemented via consumer groups that are defined by the administrator and implemented via the Exadata I/O Resource Manager. Oracle Database Resource Manager allows DBAs to create resource plans to manage CPU and parallel processing for databases. This is different than Exadata I/O Resource Manager but is a useful tool to use.

Exadata Quality of Service (QoS) Management Exadata Quality of Service (QoS) Management monitors workload on the system and manages shared resources to ensure processing continues at requested levels. QoS automatically tracks workload requests, identifies bottlenecks, and re­allocates resources to maintain the desired performance for a process. QoS helps you to maintain a stable, consistent level of performance, which is key to keeping within your Service Level Agreements with customers. Using Oracle Enterprise Manager, the administrator creates policies to define the required performance for a workload. QoS takes the inputs, tracks workload requests, and makes These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

42

Exadata For Dummies, DLT Solutions Special Edition recommendations or changes as needed to ensure that they finish within the required timeframe. If QoS needs to make changes, it provides an audit log of changes. Enterprise Manager provides dashboards and options to view performance metrics and edit policies. QoS promotes shared services and server consolidation via:

✓ Allowing databases with different levels of importance to share the same machine. Development, test, and production can coexist without risk of negatively impacting production performance. ✓ Preventing performance issues from noticeably impacting applications by detecting bottlenecks and providing recommendations to fix them before they become real problems. ✓ Providing automated, detailed, and end-to-end tracking and monitoring of workloads and logging of corrective changes. ✓ Ensuring that resources are not over- or under-allocated for a workload. This results in more intelligent management of shared resources. ✓ Simplifying performance management for administrators via Enterprise Manager and automated processes. ✓ Enforcing Service Level Agreements in an automated manner with logging and metrics to prove the SLA has been met. Exadata I/O Resource Manager and Exadata Quality of Service allow mixed workload systems to be consolidated into the same machine while ensuring performance is not hindered. The end result is lower total cost to host multiple databases while meeting processing and performance requirements.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Chapter 5

Ten (Okay, Five) Things to Look For in an Exadata Solution In This Chapter ▶ Benefiting from the Exadata Database Machine

M

oving to the Exadata Database Machine is a potential game-changer for your IT infrastructure. Here, we cover some of the benefits you can expect.

Unified Hardware and Software Exadata is unified hardware and software optimized to work together. Integration between each hardware and software component and subsystem is tested and optimized for performance and reliability. When working with complex systems, having this integration and optimization done for you makes your Exadata implementation much easier.

Preconfigured Components You don’t have to reinvent the wheel when you get an Exadata Database Machine; it comes to you with the features preconfigured and optimized! Oracle uses their best practices and lessons learned when configuring your Exadata system.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

44

Exadata For Dummies, DLT Solutions Special Edition

Scaling for Large Applications Exadata’s Capacity on Demand (COD) simplifies the situation because it allows you to start with the capacity you need and expand as your processing requirements increase. Leveraging the Exadata Database Machine X2-2, you start with a Quarter or Half Rack configuration and add additional capacity as needed. You can scale vertically into a Half or Full Rack configuration only when you need to. When you need to scale horizontally beyond a Full Rack configuration, simply add additional X2-2 or X2-8 Exadata Database Machines. This allows you to purchase and support only what you need without being wasteful by having unused capacity.

Reducing Infrastructure Costs through Consolidation Because Exadata is a single solution, you no longer need to support multiple independent database servers or their storage devices or the network. That reduction in hardware equates to large savings. Furthermore, licensing costs are reduced because less overall software is needed to support all the systems. Finally, administrators have fewer environments to manage, and the Exadata environment is standardized and optimized to allow for easier administration.

Agile Infrastructure to Meet Changing Requirements To meet new business requirements, you need an agile system. Exadata creates new environments quickly because of its virtualization and Capacity on Demand capabilities. This capability allows you to spend more time on development to meet a new requirement rather than waiting on the purchase, installation, and configuration of new hardware or software.

These materials are the copyright of John Wiley & Sons, Inc. and any dissemination, distribution, or unauthorized use is strictly prohibited.

Runs Oracle

10x Faster

*

The World’s Fastest Database Machine •

Hardware by Sun

•

Software by Oracle

* But you have to be willing to

spend 50% less on hardware.

10x faster based on comparing Oracle data warehouses on customer systems vs. Oracle Exadata Database Machines. Potential savings based on total hardware costs. Oracle Database and options licenses not included. Actual results and savings may vary.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved. Oracle and Java are registered trademarks of Oracle and/or its affiliates.

Print Ad Resize

5.5” x 8.5” Exadata For Dummies Digital Book (Full Page Ad)

PUB NOTE: Please use center marks to align page. Job No.: Headline: Date: Project: Type: Live: Trim: Bleed:

212M_EXD_10xFaster_EFD Runs Oracle 10x Faster 11/3/2011 NAS Regional Fulfillment Digital Book 5” x 8” 5.5” x 8.5” 6” x 9”

Fonts: Univers LT Std. 75 Black, 65 Bold, 55 Roman, 45 Light, 67 Bold Condensed, 57 Condensed

PRODUCTION NOTES

READER

01

LASER%

Released

11/3 2011

Please examine these publication materials carefully. Any questions regarding the materials, please contact Darci Terlizzi (650) 506-9775