D67016GC20 Exadata Workshop Part2

O ti i i Optimizing Database D t b Performance P f with ith Exadata

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives After completing this lesson, you should be able to: • Describe the recommendations for optimizing Exadata performance

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 7 - 2

Optimizing Performance • •

Start with best practices for ASM and Oracle Database Areas for special consideration: – – – – –

Flash memory usage Compression usage Index usage ASM allocation unit size Minimum extent size

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Optimizing Performance Optimizing the performance of your Exadata environment begins with following good practices for database design and application development development. From an administration viewpoint, you should continue to follow the best practices for ASM and Oracle Database in conjunction with advice and statistics provided by tools such as SQL monitor and SQL Tuning Advisor. In addition, there are a number of areas for special consideration listed in the slide. These are covered in the remainder of this lesson.

Exadata and Database Machine Administration Workshop 7 - 3

Flash Memory Usage • •

Each Exadata server contains 384 GB of high performance flash memory. Flash memory can be configured in two ways: – Exadata Smart Flash Cache: — — —

Speeds up access to frequently accessed data Uses all available flash memory by default Can be managed automatically for maximum efficiency —

—

Users can only provide optional hints to influence caching priorities.

Beneficial for OLTP and Data Warehouse workloads

– Flash-based permanent storage: — — — —

Uses flash memory as a premium storage device Must be configured by an administrator Requires deliberate planning to use efficiently Additional benefits for specific applications Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Flash Memory Usage Each Exadata server contains 384 GB of high performance flash memory. B d By default, f lt when h E Exadata d t iis fifirstt configured, fi d thi this flflash h memory iis configured fi d as E Exadata d t Smart Flash Cache. Exadata Smart Flash Cache provides a caching mechanism for frequently accessed data on each Exadata cell. It is a write-through cache which is useful for absorbing repeated random reads, and very beneficial to online transaction processing (OLTP). Exadata Smart Flash Cache also benefits data warehousing applications by caching the most frequently accessed data. y focuses on caching g frequently q y accessed data and Exadata Smart Flash Cache automatically index blocks, along with performance critical information such as control files and file headers. In addition, DBAs can influence caching priorities using the CELL_FLASH_CACHE storage attribute for specific database objects. For example, to prioritize caching for the CALLDETAIL table use the following SQL command: ALTER TABLE calldetail STORAGE (CELL_FLASH_CACHE KEEP) Because of this setting, Exadata will cache data for the CALLDETAIL table more aggressively than otherwise. Without this setting, Exadata caches individual table and index blocks. With this setting, Exadata attempts to cache scans on the CALLDETAIL table. This makes it more likely that subsequent scans on the CALLDETAIL table will also be satisfied using Exadata Smart Flash Cache. Exadata and Database Machine Administration Workshop 7 - 4

Flash Memory Usage (continued) This setting also causes Exadata to clear out other tables from the cache before clearing out cached data for the CALLDETAIL table. This means that CALLDETAIL records will remain in ada a S Smart a Flash as Cac Cache e longer o ge than a cac cached ed da data a for o o other e tables. ab es Exadata If the CALLDETAIL table is spread across several Exadata servers, each one will cache its part of the table in its own Exadata Smart Flash Cache. If the caches are sufficient in size, the CALLDETAIL table is likely to be completely cached over time. Alternatively, you can configure a smaller Exadata Smart Flash Cache and use the remaining flash memory as a flash disk drive. You can then create grid disks based on flash memory. These flash disks can be used to configure high performance ASM disk groups housing their own tablespaces. In this case, the flash memory is not used to cache data, rather it becomes a high performance permanent store for some of your data. You should consider configuring flashbased permanent storage areas in the following cases: • Heavy write applications: For applications that perform many updates, or in cases of heavy data ingest, you may consider storing specific tables, partitions or indexes on flash disks. In these cases, the high volume of writes can consume a considerable portion of the physical p y disk bandwidth. Storing g insert- and update-intensive p tables,, partitions, p , and indexes on flash-based storage means that there are no physical disk I/Os associated with those objects. The physical disk bandwidth saved using this configuration can be used by other operations increasing the overall throughput of the system. Flash disks can also be used effectively in conjunction with partitioning. In many time-based partitioning scenarios, the most current data can be quite volatile while older data is subject to few updates, if any. In these situations, you can host your most current partitions in a flash-based tablespace and routinely move these partitions to a disk-based disk based tablespace as their volatility diminishes due to age. • Star-schemas: In a data warehouse based on a star schema, you may be able to store your entire fact table on flash disk. This effectively acts as a permanent cache for this key table. If your fact table is too large to store on flash disk, you can partition it and store the most referenced partition on flash disk. The flash memory on each cell can be divided between Exadata Smart Flash Cache and permanent flash-based storage. Both areas can be any size as long as the total does not exceed 384 GB on each cell. In practice, the effectiveness of Exadata Smart Flash Cache drops if the cache size is very small so in most cases, you would allocate a substantial proportion of flash memory to Exadata Smart Flash Cache. Note that Exadata Smart Flash Cache is largely automatic. After it is configured, you can only influence caching priorities for specific tables using the CELL_FLASH_CACHE storage attribute. In contrast, configuring permanent flash-based flash based storage requires accurate space planning to make the most effective use of this premium resource.

Exadata and Database Machine Administration Workshop 7 - 5

Compression Usage Compression Method

CREATE/ALTER TABLE Syntax

Compression Ratio

CPU Overhead

Typical Applications

Basic Compression

COMPRESS [BASIC]

High for direct path inserts Conventional path inserts and updates are not compressed

Minimal: Oracle Database performs compression and decompression

DSS

OLTP Compression

COMPRESS FOR OLTP

High for all transaction types

Minimal: Oracle Database performs compression and decompression, compression for DML is performed in batches

OLTP and DSS

Warehouse Compression

COMPRESS FOR QUERY [LOW|HIGH]

Higher for direct path inserts High for conventional path inserts and updates

Higher: Decompression is performed by Exadata

DSS

Online Archival Compression

COMPRESS FOR ARCHIVE [LOW|HIGH]

Highest for direct path inserts High for conventional path inserts and updates

Highest: Decompression is performed by Exadata

Archiving

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Compression Usage Many people think of data compression as they approach the capacity limits of their storage. Commonly data compression is seen as a performance overhead, Commonly, overhead which must be tolerated in order to deliver extra storage capacity. However, in cases where I/O bandwidth is limited, data compression can be an effective tool to increase performance by using available CPU capacity to effectively increase the I/O throughput of a storage system. Oracle Database provides the following modes of data compression: • Basic compression compresses data at the Oracle block level. It allows more data to be stored in each block by replacing duplicate values with a smaller symbolic representation. t ti For F example, l if the th numeric i value l 99999 was presentt 50 titimes in i a block of data it could be replaced by 50 occurrences of the # symbol along with an entry in a symbol table (also stored in the block) defining the use of the symbol. The degree of compression depends on the level of duplication in each block. Higher compression ratios can be achieved by using larger block sizes or by sorting data to increase the coincidence of duplicate values. Data remains in row-major format where the columns in each row are stored together. Compression occurs when data is loaded using a direct path load operation, such as CREATE TABLE AS SELECT or direct path SQL*Loader. Tables that use basic compression support DML operations, however any data that is inserted or updated remains uncompressed. Basic compression is useful in decision support systems (DSS) where the compressed data is subject to minimal change. Exadata and Database Machine Administration Workshop 7 - 6

Compression Usage (continued) •

OLTP compression uses the same compression technique as basic compression, but provides support for compressing transactional data from all DML statements, not just path loads. This makes it useful in DSS and online transaction p processing g (OLTP) ( ) direct p environments. Instead of compressing changes as they are written, all the changes to a data block are compressed as a batch whenever a change causes the block to become fuller than an internally controlled threshold. In other words, only the DML statements that trigger compression experience the overhead of compression for the associated data blocks, however most DML statements experience no compression overhead at all. This means that compression can be implemented on OLTP systems without significantly impacting the overall transactional throughput of the system.

In addition to the basic and OLTP compression modes provided by Oracle Database, Exadata provides Exadata Hybrid Columnar Compression. Exadata Hybrid Columnar Compression technology uses a modified form of columnar storage instead of row-major storage. Sets of rows are stored in an internal structure called a compression unit. Within each compression unit, the values for each column are stored together along with metadata that maps the values to the rows. Compression is achieved by replacing repeating values with smaller symbolic references. Because a compression unit is much larger than an Oracle block, block and because column organization brings similar values together, Exadata Hybrid Columnar Compression can deliver much better compression ratios than both basic and OLTP compression. The best rates of compression are achieved using direct path loads. Exadata Hybrid Columnar Compression provides a choice of compression modes to achieve the proper trade-off between disk usage and CPU overhead: • Warehouse compression: This type of compression is optimized for query performance, and is intended for DSS and data warehouse applications. • Online archival compression: This type of compression is optimized for maximum compression ratios, and is intended for historical data and data that does not change. Exadata Hybrid Columnar Compression supports DML operations on compressed data. However, such operations attract a considerable CPU overhead and yield a lower compression ratio than for direct path loads. In addition, DML on tables using Exadata Hybrid Columnar Compression requires the entire compression unit to be locked, which may reduce concurrency. Finally, updates to rows using Exadata Hybrid Columnar Compression cause rowids to change. As a result, Exadata Hybrid Columnar Compression is recommended for situations where data changes are infrequent or where data sets are reloaded rather than substantially changed. In conclusion, Exadata Hybrid Columnar Compression makes effective use of Exadata server hardware to deliver the highest levels of compression for data in an Oracle database. It is best suited su ed to o cases where e e the e da data a is s not o subjec subject to o subs substantial a a cchange. a ge For o transactional a sac o a da data a se sets, s, you should consider OLTP compression instead of Exadata Hybrid Columnar Compression. In all cases, you should be aware of the relative merits and overheads associated with each compression type in order to choose the best approach for your situation.

Exadata and Database Machine Administration Workshop 7 - 7

Index Usage • • •

With Exadata, scans might perform better than index lookups for some queries. Consider removing indexes where scans deliver acceptable p p performance. Removing unnecessary indexes will: – Improve DML performance – Save storage space

•

Test the effect of removing indexes by making them invisible:

SQL> ALTER INDEX INVISIBLE;

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Index Usage Some queries that require indexes when using conventional storage will perform acceptably without indexes using Exadata Exadata. Review your queries that use indexes to determine if they would run acceptably with Exadata scans. To test if queries run acceptably without an index, you can make the index invisible to the optimizer. An invisible index still exists and is maintained by DML operations, but it is not used by the optimizer for queries. To make an index invisible, use the following command: ALTER INDEX INVISIBLE Removing unnecessary indexes will aid the performance of DML operations and will save storage space.

Exadata and Database Machine Administration Workshop 7 - 8

ASM Allocation Unit Size • •

By default, ASM uses an allocation unit (AU) size of 1 MB. For Exadata, the recommended AU size is 4 MB. – AU size must be set when a disk group is created. – AU size cannot be altered after a disk group is created created. – AU size is set using the AU_SIZE disk group attribute:

SQL> CREATE DISKGROUP data NORMAL REDUNDANCY DISK 'o/*/data_CD*' ATTRIBUTE 'compatible.rdbms' = '11.2.0.0.0', p = '11.2.0.0.0', , 'compatible.asm' 'cell.smart_scan_capable' = 'TRUE', 'au_size' = '4M';

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

ASM Allocation Unit Size To achieve fast disk scan rates, it is important to lay out segments with at least 4 MB of contiguous space space. This allows disk scans to read 4 MB of data before performing another seek at a different location on disk. To ensure segments are laid out with 4 MB of contiguous space, set the Oracle ASM allocation unit size to 4 MB. The allocation unit can be set with the AU_SIZE disk group attribute when creating the disk group.

Exadata and Database Machine Administration Workshop 7 - 9

Minimum Extent Size •

Large segments should have initial extents of 8 MB: – Stops needless proliferation of small extents in the database – Minimizes sub-optimal reads for small extents – Compliments the 4 MB ASM AU size recommendation

•

Large initial extents can be specified using the: – INITIAL storage parameter for specific objects or for all objects in a locally managed tablespace – CELL_PARTITION_LARGE_EXTENTS initialization parameter for all new segments in a partitioned object: —

—

—

TRUE: Sets an 8 MB minimum extent size, but will trim last unloaded extents to 64 KB boundary for direct loads ALWAYS: Sets an 8 MB minimum extent size and also minimizes the creation of suboptimal extent sizes at the end of a load DEFAULT: Uses traditional extent allocation Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Minimum Extent Size Extent size is managed automatically in locally managed tablespaces. This option automatically increases the size of the extent depending on segment size size, available free space in the tablespace, and other factors. By default, the extent size starts at 64 KB and increases in increments of 1 MB, 8 MB, or 64 MB. Generally speaking, it is recommended that large segments should be defined with larger than default initial extents to minimize the needless proliferation of small extents in the database. For Exadata, the recommendation is to have initial extents of 8 MB for large segments. This setting complements the recommendation to set the ASM allocation unit size to 4 MB. MB The following options can be used to influence the database to allocate large extents: • The INITIAL storage parameter will set the size of the first extent for a newly created object. It can be set specifically for individual objects or it can be set at the tablespace level for locally managed tablespaces. For objects that are large, set INITIAL to 8 MB. • The CELL_PARTITION_LARGE_EXTENTS initialization parameter can be used to set the INITIAL storage g p parameter to 8 MB for all new segments g in a p partitioned object. j This parameter is relevant only to locally managed tablespaces created with the AUTOALLOCATE option, not the UNIFORM option. The parameter does not affect nonpartitioned objects. The parameter may be set to the following values: TRUE, ALWAYS, and DEFAULT. In general, Oracle recommends setting this parameter to TRUE. Exadata and Database Machine Administration Workshop 7 - 10

Quiz For which of the following scenarios would it make sense to consider altering the default Exadata Smart Flash Cache in order to configure a flash-based tablespace in your database. 1. When you can fit 10 or more of your tables into a flash-based t bl tablespace. 2. When you can fit at least 80% of your most used tables into a flash-based tablespace. 3. When moving specific tables and indexes to a flash-based tablespace will soak up heavy write activity, freeing disk I/O bandwidth for other operations. operations 4. When you have key application objects (such as a star-schema fact table) that you wish to cache all the time, you understand their space requirements intimately, and you are prepared to accept the additional management effort of configuring a flashbased tablespace. Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 3, 4

Exadata and Database Machine Administration Workshop 7 - 11

Quiz Which of the following CREATE TABLE compression clauses are only available to use in conjunction with Exadata storage. 1 COMPRESS BASIC 1. 2. COMPRESS FOR OLTP 3. COMPRESS FOR QUERY 4. COMPRESS FOR ARCHIVE

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 3, 4

Exadata and Database Machine Administration Workshop 7 - 12

Summary In this lesson, you should have learned how to: • Describe the recommendations for optimizing Exadata performance

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 7 - 13

Additional Resources •

Lesson Demonstrations (Viewlets) – Configuring Flash-Based Disk Groups —

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/071ExadataFla shDiskGroups/071exadataflashdiskgroups_viewlet_swf.html

– Examining Exadata Hybrid Columnar Compression —

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/072ExadataCo mpressionAdmin/072exadatacompressionadmin_viewlet_swf.ht ml

– Index Elimination with Exadata —

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/073ExadataInd exElimination/073exadataindexelimination_viewlet_swf.html

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 7 - 14

Practice 7 Overview: Optimizing Database Performance with Exadata In these practices, you will explore how to optimize database performance using: • Flash-based disk groups • Exadata Hybrid Columnar Compression • Index elimination

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 7 - 15

D t b Database M Machine hi O Overview i and dA Architecture hit t

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives After completing this lesson, you should be able to: • Describe the architecture and various configuration options for Database Machine

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 8 - 2

Introducing Database Machine • • • • • •

Fully integrated platform for Oracle Database Based on Exadata storage High-performance High performance and high-availability high availability for all Oracle Database workloads Balanced hardware configurations Well suited as a database consolidation platform Si l and Simple d ffastt tto iimplement l t

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Introducing Database Machine Database Machine is a fully-integrated Oracle Database platform based on Exadata storage technology. Database Machine provides a high-performance, highly-available solution for all database workloads, ranging from scan-intensive data warehouse applications to highly concurrent OLTP applications. Special attention has been paid to ensure that Database Machine is a well balanced platform. Throughout the hardware architecture of Database Machine, components and technologies have been specially matched to eliminate bottlenecks while maintaining good hardware utilization. Using the unique clustering and workload management capabilities of Oracle Database, Database Database Machine is well-suited for consolidating multiple databases onto a single Database Machine. Delivered as a complete package of software, servers, and storage, Database Machine is simple and fast to implement. Note: While Database Machine is a fully-integrated platform solution comprised of specific hardware and software components, Oracle offers Database Machine as a series of separately purchasable components. Customers can choose from the different hardware configurations that are available. available Appropriate licensing of Oracle Database and Exadata software is also required. In addition, Database Machine is highly complementary with clustering and parallel operations, so Oracle Real Application Clusters and Oracle Partitioning are highly recommended software options for Database Machine. Exadata and Database Machine Administration Workshop 8 - 3

Database Machine X2-2 Full Rack

14 Exadata Storage Servers (all high performance or all high capacity) 1 Cisco Ethernet switch (48-port)

8 Intel-based 2 CPU Sun Fire database servers Keyboard, Video, Mouse (KVM) hardware

3 Sun Datacenter InfiniBand Switch 36 (36-port managed QDR switch i h - 40Gb/s) 40Gb/ )

2 Power Distribution Units (PDUs) at rear

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine X2-2 Full Rack Database Machine is offered in 4 configurations. The contents of a Database Machine X2-2 Full g is illustrated in the slide. Rack configuration

Exadata and Database Machine Administration Workshop 8 - 4

X2-2 Database Server Hardware Details (Sun Fire X4170 M2) Processors

2 Six-Core Intel® Xeon® X5670 processors (2.93 GHz)

Memory

96 GB (12 x 8 GB)

Local Disks

Four 300 GB 10K RPM SAS disks

Disk Controller

Disk controller HBA with 512 MB battery backed cache

Network

Two InfiniBand 4X QDR (40 Gb/s) ports (1 dual-port PCIe 2.0 HCA)

Four 1GbE Ethernet ports Two 10GbE Ethernet SFP+ ports (1 dual-port 10GbE PCIe 2.0 network card based on the Intel 82599 10GbE Controller technology)

Remote Management

1 Ethernet port (ILOM)

Power Supplies

2 redundant hot-swappable power supplies

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

X2-2 Database Server Hardware Details (Sun Fire X4170 M2) Database Machine X2-2 uses standardized database server hardware based on 64 bit Intelbased Sun Fire servers servers. The slide shows a description of the database server hardware hardware.

Exadata and Database Machine Administration Workshop 8 - 5

Start Small and Grow

X2-2 Quarter Rack

X2-2 Half Rack

X2-2 Full Rack

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Start Small and Grow In addition to a Database Machine X2-2 Full Rack, which contains eight database servers, 14 Exadata cells and 3 InfiniBand switches, Oracle offers two other Database Machine X2-2 configurations which allow customers to start small and grow: • A Quarter Rack contains two database servers, three Exadata cells and 2 InfiniBand switches. • A Half Rack contains four database servers, seven Exadata cells and 3 InfiniBand switches. All of the Database Machine X2-2 configurations are packaged in a standard 19-inch 42U rack. All configurations are pre-cabled and supplied with a KVM and Ethernet switch to complete the configuration. In addition, Oracle offers hardware upgrade kits that allow customers to upgrade from a Quarter Rack to a Half Rack, or to upgrade from a Half Rack to a Full Rack.

Exadata and Database Machine Administration Workshop 8 - 6

Database Machine X2-8 Full Rack

14 Exadata Storage Servers (all high performance or all high capacity)

2 Intel-based 8 CPU Sun Fire database servers

1 Cisco Ethernet switch (48-port)

3 Sun Datacenter InfiniBand Switch 36 (36-port managed QDR switch i h - 40Gb/s) 40Gb/ )

2 Power Distribution Units (PDUs) at rear

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine X2-8 Full Rack In addition to the three Database Machine X2-2 configurations, Oracle now offers Database Machine X2-8 X2 8. Database Machine X2-8 X2 8 is only offered in a Full Rack. Rack There are no smaller configurations. In many ways Database Machine X2-8 is essentially the same as Database Machine X2-2 Full Rack. Both contain 14 Exadata cells, 3 InfiniBand switches, 2 power distribution units (PDUs) and an Ethernet switch. The major difference is that the eight 2 CPU database servers in Database Machine X2-2 Full p by y two 8 CPU database servers. Each of the larger g database servers Rack are replaced contains more RAM and is equipped with eight 10 Gb/sec Ethernet ports for high-speed client network access in addition to the 1 Gb/sec Ethernet ports that support administrative access. Database Machine X2-8 does not come equipped with KVM hardware. Database Machine X2-8 complements the X2-2 models. Like the X2-2 models, it is designed for database consolidation, large OLTP and data warehouse applications. The large memory configuration of Database Machine X2-8 makes it a particularly good platform for database consolidation. lid ti C Customers t that th t currently tl run their th i applications li ti on llarge SMP systems t may b be more comfortable choosing Database Machine X2-8.

Exadata and Database Machine Administration Workshop 8 - 7

X2-8 Database Server Hardware Details (Sun Fire X4800)

Processors

8 x Eight-Core Intel® Xeon® X7560 Processors (2.26 GHz)

Memory

1 TB (128 x 8 GB)

Local Disks

Eight 300 GB 10K RPM SAS disks

Disk Controller

Disk controller HBA with 512 MB battery backed cache

Network

Eight InfiniBand 4X QDR (40Gb/s) ports (4 dual-port PCE 2.0 express modules) Two Network Express Modules (NEM), providing a total of Eight 1GbE Ethernet ports and Eight 10 GbE Ethernet SFP+ ports (4 Fabric Express Modules (FEM) using Intel 82599 10GbE controller technology)

Remote Management

1 Ethernet port (ILOM)

Power supplies

4 redundant hot-swappable power supplies

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

X2-8 Database Server Hardware Details (Sun Fire X4800) The slide shows a description of the database server hardware used in Database Machine X2-8 X2 8.

Exadata and Database Machine Administration Workshop 8 - 8

Database Machine Capacity

Raw

Disk1

X2-2 X2-2 Quarter Half Rack Rack

X2-2 Full Rack

X2-8 Full Rack

High Performance

21 TB

50 TB

100 TB

100 TB

High Capacity

72 TB

168 TB

336 TB

336 TB

1.1 TB

2.6 TB

5.3 TB

5.3 TB

High Performance

6 TB

14 TB

28 TB

28 TB

High Capacity

21 TB

50 TB

100 TB

100 TB

Raw Flash1 User

Data2

1 - Raw capacity calculated using 1 GB = 1000 x 1000 x 1000 bytes and 1 TB = 1000 x 1000 x 1000 x 1000 bytes. 2 - User Data: Actual space for uncompressed end-user data, computed after single mirroring (ASM normal redundancy) and after allowing space for database structures such as temporary space, logs, undo space, and indexes. Actual user data capacity varies by application. User Data capacity calculated using 1 TB = 1024 * 1024 * 1024 * 1024 bytes.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine Capacity The table in the slide gives you some key capacity metrics for the available Database Machine g configurations.

Exadata and Database Machine Administration Workshop 8 - 9

Database Machine Performance

Raw Disk Throughput1

X2-2 Quarter Rack

X2-2 Half Rack

High g Performance

5.4 GB/s

12.5 GB/s 25 GB/s

25 GB/s

High Capacity

3 GB/s

7 GB/s

14 GB/s

14 GB/s

11 GB/s

25 GB/s

50 GB/s

50 GB/s

High Performance

10,800

25,000

50,000

50,000

High Capacity

5 400 5,400

12 500 12,500

25 000 25,000

25 000 25,000

225,000

500,000

1,000,000 1,000,000

Raw Flash Throughput1 Raw Disk IOPS2 Flash IOPS2

X2-2 Full Rack

X2-8 Full Rack

1 – Bandwidth is peak physical disk scan bandwidth, assuming no compression. 2 – IOs per second (IOPS) – Based on IO requests of size 8K

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine Performance The above table gives you some key performance metrics for the different Database Machine g configurations. Note: These metrics do not take into account compression. As a result, they are quite conservative. With compressed data, you can achieve much higher throughput rates. In all cases, actual performance will vary by application.

Exadata and Database Machine Administration Workshop 8 - 10

Database Machine X2-2 Architecture

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine X2-2 Architecture The diagram in the slide shows how the major components of Database Machine X2-2 are connected to each other other. The diagram shows the KVM hardware hardware, Ethernet switch and InfiniBand switches. For the sake of clarity, only one database server and one Exadata server are shown. Database Machine contains three major networks: • Management/ILOM Network. This is a standard Ethernet/IP network which is used to manage Database Machine. Using this network, administrators can access the database servers and the Exadata servers using the KVM hardware or by using remote login facilities such as Secure Shell (SSH). The database servers and Exadata servers also l provide id an Eth Ethernett iinterface t f ffor IIntegrated t t d Li Lights-Out ht O t Management M t (ILOM). (ILOM) ILOM provides a powerful set of remote administration facilities. Using ILOM, administrators can remotely monitor and control the state of the server hardware. The InfiniBand switches also provide an Ethernet port for remote configuration and management purposes. • Client Access Network. There is also a standard Ethernet network which is primarily used to provide database connectivity via Oracle Net software. The client access network can be separate from the management network (as illustrated in the diagram) or both functions can be provided by the same network. Single-network configurations are quite typical. Exadata and Database Machine Administration Workshop 8 - 11

Database Machine X2-2 Architecture (continued) When Database Machine is initially configured, customers can choose to configure the database servers with a dedicated client network interface (NET1) or they can g a bonded network interface ((using g NET1 and NET2). ) A bonded choose to configure client access network interface can provide protection if a network interface fails however using bonded interfaces may require additional configuration in the customer’s network. • InfiniBand Network. The InfiniBand network is to provides a reliable high-speed interconnect for Database Machine. The InfiniBand network in described in greater detail later in this lesson. I addition, In dditi th the NET2 and d NET3 interfaces i t f on th the d database t b servers can b be optionally ti ll configured for additional networks. Note: Database Machine X2-8 is essentially the same except that each database server has more physical Ethernet ports and each database server also contains four bonded InfiniBand network interfaces connecting to the storage network.

Exadata and Database Machine Administration Workshop 8 - 12

InfiniBand Network Architecture •

Sun Datacenter 36-port managed QDR (40 Gb/s) switches – 2 leaf switches used to connect server InfiniBand ports – 1 spine switch in Full Rack and Half Rack configurations only

•

Exadata Servers and X2-2 Database Servers – Each server has one dual-port QDR (40 Gb/s) InfiniBand HCA – Each HCA port is connected to a different leaf switch for high availability – Active-passive bonding with single IP address —

•

Performance is limited by PCIe bus, so active-active not needed

X2 8 Database Servers X2-8 – Each server has four dual-port QDR (40 Gb/s) InfiniBand HCAs – Each pair of HCA ports are connected to different leaf switches for high availability – Active-passive bonding for each bonded pair

•

Connections are cabled in the factory y Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

InfiniBand Network Architecture Within Database Machine, the primary role of the InfiniBand network is to provide a reliable high-speed high speed interconnect to support database server to storage server communications communications. It is also used to facilitate communication and coordination between database servers using Oracle Clusterware and Oracle Real Application Clusters (RAC) software. It is important to note that the InfiniBand network is completely internal to Database Machine and there is no requirement for additional InfiniBand infrastructure within the customer’s network. Each Database Machine configuration uses at least two Sun Datacenter 36-port managed QDR (40 Gb/s) InfiniBand switches. Each database server and each Exadata server is connected t d to t both b th switches it h using i one or more bonded b d d network t k iinterfaces. t f T Two switches it h are used to provide high availability. If a switch fails, then the InfiniBand network can be maintained by the remaining switch. In such a case there would be no performance loss because each switch is capable of supporting the full system bandwidth. The switches that connect the database servers and Exadata servers within a rack are known as leaf switches. Full Rack and Half Rack Database Machine configurations contain a third InfiniBand switch, p switch. The spine p switch connects the 2 leaf switches and is also used to known as the spine connect multiple Database Machines. Quarter Rack configurations do not include a spine switch. Exadata and Database Machine Administration Workshop 8 - 13

X2-2 Leaf Switch Topology

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

X2-2 Leaf Switch Topology The diagram in the slide illustrates how the two leaf switches in each Database Machine X2-2 rack are connected to the database servers servers, Exadata servers and to each other other. Each database server and each Exadata server is configured with a bonded network interface (BOND0) which uses the two InfiniBand HCA ports on the server. Active/passive bonding is used. On each server, each InfiniBand port is connected to a different leaf switch. The active and passive connections are spread across the leaf switches to balance the workload. In addition, the leaf switches are connected to each other via seven inter-switch links. All the connections are pre-cabled in the factory. The topology described on this page is common to all Database Machine X2-2 configurations contained in a single rack; Quarter Rack, Rack Half Rack and Full Rack. Multiple rack configurations are discussed later in the lesson. Note: The switch port mappings shown in the diagram are illustrative only. Note: Database Machine X2-8 has a similar topology except that each database server has four bonded network interfaces with connections spread between the leaf switches.

Exadata and Database Machine Administration Workshop 8 - 14

Full Rack Spine and Leaf Topology

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Full Rack Spine and Leaf Topology The diagram in the slide shows the added spine switch which exists in Full Rack and Half Rack Database Machine configurations only. only As shown in the diagram diagram, each leaf switch is connected to the spine switch using a single network link. The purpose of the spine switch is to connect multiple Database Machines together, creating a single large-scale database grid.

Exadata and Database Machine Administration Workshop 8 - 15

Scale Performance and Capacity

Scalable • • •

Redundant and Fault Tolerant

Scale to eight g racks by y adding g cables Scale beyond eight racks by adding InfiniBand switches Scale to hundreds of storage servers to support multipetabyte databases

• •

Failure of any y component p is tolerated. Data is mirrored across storage servers.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Scale Performance and Capacity If you need to scale beyond the performance or capacity of a single Database Machine g yyou can scale up to eight g Full Racks by y simply y adding g some cabling g between configuration, them. There are enough free ports in the InfiniBand switches and enough bandwidth for eight Full Rack Database Machines running at full performance and with full redundancy. If you want to extend beyond eight racks, you need to add InfiniBand switches. In addition, Oracle now enables customers to interconnect a Half Rack Database Machine with other Half Rack or Full Rack configurations. Interconnecting Quarter Rack Database Machines is not supported.

Exadata and Database Machine Administration Workshop 8 - 16

Scaling Out to Multiple Full Racks •

Single InfiniBand network based on a Fat Tree topology – Up to 8 racks supported with existing switches – Database and storage server cabling unchanged

•

Two rack example:

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Scaling Out to Multiple Full Racks When multiple Full Rack Database Machines are connected to form a large-scale database grid,, the InfiniBand network is modified to resemble a Fat Tree topology. g p gy The database server and storage server connections to the leaf switches remain unchanged. In a standalone Full Rack Database Machine, each leaf switch has 8 links that connect it to the other leaf switch in the rack (7 links) and to the spine switch (1 link). When multiple full racks are interconnected, these 8 links are redistributed to the spine switches. The diagram in the slide illustrates this using a two rack example. As shown in the diagram, the seven interswitch links that originally connected the leaf switches in each rack have been redistributed so that each leaf switch has 4 links to each spine switch. There are no direct links between spine switches it h and d th there are no direct di t lilinks k b between t lleaff switches. it h To connect eight Full Rack Database Machines, the eight leaf-to-spine links on each leaf switch are configured to provide exactly one link to each spine switch. For configurations between 2 and 8 racks, the eight leaf-to-spine links on each leaf switch are distributed as evenly as possible amongst the total number of spine switches. Oracle supports interconnecting up to eight Full Rack Database Machines using the existing switches in each rack. Scaling beyond eight racks requires larger external switches. Note: Up to three Full Rack Database Machines can be interconnected using the cabling supplied from the factory. To interconnect more than three Database Machines, additional cabling must be purchased. Exadata and Database Machine Administration Workshop 8 - 17

Quiz Bonded interfaces are used for the InfiniBand storage network on Database Machine servers to provide: 1. Fault Tolerance and High Availability 2 Increased Bandwidth and Performance 2. 3. Both of the above

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 1 Each database server and Exadata server is configured with bonded network interfaces which g is used so the aggregate gg g use the InfiniBand HCA ports on the server. Active/passive bonding bandwidth of the links is not used simultaneously. For each bonded network interface, each InfiniBand port is connected to a different leaf switch. This configuration provides fault tolerance and high availability by removing the InfiniBand switch as a potential single point of failure.

Exadata and Database Machine Administration Workshop 8 - 18

Quiz Which of the following scalability options are supported? 1. Upgrading a Quarter Rack Database Machine to a Half Rack Database Machine 2 Upgrading a Half Rack Database Machine to a Full Rack 2. Database Machine 3. Interconnecting two Quarter Rack Database Machines 4. Interconnecting two or more Full Rack Database Machines

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 1, 2, 4 Interconnecting Database Machines is only supported for Half Rack and Full Rack configurations configurations.

Exadata and Database Machine Administration Workshop 8 - 19

Summary In this lesson, you should have learned how to: • Describe the architecture and various configuration options for Database Machine

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 8 - 20

D t b Database M Machine hi Configuration C fi ti

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives After completing this lesson, you should be able to: • Describe the installation and configuration process for Database Machine. • Describe the default configuration for Database Machine Machine. • Describe supported and unsupported customizations for Database Machine.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 9 - 2

Database Machine Implementation Overview Four phases: 1. Pre-installation – Various planning and scheduling activities including: — — —

Site planning: space, space power, power cooling, cooling logistics logistics… Configuration planning: hostnames, IP addresses, databases… Network preparation: DNS, NTP, cabling…

– Oracle and customer engineers work together

2. Installation and configuration – Hardware and software installation and configuration – Result is a working system based on a default configuration – Conducted by Oracle engineers

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine Implementation Overview The process of successfully implementing Database Machine requires cooperation and coordination between various Oracle and customer representatives and engineers engineers. At a high level the process involves four main phases: 1. Pre-installation involves various planning and scheduling activities including those listed in the slide. Typically, Oracle will guide the customer through this phase by collaborating with them in order to complete a key document known as the Configuration Worksheet. The Configuration Worksheet drives the installation and configuration phase (Phase 2). Once it is completed there are additional preparation tasks that the customer will t i ll complete. typically l t Thi This phase h iis d described ib d iin greater t d detail t il llater t iin th the llesson. 2. Installation and configuration is the process of taking a virgin Database Machine and installing it into the customer environment. It involves numerous steps conducted by hardware and software engineers. It is highly recommended that Oracle hardware and software engineers are used to conduct this phase because of the specialized nature of the tasks involved and to ensure that the Database Machine is configured in a standard and supportable way. The result of this phase is a working system which is configured according to the Configuration Worksheet. An example is shown later in the lesson.

Exadata and Database Machine Administration Workshop 9 - 3

Database Machine Implementation Overview 3. Additional configuration – Additional activities for production readiness including: — — — —

Reconfigure storage using non-default settings Create additional databases Configure Enterprise Manager Configure backup and recovery

– Conducted by customer or Oracle services engagement

4. Post-installation – Ongoing g g monitoring g and maintenance – Conducted by customer or Oracle services engagement

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine Implementation Overview (Continued) 3. After installation and configuration, the Database Machine is configured on the customer network Depending on the contents of the Configuration Worksheet, network. Worksheet the Database Machine may include one or more running databases. At this point customers can commence the process of loading or migrating data and prepare Database Machine for production use. However, customers may wish to vary the configuration of Database Machine in order to create customized databases or to organize the available storage in a non-default way. Such additional configuration tasks are typically conducted by the customer or Oracle services are engaged to conduct the work. Other examples of additional configuration tasks include configuring Enterprise Manager to assist with ongoing monitoring and maintenance, or the configuration of a backup and recovery environment. Common supported and unsupported additional configuration activities are discussed later in the lesson. 4. During post-installation the focus of activities shifts to on-going monitoring and maintenance of Database Machine. This is usually conducted by the customer. The remainder of this lesson focuses on topics that will assist you to understand some of the key activities in the implementation process. The aim is not to replace the available documentation, rather the material is presented to assist you to better understand the available configuration options and to better appreciate the results of making different selections. Exadata and Database Machine Administration Workshop 9 - 4

Configuration Worksheet Overview The Configuration Worksheet: • Is a Microsoft Excel spreadsheet with macros • Contains site-specific configuration settings including: – – – – – – – –

•

Host and domain names IP addresses Region and time zone information Name servers and NTP time servers Database server OS user and group settings E d t cellll notification Exadata tifi ti settings tti ASM diskgroup definitions Oracle database definitions

Generates a series of configuration files that drive the installation and configuration programs Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Configuration Worksheet Overview One of the most important Database Machine pre-installation tasks is to complete the Configuration Worksheet. Worksheet Completing the Configuration Worksheet is often a collaborative effort involving database, network and system administrators. Typically, Oracle engineers will participate to provide assistance. The information in the configuration worksheet is used to configure Database Machine. It also defines the network names and addresses that need to be configured within your network before installing Database Machine. The following section examines the Configuration Worksheet in greater detail.

Exadata and Database Machine Administration Workshop 9 - 5

Getting Started

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Getting Started The slide shows a brand new Configuration Worksheet. The values in the worksheet are all pre-defined pre defined defaults defaults. The worksheet version is displayed at the very top of the worksheet worksheet. Please check with your Oracle representative to ensure that you are using the latest available version. Notice that the Configuration Worksheet implies a set of conventions for naming the various components of Database Machine. For example, the Compute Node Hostname is derived by concatenating the DB Machine Prefix along with the Compute Node Base Name and then adding a two digit numeric identifier at the end. Notice also that the private InfiniBand network is configured by default to use addresses in the 192.168.10.X range. Because this network is self-contained within Database Machine, in most cases there is no requirement to change this setting. While the Configuration Worksheet is quite flexible and allows many opportunities for customization, it is highly recommended that customers should stick to the default conventions wherever possible. This not only reduces the possibility for errors or misconfigurations but it also helps to ensure that the resulting configuration is easier to misconfigurations, support.

Exadata and Database Machine Administration Workshop 9 - 6

Configuration Worksheet Example

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Configuration Worksheet Example The slide shows an example of the first part of a completed Configuration Worksheet. The following slide shows the rest of the example example. The example illustrates a typical configuration using as many default values as possible. In this part of the Configuration Worksheet the following fields are set to non-default values: • Customer Name: Setting the Customer Name to Example Industries has no effect on the configuration of Database Machine. The field simply helps to identify this instance of the Configuration Worksheet. • DB Machine Prefix: DB Machine Prefix is an identifier used to generate host names for network interfaces for all the systems. Indirectly, this value is also used in other identifiers such as cell disk and grid disk names. Oracle recommends that this value should be 4 characters or less to avoid possible issues with names that are too long. In this example, eidm is an abbreviation of Example Industries Database Machine. Customers should choose a setting that reflects their own naming conventions. • dbMachine Model: Select a value from the drop-down list to reflect the environment being be g co configured. gu ed Sett Setting g this t s field e d modifies od es the t e rest est of o the t e Configuration Co gu at o Worksheet o s eet so that the correct number of database servers and Exadata cells are configured.

Exadata and Database Machine Administration Workshop 9 - 7

Configuration Worksheet Example (Continued) • • • •

•

•

•

Domain Name: This field is set to the domain name of the management network where the Database Machine resides. Region: This field is set to the geographic region where the database machine resides resides. Time Zone: This field is set to the time zone where the database machine resides. NameServer: Use these fields to identify up to four DNS servers in the network where the Database Machine resides. While it is possible to implement Database Machine without DNS, Oracle recommends that DNS should be used in conjunction with Database Machine. NTP Time Server: Use these fields to identify up to four NTP time servers in the network where the Database Machine resides. NTP services are a mandatory requirement for Database Machine. NTP provides coordinated timing which synchronizes services across Database Machine. Without NTP, a lack of coordination can lead to database nodes being evicted from the cluster or Exadata cells being excluded from the storage pool. NTP services also ensure that the timestamps written to the various log files are coordinated across Database Machine. Note that the Cluster Time Synchronization Service (CTSS) introduced in Oracle Clusterware11g Release 2 cannot be used to provide time services to Exadata cells, hence it is not used for Database Machine. First Management/ILOM IP address [eth0]: Set this field to the first IP address in the range of IP addresses allocated for the management network. When this field is set, the Configuration Worksheet sequentially allocates IP addresses for each network interface as shown in the Specific Network Information section of the example worksheet. You can change the default addressing policy by changing values in the Specific Network Information section of the worksheet or you can change individual addresses by modifying the addresses generated after you click the Generate button (discussed later in the lesson). Beware that if you modify the default addressing policy you must ensure that the resulting addresses are valid and there are no duplicates. First Client Access IP address [eth1]: Set this field to the first IP address in the range of IP addresses allocated for the client access network. When this field is set, the Configuration Worksheet sequentially allocates IP addresses for each network interface as shown in the Specific Network Information section of the example worksheet. You can change the default addressing policy by changing values in the Specific Network Information section of the worksheet or you can change individual addresses by modifying the addresses generated after you click the Generate button (discussed later). Beware that if you modify the default addressing policy you must ensure that the resulting addresses are valid and there are no duplicates.

In this example all the other fields contain default values. Other fields that might typically be set to non-default values include: • Cell Disk Size: Set this field to 600G for systems with high performance disks or 2TB for systems with high capacity disks. • Backup requirements: Select from one of the available options (Tape backup, Disk backup only, Both Disk and Tape backup). This setting influences the default size of the ASM disk groups that are configured later in the worksheet. • Subnet Mask: If the default values are not suitable, the various subnet mask fields should be adjusted to reflect the network where the Database Machine resides. • Gateway: If the default values are not suitable, the various gateway fields should be adjusted to reflect the network where the Database Machine resides. Exadata and Database Machine Administration Workshop 9 - 8

Configuration Worksheet Example

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Configuration Worksheet Example (Continued) Following on from the previous slide, this slide shows an example of the second portion of a completed Configuration Worksheet Worksheet. In this part of the Configuration Worksheet the following fields are set to non-default values: • Email To Notification Address, Email From Notification Address, Email From Name, Email Server, Notify via Email: This group of fields are set to facilitate email notification of Exadata cell alerts. Depending on the email server setup you may also need to set other fields in the SMTP and SNMP Details section of the worksheet. Exadata cell alerts can also be delivered via SNMP however this option is not configured in the example. • Number of Diskgroups: Set this field to a non-zero value to configure ASM disk groups. By default, a setting of 2 results in the creation of disk groups named DATA and RECO as shown in the slide. This section of the Configuration Worksheet is described in greater detail on the next slide. • Number of Databases: Set this field to a non-zero value to create databases on the Database Machine. The example in the slide shows the default settings for one database database.

Exadata and Database Machine Administration Workshop 9 - 9

Configuration Worksheet Example (Continued) Note the limited set of database configuration attributes that are available in the Configuration Worksheet. Using the Configuration Worksheet you cannot set many common database attributes, such as database character set for example. The default database created during installation uses Unicode database character set AL32UTF8. Oracle recommends Unicode for all new deployments. To use a different database character set after installation, or to create databases with other non-default attributes, run Database Configuration Assistant (DBCA) to delete the default database, and to create a new database using the desired settings. Note also that if you do not specify any databases in the Configuration Worksheet, the configuration programs will not install the Oracle Database software on the Database Machine. Therefore it is highly recommended that you specify at least one database in the Configuration Worksheet, which can be deleted later if it is not required.

Exadata and Database Machine Administration Workshop 9 - 10

Configuring ASM Disk Groups with Configuration Worksheet

System Area (Disks 0-1) or SYSTEMDG (Disks 2-11) (29G)

Exadata Cell 1 Exadata Cell 2 Exadata Cell n

DATA (220G)

RECO (ALL ~ 309G)

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Configuring ASM Disk Groups with Configuration Worksheet The slide illustrates how ASM disk groups are configured using the Configuration Worksheet. By d B default, f lt ASM di disk k groups configured fi d using i th the C Configuration fi ti W Worksheet k h t are striped t i d across every disk in every Exadata cell in the Database Machine. The worksheet allows you to specify a range of cells to limit the striping to all the disks those cells. However, you cannot specify a subset of disks within Exadata cells using the Configuration Worksheet. The policy of striping disk groups across all the available disks is recommended by default. Deviating from this policy is recommended only where data isolation is required; such as in circumstances where security policies govern that data in different databases must be stored on physically h i ll separate t di disks. k Regardless of how many disk groups are specified in the Configuration Worksheet, a disk group named SYSTEMDG is always created. The primary purpose of the SYSTEMDG disk group is to store the shared clusterware files; Oracle Cluster Registry (OCR) and voting disks. The SYSTEMDG disk group occupies a grid disk that consumes approximately 29 Gigabytes on every disk except the first two disks in each cell. On the first two disks (numbered 0 and 1) in pp y 29 Gigabytes g y is used to store the System y Area,, which contains the each cell,, approximately OS image, swap space, Exadata software binaries, metric and alert repository and various other configuration and metadata files. Exadata and Database Machine Administration Workshop 9 - 11

Configuring ASM Disk Groups with Configuration Worksheet (Continued) Because of the layout of the System Areas and the SYSTEMDG disk group, the remaining space on every disk is equal. As shown in the slide slide, specifying two disk groups in the Configuration Worksheet results in a configuration with disk groups named DATA and RECO. The disk group names can be modified. The amount of space allocated to each disk group on each disk can also be set. The default size values are provided as a guide only and are influenced by Backup Requirements setting which is near the top of the worksheet. In this example the default sizes are used, so the DATA disk group occupies a grid disk that is allocated 220 Gigabytes on each cell disk, and the RECO disk group occupies another grid disk that is allocated the remaining space (approximately 309 Gi b t assuming Gigabytes i hi high h performance f di disks) k ) on each h di disk. k Specifying only 1 disk group results in a disk group named DATA which contains all of the nonsystem disk space on each disk (approximately 529 Gigabytes assuming high performance disks). If you wish to specify 3 or more disk groups, the first two disk groups will be named DATA and RECO by default and you will need to supply names for the additional disk groups. You will also need to specify the per-disk per disk space allocation for each disk group group. Note that the grid disks that are created to allocate disk group space on each cell disk are created in the order in which the associated disk groups appear in the Configuration Worksheet. Because of this ordering, it is recommended that you list the disk groups with the highest performance requirements first in the Configuration Worksheet. Note also that the values generated by the worksheet under the heading Estimated Size of DG represent the approximate space available for user data factoring in the default 2-way mirroring policy (ASM NORMAL redundancy) that is applied during disk group creation. The Estimated Size of DG values also factor in the loss of one Exadata cell. For example, in the example worksheet the Estimated Size of DG value for the DATA disk group is approximately 2.6 TB, however the actual disk space allocated to the disk group is approximately 7.9 TB (220 GB x 12 disks per cell x 3 cells) and the space available for user data is approximately 3.9 TB after 2-way mirroring is factored in.

Exadata and Database Machine Administration Workshop 9 - 12

Generating the Configuration Files

1 2

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Generating the Configuration Files After the colored fields in the Configuration Worksheet are set, click the Generate button to generate individual host names and IP addresses in the area below the colored fields. An g example is shown in the slide. You can customize the entries in this area of the worksheet to set specific host names and IP addresses, however it is recommended that the default settings should be used wherever possible to minimize the risk of errors or misconfigurations. If you change one of the colored fields in the Configuration Worksheet you must click the Generate button again for the change to be reflected in the generated section. After you are satisfied with the generated settings, click the Create Config Files button to generate the configuration files that are used to drive the Database Machine installation and configuration programs programs. A series of files is written to a directory named according to the following convention: c:\dbmachine_\\ While you can examine the entries in the generated files it is not recommended that the files are modified directly because doing so may result in errors during the execution of the installation and configuration programs. If you make a change to the Configuration Worksheet after creating the configuration files, you must re-create the configuration files for the change to be incorporated in them them.

Exadata and Database Machine Administration Workshop 9 - 13

Other Pre-Installation Tasks •

Network Preparation – Configure Domain Name Service (DNS) server – Configure Network Time Protocol (NTP) server

•

Physical Preparation – – – – –

Provide adequate cooling capacity Provide proper ventilation and airflow Provide adequate floor space with adequate floor loading Provide required power outlets Provide required network drops

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Other Pre-Installation Tasks Once the Configuration Worksheet is complete there are a series of important tasks that must be completed before the installation of Database Machine. Machine These can be logically grouped into two main areas; network preparation and physical preparation. To prepare your network for Database Machine configure DNS to include the host names and IP addresses allocated in the Configuration Worksheet. While the use of DNS is not mandatory, it is highly recommended. If one is not already available, you must configure a NTP service for use in conjunction with Database Machine. NTP configuration is mandatory because it is used to synchronize the InfiniBand switches, not just the Oracle Clusterware software. ft Physical preparation involves provisioning your data center to accept Database Machine. The main provisioning tasks are listed in the slide. Your Oracle representative will provide specific requirements based on your proposed Database Machine configuration and location.

Exadata and Database Machine Administration Workshop 9 - 14

The Result After Installation and Configuration eidm01 172.16.1.101 eidm01-vip

eidm02 172.16.1.102

172.16.1.103 eidmdb01 10.7.7.101

eidm02-vip eidmdb01-ilom 10.7.7.106

172.16.1.104 eidmdb02 10.7.7.102

DB Server 1

eidmdb02-ilom 10.7.7.107

eidm-scan 172.16.1.105 172.16.1.106 172.16.1.107

DB Server 2

DB Instance: dbm1

DB Instance: dbm2

ASM Instance: +ASM1

ASM Instance: +ASM2

Cluster: eidm

Cluster: eidm eidmdb02-priv 192.168.10.2

eidmdb01-priv 192.168.10.1

InfiniBand Storage Network eidmcel02-priv 192.168.10.4

eidmcel01-priv 192.168.10.3

eidmcel03-priv 192.168.10.5

873GB total

SYSTEMDG

SYSTEMDG

SYSTEMDG

DATA

DATA

DATA

7,920GB total

RECO

RECO

RECO

11,112GB total

Exadata Server 1

Exadata Server 2

Exadata Server 3

eidmcel01 10.7.7.103

eidmcel02 10.7.7.104

eidmcel03 10.7.7.105

eidmcel01-ilom 10.7.7.108

eidmcel02-ilom 10.7.7.109

eidmcel03-ilom 10.7.7.110

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

The Result After Installation and Configuration The diagram in the slide illustrates the result after installation and configuration based on the Configuration Worksheet example shown earlier in this lesson. lesson The host names and IP addresses are based on the Configuration Worksheet settings. Note that each host name starts with the DB Machine Prefix (eidm). Note also the default IP address allocation scheme. By default, the storage is configured symmetrically. That is, the storage in each Exadata cell is allocated in the same way. On each storage server: • A cell disk is defined for each physical disk. The cell disks use the default naming convention of CD__ where is a two digit identifier in the range 00-11 and is the canonical host name for the storage server. For example, CD_01_eidmcel01. • On every cell disk, apart from the first two (=00 or 01), a grid disk with the prefix SYSTEMDG is created. These grid disks are each approximately 29 GB in size. These grid disks are allocated to the SYSTEMDG disk group. The 10 grid disks on each Exadata server total approximately 291 GB, GB adding up to 873 GB across all three storage servers in the example Quarter Rack configuration.

Exadata and Database Machine Administration Workshop 9 - 15

The Result After Installation and Configuration (continued) •

•

On every cell disk, a grid disk with the prefix DATA is created and another grid disk with the prefix RECO is created. These grid disks are allocated to the DATA and RECO disk groups p y Each DATA g grid disk is 220 GB which adds up p to 2640 GB p per cell ((12 x 220 respectively. GB) or 7920 GB across the Quarter Rack Database Machine. Likewise, each RECO grid disk is approximately 309 GB, totalling 11112 GB across the Quarter Rack Database Machine. All the available flash storage is configured as Exadata Smart Flash Cache.

The Database Servers are also configured symmetrically by default. Each database server is installed with the same operating system configuration, including the same Oracle user account and group definitions. The database servers are configured as a cluster under the control of Oracle Clusterware software. An ASM cluster is also configured. Finally, Oracle Database software is installed on each database server and a Real Application Clusters (RAC) database is established across all the cluster nodes. Note the following particulars regarding the installation and configuration of Oracle Grid Infrastructure and Oracle Database on Database Machine. • Grid Naming Service (GNS) is not configured configured. DNS is the preferred naming service for Database Machine. • RAC databases created as part of the installation and configuration process are administrator-managed by default. To create a policy-managed RAC database on Database Machine you must manually create the database using the Database Configuration Assistant (DBCA). • By default, Database Control is not configured for databases created by the installation and configuration process.

Exadata and Database Machine Administration Workshop 9 - 16

Supported Additional Configuration Activities •

Earthquake protection using a 3rd party Seismic Isolation Platform

• • • •

Replace the Ethernet switch Connect a tape library for backup Create and configure additional databases Configure database features – Oracle Data Guard – Database File System (DBFS)

• •

Configure Enterprise Manager Customize the storage configuration; ASM and Exadata Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Supported Additional Configuration Activities The slide lists some typical configuration activities which customers can undertake using their own labor and at their own expense expense. The following points expand on some of these: • Customers are permitted to implement earthquake protection with a 3rd party Seismic Isolation Platform as long as the Database Machine is not physical altered or re-racked. • Customers are permitted to replace the Cisco Ethernet switch with an equivalent switch of their choice. Alternatively, customers can implement 3rd party gateway switches to isolate Database Machine from other components on the network. • Customers are p permitted to connect additional servers or devices to Database Machine via Ethernet or InfiniBand. Typically, customers require such connections to connect their backup and recovery infrastructure or to facilitate data transfer from other systems.

Exadata and Database Machine Administration Workshop 9 - 17

Unsupported Configuration Activities • • • • • •

Hardware re-racking Adding components to servers Adding servers to Quarter Rack or Half Rack configurations Interconnecting Quarter Rack configurations Swapping Linux distributions Configuring ACFS

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Unsupported Configuration Activities The following configuration activities are not supported with Database Machine: • Hardware re re-racking: racking: Customers sometimes wish to re re-rack rack a Database Machine to comply with a data center policy, to achieve earthquake protection or to overcome a physical limitation of some sort. Apart from being inherently error-prone, re-racking can cause component damage, thermal management issues, cable management issues and other issues. As a result, hardware re-racking of Database Machine is not supported. • Adding components to servers: Customers sometimes wish to add components to Database Machine. A typical example is the desire to add a Host Bus Adapter (HBA) to the database servers so that they can be attached to existing SAN storage. Adding components to servers in Database Machine is not supported because of the potential for driver and firmware incompatibilities that could undermine the system. • Adding servers to Quarter Rack or Half Rack configurations: Outside of a supported upgrade, physically adding servers to a Quarter Rack or Half Rack is not supported because it changes the environmental and power characteristics of the system. It also impacts pacts on o the t e future utu e ability ab ty to conduct co duct a suppo supported ted upg upgrade. ade Note ote tthat at custo customers e s ca can add Exadata cells to any Database Machine configuration by placing the additional cells in a separate rack and by using the spare InfiniBand ports to connect to them. Exadata and Database Machine Administration Workshop 9 - 18

Unsupported Configuration Activities (continued) •

•

•

Interconnecting Quarter Rack configurations: Interconnecting Quarter Rack configurations has not been tested and certified by Oracle. In addition, such configurations cannot be replicated by Oracle Support in order to analyze and remedy problems. Swapping Linux distributions: Oracle Linux is provided as the operating system underpinning the database servers and Exadata servers in Database Machine. Swapping Linux distributions is not supported. Configuring ACFS: ASM Cluster File System (ACFS) is currently unavailable on Database Machine.

Note: Consult yyour Oracle representative p to determine whether other configuration g activities are supported or not.

Exadata and Database Machine Administration Workshop 9 - 19

Quiz Using the Configuration Worksheet you can set specific IP addresses for each database server and Exadata server in Database Machine: 1. True 2. False

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 1 You can set individual IP addresses by modifying the addresses generated after you click the g Worksheet. Beware that if yyou modify y the default Generate button in the Configuration addressing policy you must ensure that the resulting addresses are valid and there are no duplicates. Note that while the Configuration Worksheet is quite flexible and allows customers many opportunities for customization, it is highly recommended that customers should stick to the default conventions wherever possible. This not only reduces the possibility for errors or misconfigurations, but it also helps to ensure that the resulting configuration is easier to support.

Exadata and Database Machine Administration Workshop 9 - 20

Quiz Which of the following options for connecting to existing SAN storage is supported in conjunction with Database Machine? 1. Install a fiber channel HBA into each database server 2 Use a server connected to the existing SAN as a storage 2. gateway and connect it to Database Machine using NFS over Ethernet 3. Use a server connected to the existing SAN as a storage gateway and connect it to Database Machine using NFS over InfiniBand

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 2, 3 Connecting Database Machine to other servers using Ethernet or InfiniBand is supported. Adding g components to Database Machine servers is not supported.

Exadata and Database Machine Administration Workshop 9 - 21

Summary In this lesson, you should have learned how to: • Describe the installation and configuration process for Database Machine. • Describe the default configuration for Database Machine Machine. • Describe supported and unsupported customizations for Database Machine.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 9 - 22

Additional Resources •

Lesson Demonstrations (Viewlets) – Database Machine Configuration Example using Configuration Worksheet —

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/091DBMConfig Worksheet/091dbmconfigworksheet_viewlet_swf.html

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 9 - 23

Mi Migrating ti Databases D t b to t Database D t b M Machine hi

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives After completing this lesson, you should be able to: • Describe the general steps to migrate your database to Database Machine • List the main approaches for migrating your database to Database Machine • Identify the most appropriate approach for different circumstances

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 10 - 2

Migration Best Practices Overview

Legacy storage to Database Machine

1

2

3

4

Capacity planning

Choose migration approach

Migration

Post-migration tasks

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Migration Best Practices Overview The slide depicts the four important phases to migrate your existing database to Database Machine. Each phase is discussed in the remainder of this lesson.

Exadata and Database Machine Administration Workshop 10 - 3

Performing Capacity Planning Exadata Storage Server

Source database

AWR Report

Size

IOPS

MBPS

… CALIBRATE

High Performance Disk

High Capacity Disk

Exadata Smart Flash Cache

384 GB

384 GB

Raw Disk Capacity

7.2 TB

24 TB

Uncompressed Data Capacity

Up to 2 TB

Up to 7 TB

Raw Disk Throughput (MBPS)

Up to 1 1,800 800

Up to 1 1,000 000

Effective Throughput with Flash (MBPS)

Up to 3,600

Up to 3,600

Effective Throughput with Flash and Compression (MBPS)

Up to 36,000

Up to 36,000

Disk I/Os per Second (IOPS)

Up to 3,600

Up to 1,440

Flash I/Os per Second (IOPS)

Up to 75,000

Up to 75,000

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Performing Capacity Planning Before migrating to Database Machine, you must understand the I/O characteristics of your g of yyour current system. y The key y measures current environment. Collect the size and throughput of I/O throughput are I/Os per second (IOPS) and megabytes per second (MBPS). Use the system statistic ‘physical I/O disk bytes’ to derive the current MBPS for the system. Use the system statistics ‘physical reads’ and ‘physical writes’ to determine the current IOPS for the system. These statistics are available in an Automatic Workload Repository (AWR) report. After you understand the capacity of your current system, you can determine the required number of Exadata cells, and from there determine the appropriate Database Machine configuration It is important to size both for performance and capacity. configuration. capacity One Exadata server configured with 12 high performance drives provides approximately 2 TB of mirrored uncompressed usable storage, 1,800 MBPS throughput for large I/Os, and 3,600 IOPS for small I/Os, however much greater effective throughput can be achieved by making efficient use of Exadata Smart Flash Cache and Exadata Hybrid Columnar Compression. Verify that the cell provides the expected throughput by running the CALIBRATE command. It is also important to consider failures when planning capacity. Exadata cell and disk failures are transparently l tolerated l d using i A Automatic i S Storage M Management (ASM) redundancy. d d H However, it is a best practice to ensure that post-failure I/O capacity is sufficient to meet the redundancy requirements and performance service levels. Exadata and Database Machine Administration Workshop 10 - 4

Database Machine Migration Considerations •

Platform is Oracle Linux Release 5 x86_64 – Byte order is little endian

• •

Exadata, ASM and database software must run release 11.2.0.1 or greater. ASM disk group attributes: – – – –

•

COMPATIBLE.ASM=11.2.0.0.0 COMPATIBLE.RDBMS=11.2.0.0.0 CELL.SMART_SCAN_CAPABLE=TRUE AU SIZE 4M AU_SIZE=4M

Database initialization parameters: – COMPATIBLE=11.2.0.0.0 – DB_BLOCK_CHECKSUM=TYPICAL|FULL

•

Database extent sizes should be a multiple of 4 MB. Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine Migration Considerations Following are the configuration requirements and recommendations for Database Machine that you should bear in mind when planning your database migration: • Database Machine is based on Oracle Linux Release 5 x86_64. The byte order for Oracle Linux x86_64 is little endian. • The Exadata, ASM and database software on Database Machine must be release 11.2.0.1 or later. • For ASM disk groups containing Exadata grid disks, specify the attribute settings shown in the slide. Although these are not mandatory, they are highly recommended in order to enable all of the Exadata features and to optimize performance performance. • To enable all the Exadata features in your database, your Oracle Database software must be release 11.2.0.1 or later, and you must set the COMPATIBLE initialization parameter as shown in the slide. Because Exadata storage is Hardware Assisted Resilient Data (HARD) compliant, you should set DB_BLOCK_CHECKSUM to TYPICAL or FULL to ensure that database checksums are computed and stored in the blocks. Note that this initialization parameter is set to TYPICAL by default. Exadata performs best when scanning contiguous chunks at least 4 MB in size. To ensure this occurs, the ASM disk group allocation unit size (AU_SIZE) should be set to 4 MB and the database extent sizes should be a multiple of 4 MB. Note: The parameters in red in the slide are required to enable smart storage capabilities. Exadata and Database Machine Administration Workshop 10 - 5 •

Choosing the Right Migration Path •

Determine what to migrate – Avoid methods that migrate what you will discard

•

Consider the configuration of the source system – Source Oracle Database version and platform matters – Target system is fixed: 11.2, ASM, Linux x86_64

•

Weigh up the costs and favor methods that facilitate best practices – Implementing best practices is important in the long term because your future performance depends on it – Remember: — —

ASM 4MB AU size can only be set at disk group creation Large extents (8MB) are set at extent allocation

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Choosing the Right Migration Path Migrating an existing database to Database Machine can be achieved using a number of different approaches. The recommended approaches are discussed later in this lesson. Before considering id i th them, it iis worthwhile th hil tto contemplate t l t some ffactors t th thatt can h help l iin choosing h i th the right migration path. Before choosing a migration approach, you should clearly define what you want to migrate. This will help you to avoid wasted effort such as migrating data which is not required. Clearly defining the scope of the migration also helps you to clearly identify the source systems. You need to understand the source systems because their composition may limit the available migration options. For example, Database Machine is a little endian platform, so if you are migrating from a big endian platform there are a number of physical migration approaches which become unavailable. When choosing the right migration approach, you should weigh up the short term requirement to perform the migration with the long term impact of using the selected migration approach. Specifically, you should favor methods that facilitate the implementation of best practices and avoid methods that hinder your ability the implement best practices. For example, Exadata performs best with an ASM AU size of 4 MB and database extents that are a multiple of 4 MB. If the source database extent sizes are not a multiple of 4 MB and it is impractical to reorganize the database before migration, then you should favor a migration approach that allows you to reorganize the database during the migration. If you choose an approach that does not allow the extents to be reorganized, you may be able to deliver a quicker and easier migration however you may also end up paying an ongoing performance penalty. Exadata and Database Machine Administration Workshop 10 - 6

Logical Migration Approaches Migration approach

Source database must be release 11.2

Can adjust ASM AU size to 4MB

Source platform support

Time and effort requirement

Migration outage time

L i l standby Logical t db d database t b

N No

Y Yes

Li it d Limited

Hi h High

L Low

No

Yes

Broad

High

Low

No

Yes

Broad

Low

High

Minimal down time

Streams Minimal down time

Data Pump For migration during a planned maintenance period

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Logical Migration Approaches Migration to Database Machine can be done logically or physically. Using a logical approach, approach you can change the database extent size and alter other physical characteristics of your database, which is not possible using a physical migration approach. The table in the slide summarizes some logical migration approaches: • Logical standby database: If your application service-level agreements permit little or no down time, you can use an Oracle Data Guard logical standby database to replicate the database on Database Machine, and track and merge the changes while the source database continues to run. After the configuration is established, the logical standby database (on Database Machine) can be switched to assume the role of primary database and the original source database can be decommissioned. The procedure documented in My Oracle Support bulletin 737460.1 can be used to change physical storage attributes of the database, such as segment extent sizes. Refer to My Oracle Support bulletin 1085687.1 for information on support for heterogeneous platform support in the same Data Guard configuration.

Exadata and Database Machine Administration Workshop 10 - 7

Logical Migration Approaches (continued) •

•

Streams: If your application service-level agreements permit little or no down time, you can also use Oracle Streams to propagate the data, and to track and merge the g while the source database continues to run. For more information about this changes approach, see Appendix D in Oracle Streams Concepts and Administration 11g Release 2 (11.2). Streams provides broader source database platform support than the logical standby database approach. Data Pump: If a suitable maintenance window can be accommodated, use Data Pump to move the data in bulk from the legacy system to Database Machine. Data Pump is easy to use and provides broad support across different platforms and database versions. versions

Note: Logical Standby Database and Streams both have additional considerations which may contradict their use. Firstly, both approaches are unable to natively handle all Oracle Database data types. While there are methods to overcome this limitation, the extra effort required to implement and maintain these methods should not be overlooked. Also, both approaches will fail to duplicate NOLOGGING operations that are conducted on the primary database.

Exadata and Database Machine Administration Workshop 10 - 8

Physical Migration Approaches Migration approach

Source database must be 11.2

Can adjust ASM AU size to 4MB

Source platform support

Time and effort requirement

Migration outage time

Yes

No

Linux x86_64 only y

Medium

Low

Yes

Yes

Linux x86_64 only

Medium

Low

Yes

Yes

Linux x86_64 only

Medium

Low

No

Yes

Limited

Low

Low

Yes

Yes

Little endian

Low

Medium

No

Yes

Broad

Medium

Medium

ASM Online Migration Best when already using ASM and 4 MB AU's

Recovery Manager Tablespace level migration

Create new tablespaces In conjunction with rolling partitions

Physical standby database Migration of DB server and storage

Transportable Database Migration to a different platform with the same endian format

Transportable Tablespace Migration to a different platform with different endian format

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Physical Migration Approaches The table in the slide summarizes some physical migration approaches: • ASM Online Migration: This method is only applicable if your database is already using ASM and you do not need to adjust the ASM allocation unit (AU) size. To use this method you must also be able to connect your current database storage to Database Machine and migrate the database instances to Database Machine. After migrating the database instances, the data migration is very simple, just add new Exadata-based grid disks to your ASM disk groups and drop existing disks from your ASM disk groups to migrate the data. • Recovery Manager: This method is useful if your source database does not use ASM and you are already using the Linux x86_64 platform for your current database server. To use this method you must be able to connect your current database storage to Database Machine and migrate the database instances to Database Machine. Then you essentially migrate the database into ASM by creating RMAN backups of the source database files and using the RMAN SWITCH...TO COPY command to perform the migration. See Chapter 8 of the Oracle Database Storage Administrator's Guide 11g Release 2 (11.2) for f rther information about further abo t using sing RMAN for ASM migrations migrations.

Exadata and Database Machine Administration Workshop 10 - 9

Physical Migration Approaches (continued) •

•

•

•

Create new tablespaces: This method is useful if you are already using the Linux x86_64 platform for your current database server and you use partition roll-in and rollg data in and out of the database. To use this method yyou must be out to naturallyy age able to connect your current database storage to Database Machine and migrate the database instances to Database Machine. After migrating the database instances, you create new Exadata-based tablespaces and create your new data partitions in the new tablespaces. Over time more of your database is housed on Database Machine. To migrate the remainder of your database to Database Machine you need to employ other methods such as ASM Online Migration or Recovery Manager. Physical standby database: Create a physical standby database on Database Machine and perform a Data Guard switchover to migrate. See My Oracle Support bulletin 413484.1 for information about heterogeneous platform support. If the source database version is earlier than 11.2 you will need to use the transient logical rolling database upgrade feature. My Oracle Support bulletin 413484.1 also contains additional information on this feature. Transportable Database: Use the Transportable Database feature to migrate the entire ti database. d t b T To use thi this method th d iin conjunction j ti with ith D Database t b M Machine, hi th the source database must be on a little endian platform. See the Platform Migration Using Transportable Database white paper at http://www.otn.oracle.com/goto/maa for more information about Transportable Database. Transportable Tablespaces: Use the Transportable Tablespace feature to migrate tablespaces from your current system to a new database hosted on Database Machine. This is the onlyy physical y database migration g method that provides broad platform support and supports migration from earlier Oracle Database versions.

Note: Database segments created using Oracle Database versions prior to 11g may not align properly on ASM extent boundaries. For data files copied from pre-11g databases, this misalignment can cause a single I/O request issued by the database to result in two physical I/Os split across an ASM extent boundary, and possibly across two physical disks. Because this behavior is most evident when large contiguous I/Os are issued, the impact is likely to be low for OLTP workloads workloads. However, However for workloads that contain many full table scans, the impact can be significant. To create the most efficient extent alignment, data must be loaded into new segments. This can be achieved using a logical migration approach, or using a CREATE TABLE ... AS SELECT command to re-create the segments into a new tablespace after a physical migration.

Exadata and Database Machine Administration Workshop 10 - 10

Other Approaches •

Data Integration Tools – Oracle GoldenGate – Oracle Data Integrator

•

Custom Code – Query over database link – PL/SQL routines

•

Hybrid Approaches – For example, Transportable Tablespaces from current production database to staging database outside Database Machine, then Data Pump to unload from staging database and load into Database Machine

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Other Approaches The previous slides introduced a number of migration approaches based on tools and technologies g that are p part of Oracle Database. In addition,, there are many y other migration g technologies and approaches including: • Data integration technologies such as Oracle GoldenGate or Oracle Data Integrator can be used to migrate, and if necessary transform data. Typically, such environments provide a high degree of flexibility along with a productive way of defining and implementing the required data integrations and transformations. Also, these technologies are often used to facilitate data migration while the source system continues to run. • In some cases, customers have specific circumstances that result in the use of custom code for data migration. This approach typically uses queries over database links as the main data transportation mechanism. These distributed queries are sometimes encapsulated in PL/SQL routines. Custom code is often used in situations where existing routines can be easily repurposed to perform a migration to Database Machine. • There are also many potential hybrid approaches that combine two or more of the methods introduced in this lesson. For example, a customer may already use Transportable Tablespaces in their environment so using them as a data source for a migration would be easy and convenient. However, the physical organization of the data in the Transportable Tablespaces may not be ideal for Database Machine so they may choose to re-load the data in fresh segments using Data Pump. Exadata and Database Machine Administration Workshop 10 - 11

Post-Migration Best Practices •

Check ASM disk groups are balanced: – Manual rebalance may be needed. – Script available at My Oracle Support bulletin 367445.1 – Enterprise Manager alert for disk group imbalance

•

Assess index requirements: – With Exadata, full scans might deliver acceptable performance – Determine indexes that are not required and remove them

•

Configure I/O Resource Management

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Post-Migration Best Practices Perform the following tasks after completing the migration: • One of ASM ASM’s s functions is to ensure that data is evenly distributed across all disks in a disk group. This happens automatically, but occasionally a disk group may become imbalanced due to uncommon errors, such as a failed rebalance. It is, therefore, an operational best practice to check disk groups on a regular basis and run a manual rebalance if necessary. A script is available to check disk group balance in My Oracle Support bulletin 367445.1. Also, Enterprise Manager Grid Control displays an alert if a disk group becomes unbalanced beyond a customizable threshold. • The superior scan rates available from Database Machine make it possible that indexes indexes, previously required for good performance, are no longer required. You should assess execution plans that use indexes to see if they would run acceptably with Smart Scans. To determine if queries would perform acceptably without an index, you can make the index invisible to the optimizer. An invisible index is maintained by DML operations, but it is not to be used by the optimizer for queries. To make an index invisible, use the following command: ALTER INDEX INVISIBLE; •

After you perform the above tasks, you can configure I/O Resource Management (IORM). Exadata and Database Machine Administration Workshop 10 - 12

Quiz What are two important best practices when migrating to Exadata storage? 1. Double the size of your SGA. 2 Configure ASM allocation units to 4 MB 2. MB. 3. Double the size of your disk space. 4. Configure database extents to multiples of 4 MB.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 2, 4

Exadata and Database Machine Administration Workshop 10 - 13

Quiz Which of the following physical migration approaches is most universally applicable? 1. ASM Online Migration 2 Physical Standby Database 2. 3. Transportable Tablespaces

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 3 Using transportable tablespaces you can migrate to Database Machine from a much broader g of platforms than either of the other approaches. range

Exadata and Database Machine Administration Workshop 10 - 14

Summary In this lesson, you should have learned how to: • Describe the general steps to migrate your database to Database Machine • List the main approaches for migrating your database to Database Machine • Identify the most appropriate approach for different circumstances

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 10 - 15

Additional Resources •

Lesson Demonstrations (Viewlets) – Migrating to Database Machine Using Transportable Tablespaces —

•

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/101DBMMigration TTS/101dbmmigrationtts viewlet swf html TTS/101dbmmigrationtts_viewlet_swf.html

My Oracle Support Notes – Changing Storage Definition in a Logical Standby Database —

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT &id=737460.1

– Data Guard Support for Heterogeneous Primary and Standby Systems in Same Data Guard Configuration —

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT htt // t l /CSP/ i / ti l ? d h &t NOT &id=413484.1

– Script to Report the Percentage of Imbalance in all Mounted Diskgroups —

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT &id=367445.1

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 10 - 16

Additional Resources •

Documentation and White Papers – Oracle Streams Concepts and Administration 11g Release 2 (11.2) —

http://www.oracle.com/pls/db112/to_toc?pathname=server.112/ e10704/toc.htm

– Oracle Database Storage Administrator's Guide 11g Release 2 (11.2) —

http://www.oracle.com/pls/db112/to_toc?pathname=server.112/ e10500/toc.htm

– Platform Migration Using Transportable Database —

http://www.oracle.com/technology/deploy/availability/pdf/MAA_ WP_10gR2_PlatformMigrationTDB.pdf

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 10 - 17

Practice 10 Overview: Migrating to Databases Machine using Transportable Tablespaces In this practice you will use Oracle Recovery Manager (RMAN), in conjunction with the transportable tablespace feature of Oracle Database,, to migrate g data from a big g endian p platform to Database Machine; a little endian platform.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 10 - 18

B lk D Bulk Data t L Loading di with ith Database D t b Machine M hi

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives After completing this lesson, you should be able to: • Describe the best practices for bulk data loading into Database Machine • Describe the process to configure the Database File System (DBFS) feature for staging input data files • Use external tables based on input data files stored in DBFS to perform high-performance data loads

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 11 - 2

Bulk Data Loading Overview

Source System

Database Machine

Target Database Staging Database

CREATE ... AS SELECT ... INSERT ... SELECT ...

DBFS

Network File Transfer (FTP, NFS, etc.)

Data file staged in Database File System (DBFS)

Data File

External table based on DBFS staged data

Target tables

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Bulk Data Loading Overview This lesson provides information on how to load your Database Machine databases with data from other systems using data files as the intermediate data store. store The lesson focuses on the recommended approach for loading file-based data into Oracle on Database Machine. This approach is illustrated on the slide. The remainder of this lesson describes this approach in greater detail.

Exadata and Database Machine Administration Workshop 11 - 3

Preparing the Data Files • •

Data files can be in any format supported by the external table feature To facilitate high-performance parallel access: – Oracle automatically divides the files into 10MB granules —

Exceptions: compressed files, data read from a pipe or a tape

– If granules cannot be created then: — —

—

Each file is treated as a granule The number of files determines the maximum degree of parallelism You must use multiple files to manually enable parallelism

– General rules of thumb — —

If using multiple files, then try to keep them similar in size If the file sizes vary significantly, then list them in order from largest to smallest in the external table definition Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Preparing the Data Files The recommended approach for bulk data loading into Database Machine relies on the external table feature of Oracle Database. Database The data files used for bulk data loading can be in any format supported by external tables. The process for creating the data files is outside the scope of this lesson and mostly depends on the facilities available in the source system. However the data files are created, the following should be taken into account in order to facilitate high-performance parallel access to the data files while they are being loaded: • When accessing large data files through external tables, where possible Oracle automatically divides the files into 10 MB granules. These granules can be processed in separate parallel processing threads. Oracle is unable to use this approach with compressed files or data read from a pipe or a tape device. • If granules cannot be used then each separate data file can be treated as a granule and the number of files determines the maximum degree of parallelism that is available. You can manually divide a large file into separate smaller files and use them to manually enable parallelism. • If you are using multiple input data files in conjunction with a single external table table, then you should try to keep the data files similar in size. If the file sizes do vary significantly, then list them in order from largest to smallest in the external table definition. Exadata and Database Machine Administration Workshop 11 - 4

Staging the Data Files •

Data files should always be staged in DBFS – Not database server internal drives

•

DBFS enables the database to be used as a file system – Shared storage for staging or storing data files files, scripts scripts, reports and other application files – Files are stored as SecureFiles LOBs inside database tables that are stored in Exadata – Files are exposed to the database servers using a POSIXcompatible file system interface – Files are protected like any Oracle data – ASM mirroring, Data Guard, Flashback, and so on – 5 to 7 GB/sec file system I/O throughput possible on a Full Rack Database Machine

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Staging the Data Files It is recommended that you stage your data files inside Database Machine using Database File y (DBFS). ( ) DBFS is an Oracle Database feature that enables the database to be used as System a high-performance POSIX-compatible file system on Linux. Using the available space on internal database server disk drives for staging data files is highly discouraged. Inside DBFS files are stored as SecureFiles LOBs. A set of PL/SQL procedures implement the file system access primitives, such as open, close, create, and so on. The dbfs_client utility enables the mounting of a DBFS file system as a mount point on Linux. It also provides the mapping from file system operations to database operations. The dbfs_client utility runs completely in user space and interacts with the kernel through the FUSE library infrastructure. infrastructure Note: ASM Cluster File System (ACFS) is not supported in conjunction with Exadata.

Exadata and Database Machine Administration Workshop 11 - 5

Configuring the Staging Area DBFS should be configured in a separate staging database • Use DBCA to create a database based on the OLTP template – – – –

• •

Redo logs at least 8 GB 4 GB buffer cache 1 GB shared pool 8 KB or 16 KB block size

Create a bigfile tablespace for DBFS storage C t a DBFS user accountt Create

SQL> create user dbfs identified by dbfs quota unlimited on ; SQL> grant create session, create table, create procedure, dbfs_role to dbfs;

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Configuring the Staging Area While DBFS is fully functional if it is co-located with your target database, it is recommended to configure DBFS in a separate staging database database. Use the Database Configuration Assistant (DBCA) to create a database based on the provided OLTP template. If you wish to access DBFS across multiple Database Machine database servers create the staging database as a RAC database across the desired servers. Default settings are mostly suitable when configuring the staging database however bear in mind the following: • Redo logs should be at least 8 GB in size • A 4 GB buffer cache is generally adequate • A 1 GB shared pool is generally adequate • An 8 KB or 16 KB block size is suitable After the staging database is created, create a bigfile tablespace to use as the DBFS store. Also, create a DBFS user account as suggested on the slide.

Exadata and Database Machine Administration Workshop 11 - 6

Configuring the Staging Area •

Additional database server operating system configuration – Add the Oracle software owner, or user that will mount the DBFS file system, to the fuse group

# usermod –a –G fuse oracle

– As root, create /etc/fuse.conf containing the entry: user_allow_other # echo "user_allow_other" > /etc/fuse.conf # chmod 644 /etc/fuse.conf

– Create a mount point for DBFS with ownership and group permissions i i sett tto th the O Oracle l software ft owner, or user th thatt will mount the DBFS file system # mkdir /data # chown oracle:dba /data

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Configuring the Staging Area (continued) To configure DBFS you must also perform the operating system configuration tasks listed on the slide. slide They should be performed on each database server where DBFS will be mounted mounted. The examples show a typical configuration using the Oracle software owner as the DBFS mount point owner. You can create and use an alternative operating system user if you wish to separate DBFS access away from the Oracle software owner and it’s database administration privileges. You can also use the dcli utility to replicate these configuration steps to multiple database servers.

Exadata and Database Machine Administration Workshop 11 - 7

Configuring the Staging Area •

Creating the DBFS store

$ cd $ORACLE_HOME/rdbms/admin $ sqlplus dbfs/dbfs SQL> @dbfs_create_filesystem_advanced.sql nocompress nodeduplicate noencrypt non-partition

•

Mounting DBFS

$ nohup $ORACLE_HOME/bin/dbfs_client dbfs@ -o allow_other,direct_io /data < passwd.txt &

•

Using DBFS – Access DBFS through / —

For example: /data/mydbfs

– Copy data files to DBFS using network file transfer methods such as FTP and NFS Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Configuring the Staging Area (continued) After the staging database is created and the required operating system configuration is completed you can create the DBFS store completed, store. Use the script located at $ORACLE_HOME/rdbms/admin/dbfs_create_filesystem_advanced.sql. The script must be run by the DBFS database user (created earlier in the configuration process). The script accepts numerous parameters. In the example on the slide, represents the name of the tablespace you created to house the DBFS store, and represents the name of the DBFS store, such as mydbfs for example. This name is used later after DBFS is mounted to name the directory that appears under the DBFS mount directory. directory To mount DBFS you can use dbfs_client as suggested in the example on the slide. Note that the example references /data as the DBFS mount directory. Also, note that the DBFS user password is contained in a file called passwd.txt. The password file is only required when mounting DBFS and is not required while DBFS is being used. Alternatively you can configure an Oracle Wallet so that the DBFS client can mount a DBFS store without a password. You can also configure DBFS to automatically mount using fstab.

Exadata and Database Machine Administration Workshop 11 - 8

Configuring the Staging Area (continued) After DBFS is mounted you can access it through the mount directory (/data in the slide example). In the mount directory you will find a subdirectory named after the DBFS store ( in the slide); the contents of this subdirectory is the contents of the DBFS store. Note that it is possible to have multiple copies of dbfs_client accessing the same shared file system. The sharing and caching semantics are similar to NFS. Like NFS, the default mode caches writes on the client and flushes them after a timeout or when the file being modified is closed. Also like NFS, writes to a file are only visible to clients that open the file after the writer closed the file. This behavior is commonly referred to as close-to-open cache consistency. For more information regarding DBFS on Database Machine refer to My Oracle Support bulletin 1054431.1. See also the chapters on DBFS located in the Oracle Database SecureFiles and Large Objects Developer's Guide 11g Release 2 (11.2).

Exadata and Database Machine Administration Workshop 11 - 9

Configuring the Target Database •

Prerequisites for data file access using external tables – Create an Oracle directory object that references the DBFS staging area directory – Grant the required q p permissions on the Oracle directoryy object j – Create the required external tables

•

Ensure efficient space management – Use bigfile tablespaces – Use 8MB initial extents for large segments — —

• •

Set the INITIAL storage parameter to 8MB Set the CELL_PARTITION_LARGE_EXTENTS initialization parameter to TRUE or ALWAYS

Use unlimited quotas to bypass quota management Use the parallel clause to set the default degree of parallelism for the target tables Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Configuring the Target Database After the DBFS-based staging area is configured it can be populated with the required data files To access the data files using external tables you must: files. • Create an Oracle directory object that references the DBFS staging area directory. • Grant the required permissions on the Oracle directory object so that the data files can be referenced by the required target database users. • Create the required external tables referencing the data files stored in DBFS. You should also prepare your target database more generally to optimize the load process. You should ensure efficient space management by following the general space management recommendations for use in conjunction with Exadata that are listed on the slide. In addition, you can optimize the performance of your loads if you bypass tablespace quota management and you set an appropriate default degree of parallelism for your target tables.

Exadata and Database Machine Administration Workshop 11 - 10

Loading the Target Database •

Recommended approach uses external tables – Parallel direct path loading for high-performance – In-flight processing using SQL — —

Transformations using g SQL Q functions Sort data while loading to optimize Exadata storage indexes

– No need to re-stage the data – Other advanced features —

•

For example: Input file preprocessing

SQL*Loader SQL Loader can also be used – Parallel direct path loading can also be achieved – Less processing flexibility compared with external tables – Existing SQL*Loader scripts can be easily reused

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Loading the Target Database The recommended method for loading the data files uses external tables. Loading data using external tables provides the following key benefits: • You can use direct path loading and parallel processing to achieve high-performance. • You can use the flexibility and power of SQL to efficiently process the data while it is being loaded. For example, you can use standard or user-defined SQL functions to transform the data during loading and you can sort the data while you load it to optimize the storage indexes that are automatically maintained by Exadata. • Because yyou can p process the data while it is being g loaded,, it is unlikely y that yyou will need to re-stage the data inside the target database. • External tables provide some useful advanced features. A primary example is the ability to preprocess a data file using a user-defined program which provides additional flexibility and power to process the data using routines outside the database. SQL*Loader can also be used to load data files. It can deliver comparable performance but does not provide the same level of processing flexibility when compared with external tables. SQL*L d remains SQL*Loader i a good d choice h i when h customers t already l d h have SQL*L SQL*Loader d b based d scripts i t that they wish to reuse in a Database Machine environment.

Exadata and Database Machine Administration Workshop 11 - 11

Loading the Target Database •

Parallel loading using external tables – A CREATE TABLE ... AS SELECT statement automatically uses the defined degree of parallelism – An INSERT ... SELECT statement needs parallel DML to be enabled

SQL> alter session enable parallel dml;

•

Direct path loading using external tables – A CREATE TABLE ... AS SELECT statement automatically uses direct path loading – An INSERT ... SELECT statement needs an APPEND hint to enable direct path loading

SQL> insert /*+ APPEND */ into my_table select * from my_external_table;

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Loading the Target Database (continued) Bulk data loading using external tables is achieved using a CREATE TABLE ... AS SELECT statement or an INSERT ... SELECT statement. statement To utilize parallel processing, it is recommended that you set the default degree of parallelism for the external tables used in the load and the target table being created or inserted into. Alternatively, you can set parallelism within the statement using a PARALLEL hint. After parallelism is configured, a CREATE TABLE ... AS SELECT statement will automatically use the defined degree of parallelism. An INSERT ... SELECT statement requires q p parallel DML to be enabled before it can execute in parallel. p To utilize direct path lading from external tables, you must use an APPEND hint for an INSERT ... SELECT statement. CREATE TABLE ... AS SELECT statements automatically use direct path loading. Remember that the bulk data loading approach outlined in this lesson can be used in conjunction with other techniques such as partition exchange loading for example.

Exadata and Database Machine Administration Workshop 11 - 12

Quiz To facilitate parallel loading from typical flat files your external table definitions must reference numerous smaller data files rather than one file containing all the data: 1. True 2. False

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 2 Oracle automatically divides typical flat files into 10MB granules to facilitate parallel processing Oracle is unable to use this approach with compressed files or data read from a processing. pipe or a tape device.

Exadata and Database Machine Administration Workshop 11 - 13

Quiz While DBFS is fully functional if it is co-located with your target database, it is recommended to you should configure DBFS in a separate staging database: 1. True 2. False

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 1

Exadata and Database Machine Administration Workshop 11 - 14

Summary In this lesson, you should have learned how to: • Describe the best practices for bulk-data loading into Database Machine • Describe the process to configure the Database File System (DBFS) feature for staging flat files • Use external tables based on flat files stored in DBFS to perform high-performance data loads

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 11 - 15

Additional Resources •

Lesson Demonstrations (Viewlets) – Bulk Data Loading with Database Machine —

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/111DBMBulkD ataLoad/111dbmbulkdataload_viewlet_swf.html

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 11 - 16

Practice 11 Overview: Bulk Data Loading with Database Machine In this practice you will perform a bulk data load on Database Machine. You will configure a database file system (DBFS) and use it to stage a CSV formatted file. You will then use the external table feature of Oracle Database to reference the CSV file. Finally, you will use a CREATE TABLE AS SELECT statement to copy the CSV file data into a table in your database.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 11 - 17

B k Backup and dR Recovery with ith D Database t b M Machine hi

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives After completing this lesson, you should be able to: • Describe how RMAN backups are optimized using Exadata • Describe the recommended approaches for disk-based disk based and tape-based backups of databases on Database Machine. • Describe the recommended best practices for backup and recovery on Database Machine.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 12 - 2

Backup and Recovery Overview •

Backup and recovery of databases on Database Machine – Use RMAN – Typical strategies: — — —

•

Disk-based backups p Tape-based backups Hybrid strategy

Backup and recovery of Database Machine software – Database server software – Exadata storage server software

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Backup and Recovery Overview This lesson focuses on the best practice recommendations for backup and recovery in conjunction with Database Machine Machine. The slide lists the topics that are considered throughout the lesson.

Exadata and Database Machine Administration Workshop 12 - 3

Using RMAN with Database Machine •

Using RMAN on Database Machine is essentially the same as using RMAN elsewhere – Same concepts – Same commands

•

Incremental backup performance is improved – Block filtering is offloaded to Exadata – Fewer blocks need to be processed by RMAN – Offload processing is automatic and transparent

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Using RMAN with Database Machine Oracle recommends the use of RMAN for database backup and recovery in conjunction with Database Machine Machine. In essence essence, using RMAN with Database Machine is the same as using RMAN without Database Machine. To optimize the performance of incremental backups, the database can offload block filtering to Exadata. This optimization is only possible when taking backups using RMAN. The offload processing is done transparently without user intervention. During offload processing, Exadata filters out the blocks that are not required for the incremental backup in progress. Therefore, only the blocks that are required for the backup are sent to the database. This can significantly i ifi tl iimprove b backup k performance, f especially i ll if th the proportion ti off changed h d bl blocks k iis low.

Exadata and Database Machine Administration Workshop 12 - 4

General Recommendations for RMAN Use RMAN to backup and recover databases on Database Machine • Use RMAN incremental backups and block change tracking g • Use an external RMAN recovery catalog repository • Set DB_RECOVERY_FILE_DEST_SIZE to bound space used in the Fast Recovery Area

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

General Recommendations for RMAN The following recommendations apply when using RMAN in conjunction with Database Machine: • Use RMAN incremental backups and block change tracking. For fast incremental backups, enable block change tracking. Block change tracking allows RMAN to avoid scanning blocks that have not changed, when creating incremental backups. Also, when performing incremental backups of databases on Exadata storage, additional block inspection is offloaded from the database servers. Block change tracking provides the greatest benefit for databases where fewer than 20% of the blocks are changed b t between iincremental t lb backups. k Y You may still till benefit b fit by b using i bl block k change h ttracking ki with ith change rates greater than 20%, but testing is recommended to ensure that backup times are reduced. • Use an external RMAN recovery catalog repository. The RMAN recovery catalog should be hosted on a server outside Database Machine. In practice, it is common to have a server that hosts the RMAN catalog along with other management repositories such as the Oracle Enterprise Manager repository and the Oracle Secure Backup catalog.

Exadata and Database Machine Administration Workshop 12 - 5

General Recommendations for RMAN (continued) •

Set DB_RECOVERY_FILE_DEST_SIZE to bound space used in the Fast Recovery Area. The database writes archived redo log files and any additional recovery files to the Fast Recovery Area. These include any disk backup files such as level 0 image copies and level 1 backup sets as well as Flashback log files (if enabled). It is important that you set the value of this parameter to less than the total free space in the disk group, taking into account at least one disk failure, and preferably one Exadata cell failure. Additionally, if multiple databases are sharing the Fast Recovery Area, ensure that the sum of the space allocated to the different databases is less than the free space in the disk group.

Exadata and Database Machine Administration Workshop 12 - 6

Disk Based Backup Strategy For disk-based database backups, Oracle recommends: • Use a Fast Recovery Area • Perform an initial level 0 (full) backup • Perform periodic incremental level 1 backups • Update your level 0 backup by applying the second to last level 1 backup

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Disk Based Backup Strategy For disk-based database backups, Oracle recommends incremental backups to a Fast Recovery Area Area. The recommended strategy is outlined in the slide slide. Using this approach approach, you can achieve a good compromise between the time and effort required during backup and recovery operations. The approach also efficiently manages the amount of storage required for backups.

Exadata and Database Machine Administration Workshop 12 - 7

Disk Based Backup Configuration •

Fast Recovery Area (FRA) configuration: – Default: FRA disk group striped across all available Exadata storage along with data disk groups – Alternative: FRA disk g group p and data disk g groups p on separate Exadata storage servers

•

Additional RMAN recommendations: – Instances and channels: —

— —

Initially, run RMAN using one database instance and two RMAN channels Add another instance with two channels if required Add an additional two channels per instance if required

– Configure an Oracle Service to use as the RMAN target

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Disk Based Backup Configuration By default, the installation and configuration procedure for Database Machine yields a disk group for the Fast Recovery Area (FRA) which is spread across all of the disks in all of the Exadata storage servers. In this configuration, the FRA shares each disk with disk groups containing database data files. By default, all the disk groups are created with NORMAL ASM redundancy (2-way mirroring). The result is that Database Machine can tolerate the loss of an entire Exadata storage server and still maintain full data integrity. Where customers desire a separation between database files and the FRA, the recommended alternative is to configure the FRA disk group on a dedicated set of Exadata storage t servers and d to t configure fi the th required i dd data t di disk k groups on separate t storage t servers. This configuration means that the I/O for a single workload can no longer benefit from being striped across all the storage servers, however it also provides additional protection if multiple simultaneous failures affect either the database or FRA. Multiple simultaneous failures to both the database and FRA can still result in data loss. Another strategy is to purchase additional high capacity Exadata servers specifically to store g the full capacity p y of the Database Machine the FRA. This allows yyour databases to leverage storage servers. To implement this strategy, the FRA storage servers need to be hosted in a separate rack and they need to be connected to the Database Machine storage network using the spare ports on the Database Machine InfiniBand switches. Exadata and Database Machine Administration Workshop 12 - 8

Disk Based Backup Configuration (continued) In addition to the general recommendations for RMAN in conjunction with Database Machine, Oracle recommends the following configuration for disk based backups: • Testing shows that optimal backup rates are achieved with 2 database instances and 2 to 4 RMAN channels per instance. For the 2 database instances designated as backup servers, less than 10% CPU and less than 40% IO bandwidth are utilized. During backup operations, I/O intensive parallel queries should avoid these designated backup servers. Start by running RMAN on one database instance using two RMAN channels. If greater backup throughput is required, use a second instance with two more channels. Finally, if required add two more channels per instance. • Configure an Oracle Service to run against specific database instances in the cluster. The service is used by RMAN to automatically spread the backup load evenly among the target instances offering the service.

Exadata and Database Machine Administration Workshop 12 - 9

Tape Based Backup Strategy For tape based database backups, Oracle recommends: • Use media management software that is integrated with RMAN, like Oracle Secure Backup • Perform periodic level 0 (full) database backups • Perform more frequent cumulative level 1 backups and also backup the Oracle Secure Backup catalog

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Tape Based Backup Strategy For tape based database backups, Oracle recommends the use of Oracle Secure Backup or other media management software that is integrated with RMAN RMAN. The recommended backup strategy is to perform periodic (weekly) level 0 (full) database backups. In addition, more frequent (daily) cumulative level 1 backups should be taken along with a backup of the Oracle Secure Backup catalog if it is the media management software which is used.

Exadata and Database Machine Administration Workshop 12 - 10

Tape Based Backup Configuration Database Machine

Sun StorageTek SL500 Oracle Secure Backup Administration Server (Sun Fire X4170)

Oracle Secure Backup Media Servers (Sun Fire X4275)

InfiniBand Network

Fiber Channel SAN

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Tape Based Backup Configuration The slide illustrates the recommended reference architecture for a tape based backup configuration using Oracle Secure Backup. Backup The key recommendations associated with the configuration follow: • Connect the media servers to Database Machine using the high-performance InfiniBand network. The InfiniBand connection to each media server should be bonded for high availability. Alternatively, you can use a Gigabit Ethernet network between the media servers and Database Machine, however if you use this configuration be aware that the network will likely be the constraining factor for backup and recovery performance. • Typically, backup performance is limited by tape drive throughput. Backup performance scales when you add more tape drives and RMAN channels. Allocate a sufficient number of tape drives so the media servers can achieve their maximum backup and restore rates. Add tape drives until the bandwidth of the media servers is saturated. • Start with at least two media servers. Add media servers if you have enough tape drives to keep them busy without saturating Database Machine resources.

Exadata and Database Machine Administration Workshop 12 - 11

Tape Based Backup Configuration •

Media server InfiniBand configuration recommendations: – Configure bonding of the media server InfiniBand interfaces – Update OpenFabrics Enterprise Distribution on the media server – Configure IP over InfiniBand connected mode for best performance – Set the message transfer unit (MTU) size to 65520 for the InfiniBand interface – Configure the media management software to use the InfiniBand network

•

Media server SAN configuration recommendation: – Configure persistent bindings for tape devices

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Tape Based Backup Configuration (continued) Media servers can be directly connected to the Database Machine InfiniBand fabric by adding an InfiniBand Quad Data Rate (QDR) host channel adapter (HCA) to the media server server. For high availability, connect the HCA to two different Database Machine InfiniBand leaf switches to eliminate the switch as a single point of failure. This provides transparent failover if connectivity is lost to one of the ports. Follow these recommendations: • Configure bonding of the media server InfiniBand interfaces. Active-passive bonding is recommended, which is consistent with the database servers and Exadata servers in Database Machine. • You must use an OpenFabrics Enterprise Distribution (OFED) version on the media server that is compatible with the version found in Database Machine. For details, refer to My Oracle Support bulletin 888828.1. • Configure IP over InfiniBand connected mode for best performance. On Linux, edit the /etc/ofed/openib.conf file so that it contains the entry SET_IPOIB_CM=yes. Reboot the server to enable the setting. To verify the setting, check the contents of /sys/class/net/ib0/mode / y / / / / and / /sys/class/net/ib1/mode. y / / / / Both files should contain the entry connected.

Exadata and Database Machine Administration Workshop 12 - 12

Tape Based Backup Configuration (continued) •

•

Set the message transfer unit (MTU) size to 65520 for the InfiniBand interface. On Linux, edit the /etc/sysconfig/network-scripts/ifcfg-ib* and /etc/sysconfig/network-scripts/ifcfg-bond0 / / y g/ p / g files so that they y contain the entry MTU=65520. Verify the MTU setting by examining the output of the ifconfig command for the InfiniBand interfaces. To direct the backup and restore traffic over the InfiniBand network, configure the media management software to favor InfiniBand. Note that each media management software type has its own method of enabling this configuration. For example, Oracle Secure Backup has the concept of a preferred network interface, which can be set on the media server for a specific list of clients clients. Other media management software may require this configuration to be defined when the software is installed. See the media management software for information about how to direct traffic over a particular network.

For SAN attached tape devices you must configure persistent bindings so the device address does not change. If the device address changes, the media servers cannot access the device unless you update the device configuration within the media server software. Therefore, it is very important that your environment maintains consistent device addresses. Persistent bindings are a part of your SAN infrastructure setup. Typically you will configure persistent bindings through the HBA or the operating system. The configuration steps may vary by platform and vendor.

Exadata and Database Machine Administration Workshop 12 - 13

Tape Based Backup Configuration •

Run RMAN across all the available database instances – Create a Database Service that runs across the cluster

$ srvctl add service –d -s -r , ... ,

– Use the service name and SCAN address to connect RMAN $ rman target sys/@/ catalog …

• •

Allocate one RMAN channel per tape drive Configure IORM and DBRM to control resource allocation between backups and application workloads

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Tape Based Backup Configuration (continued) Further recommendations: • For your RAC databases databases, configure RMAN to run across all the database instances instances. It is recommended that you create a Database Service to run across the RAC cluster. Then when running RMAN, use the service name and SCAN address in the connect string for the RMAN target as shown in the slide. • Allocate one RMAN channel per tape drive. A single RMAN channel in Database Machine can stream data at a rate of 749 MB/sec from Database Machine to the media server. Typical tape drive backup rates are between 100 MB/sec and 240 MB/sec, depending on the drive type and compression options. Note that tape drive compression becomes less effective when backing up tables that are compressed at the database level. • If Database Machine resources must be prioritized between application workloads and backups, then configure I/O Resource Manager (IORM) and Database Resource Manager (DBRM). This is more likely to be required in cases where time-consuming backups bac ups to large a ge databases must ust run u at the t e same sa e ttime e as p production oduct o app application cat o workloads.

Exadata and Database Machine Administration Workshop 12 - 14

Hybrid Backup Strategy •

A hybrid backup strategy combines the disk-based and tape-based backup approaches: – Level 0 (full) database backups are stored on tape —

Relatively y cheap p backup p media that can be stored off site

– Level 1 incremental backups are stored on disk in the FRA —

•

Readily available with high performance access

Follow recommendations for both approaches

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Hybrid Backup Strategy A hybrid backup strategy combines the disk-based and tape-based backup approaches. For example a reasonable hybrid approach would result in level 0 (full) database backups being example, stored on tape, while the level 1 incremental backups would be stored on disk in the FRA. The hybrid approach also combines the benefits of each storage type as indicated on the slide. If you choose to implement a hybrid backup approach you should follow the recommendations for disk-based backups and tape-based backups.

Exadata and Database Machine Administration Workshop 12 - 15

Restore and Recovery Recommendations •

Restore into existing data files if possible – Restore performance is better – Restore using all database instances

• •

If no existing data files are present present, restore using up to 2 database instances Recommended number of RMAN channels: – For disk-based restoration, use 2 to 4 RMAN channels per database instance – For tape-based restoration restoration, set the number of RMAN channels based on the total number of tape drives

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Restore and Recovery Recommendations Higher restore rates are achieved when avoiding the overhead of initial data file allocation. If existing files are present prior to database restoration restoration, do not delete the data files and perform database restoration into the existing files to take advantage of this optimization. With pre-existing files, create a restore service across all database instances. If no existing data files are present prior to the restore operation, create a restore service with only two database instances. Use 2 to 4 RMAN channels per database instance for disk-based restoration. For tape-based restoration, the number of channels should be set to the number of tape drives.

Exadata and Database Machine Administration Workshop 12 - 16

Backup and Recovery of Database Machine Software •

Database Server software – Perform file system level backup and recovery —

—

•

Use your chosen file system backup management software or Oracle Secure Backup can be used Copies of the Oracle Cluster Registry are automatically maintained on the Database Server file system and should be included in Database Server file system backups

Exadata software – File system level backups are not recommended —

System areas are mirrored —

—

Use CellCLI commands to recover if one system disk fails

Use the Exadata Software Rescue Procedure if both system disks fail simultaneously —

The rescue procedure uses a built-in USB flash drive

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Backup and Recovery of Database Machine Software It is recommended that you use file system level backup and recovery techniques for the database servers. servers Use your chosen file system backup management software and infrastructure or use Oracle Secure Backup if desired. Note that a backup of the Oracle Cluster Registry, which also contains Voting Disk information, is automatically maintained on the file system of the first database server and should be included in your database server file system backups. The default location for the Oracle Cluster Registry backup is /cdata/, where: • is the location of the Grid Infrastructure software as specified during the initial configuration of Database Machine. Machine The default G id H location is /u01/app/11.2.0/grid. • is the name of your cluster. This is the same as the DB Machine Prefix specified during the initial configuration of Database Machine. File system level backups are not recommended for Exadata software. Exadata maintains mirrored system areas on separate physical disks. In the rare event that both system disks fail simultaneously, you must use the rescue functionality provided on the CELLBOOT USB flash drive that is built into every Exadata server.

Exadata and Database Machine Administration Workshop 12 - 17

Quiz To facilitate disk-based backups you can install additional high capacity Exadata servers into the available space in a Half Rack Database Machine based on high performance disks: 1. True 2. False

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 2 Adding Exadata servers into the same rack as a Half Rack Database Machine is only supported using the Half Rack to Full Rack upgrade package package, and the resulting Full Rack Database Machine can only utilize one disk type; high performance or high capacity. The only supported way to add high capacity cells to a Half Rack Database Machine using high performance cells is to install the high capacity cells into a separate rack and connect them to the Database Machine storage network using the spare ports on the Database Machine InfiniBand switches.

Exadata and Database Machine Administration Workshop 12 - 18

Quiz How many RMAN channels should you use for tape-based backups? 1. 2 per database instance 2 4 per database instance 2. 3. 1 per tape drive

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 3

Exadata and Database Machine Administration Workshop 12 - 19

Summary In this lesson, you should have learned how to: • Describe how RMAN backups are optimized using Exadata • Describe the recommended approaches for disk-based disk based and tape-based backups of databases on Database Machine. • Describe the recommended best practices for backup and recovery on Database Machine.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 12 - 20

Additional Resources •

Lesson Demonstrations (Viewlets) – Backup Optimization Using RMAN and Exadata —

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/121DBMBacku pRMAN/121dbmbackuprman_viewlet_swf.html

– Recovery Optimization Using RMAN and Exadata —

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/122DBMRecov erRMAN/122dbmrecoverrman_viewlet_swf.html

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 12 - 21

Practice 12 Overview: Using RMAN Optimizations for Database Machine In these practices, you will examine the backup and recovery optimizations that are enabled when Oracle Recovery Manager (RMAN) is used in conjunction with Exadata storage.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 12 - 22

M it i Monitoring and d Maintaining M i t i i Database D t b Machine M hi

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives After completing this lesson, you should be able to: • Describe the monitoring tools provided with Database Machine • Describe the support support, update and patching considerations that apply to Database Machine

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 13 - 2

Monitoring Tools Overview •

Most Database Machine monitoring tools are not specific to Database Machine; for example: – Enterprise Manager – Operating system tools and utilities – Dynamic performance views (V$ views) and wait events – Alerts logs and trace files

•

Exadata-specific monitoring capabilities – Exadata metrics, alerts and active requests – Exadata-specific V$ view entries and wait events – Enterprise Manager Plug Plug-In In for Exadata

•

Additional monitoring capabilities for Database Machine – Integrated Lights Out Manager (ILOM) – The dcli utility – InfiniBand diagnostic utilities

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Monitoring Tools Overview Monitoring Database Machine is largely the same as monitoring Oracle RAC on Linux. The biggest difference is the use of Exadata storage and the use of InfiniBand for the storage network and cluster interconnect. Most of the tools and utilities available for Database Machine are the same tools and utilities used by Oracle administrators and system administrators apart from Database Machine. As a Database Machine administrator, Oracle recommends that you use Enterprise Manager in conjunction with other lower level monitoring capabilities such as the dynamic performance views, wait events, alert logs and trace files available in both ASM and Oracle Database. In addition, dditi you can use operating ti system t ttools l and d utilities tiliti such h as vmstat, iostat, i top, syslog, and so on. The slide also lists a set of Exadata-specific monitoring capabilities that have already been discussed in the lesson Exadata Performance Monitoring and Maintenance. Finally, the slide lists three additional monitoring capabilities which are likely to be unfamiliar to Oracle administrators outside of Database Machine. The following slides introduce the Integrated Lights Out Manager (ILOM) (ILOM), the dcli utility, utility and the InfiniBand diagnostic utilities utilities.

Exadata and Database Machine Administration Workshop 13 - 3

ILOM Overview ILOM Capability

What You Can Do

Dedicated service processor and resources

• Manage the server without consuming system resources • Continue to manage the server using standby power even when the server is powered-off

Remote hardware monitoring

• Monitor system status and event logs supplies fans, fans disks, disks CPUs, CPUs memory, memory and motherboard • Monitor power supplies, • Monitor component temperatures • Monitor sensors, including voltage and power, and indicators (LEDs)

Hardware inventory and presence

• Identify part numbers, versions, and product serial numbers • Identify NIC card MAC addresses

Remote Access

• Redirect the system serial console • Access keyboard, video, and mouse (KVM) on remote systems

System power control and monitoring

• Power the system on or off, either locally or remotely • Force power-off for emergency shutdown or perform a graceful shutdown

Error and fault management

• Monitor system BIOS, POST, and sensor messages • Monitor hardware and system-related errors, as well as ECC memory errors, reported into SP logs, syslog, and remote log-host

System alerts, including SNMP traps, IPMI PETs, remote syslog, and email alerts

• Monitor components using industry-standard SNMP commands and the IPMItool utility

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

ILOM Overview Integrated Lights Out Manager (ILOM) provides advanced service processor hardware and software that you can use to manage and monitor Database Machine servers servers. ILOM’s ILOM s dedicated hardware and software is preinstalled on the database servers and Exadata servers inside Database Machine. ILOM enables you to actively manage and monitor the server independently of the operating system state. With ILOM, you can proactively: • Learn about hardware errors and faults as they occur • Remotely e ote y co control t o tthe e po power e state o of you your se server e • View the graphical and non-graphical consoles for the host • View the current status of sensors and indicators on the system • Determine the hardware configuration of your system • Receive generated alerts about system events in advance through IPMI PETs, SNMP traps, or email alerts. The slide shows a broader list of ILOM capabilities capabilities.

Exadata and Database Machine Administration Workshop 13 - 4

ILOM Overview (continued) The ILOM service processor (SP) runs its own embedded operating system and has a dedicated Ethernet port, which together provide an out-of-band management capability. In addition you can access ILOM from the server addition, server’s s host operating system. system Using ILOM, ILOM you can remotely manage your server as if you were using a locally attached keyboard, monitor, and mouse. ILOM automatically initializes as soon as power is applied to the server. It provides a fullfeatured, browser-based web interface and has an equivalent command-line interface (CLI). There is also an industry-standard SNMP interface and IPMI interface. You can easily integrate these management interfaces with other management tools and processes that you might have working already with your servers, such as Oracle Enterprise Manager Ops Center. In addition, you can integrate ILOM with many third-party management tools, such as BMC Patrol, CA Unicenter, and HP OpenView for example.

Exadata and Database Machine Administration Workshop 13 - 5

ILOM Example $ ssh root@eidmdb01-ilom Password: Sun(TM) Integrated Lights Out Manager Version 3.0.6.10.b r52264 ... -> show /SYS/T_AMB /SYS/T_AMB Targets: Properties: type = Temperature ipmi_name = T_AMB class = Threshold Sensor value = 23.000 degree C upper_nonrecov_threshold = 50.000 degree C upper_critical_threshold = 45.000 degree C ... -> reset /SYS Are you sure you want to reset /SYS (y/n)? y Performing hard reset on /SYS

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

ILOM Example The slide shows a simple example using the command line interface to ILOM. In the example, a remote connection is established to ILOM using the dedicated network address associated with ILOM on the specified server. After the connection is established you can execute ILOM commands to perform various functions. In the example, the first ILOM command shows information relating to the server’s temperature sensors, while the second command reboots the server. Note that the ILOM session continues beyond the reboot and other ILOM commands can be executed even while the server reboots. For more information, refer to the ILOM documentation located at htt //d http://docs.sun.com/app/docs/coll/ilom3.0 / /d / ll/il 3 0

Exadata and Database Machine Administration Workshop 13 - 6

DCLI Overview • •

The dcli utility allows you to simultaneously execute a command on multiple Database Machine servers Command types: – – – –

• • • •

Operating system commands CellCLI commands Operating system scripts CellCLI scripts

Commands are executed in separate parallel threads I t Interactive ti sessions i are nott supported t d Python 2.3 and SSH user-equivalence are required Command output is collected and displayed in the terminal executing the dcli utility

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

DCLI Overview The dcli utility facilitates centralized management across Database Machine by automating the execution of a command on a set of servers and returning the output to the centralized management location where dcli was run. The types of commands supported by dcli includes operating system commands, CellCLI commands, operating system scripts, and CellCLI scripts. The dcli utility runs commands on multiple servers in parallel threads. However, it does not support an interactive session with a remote application on a server. By default, the dcli utility is located at /opt/oracle/cell/cellsrv/bin/dcli on each Exadata server and att /usr/local/bin/dcli / /l l/bi /d li on each hd database t b server. Y You can also l copy th the dcli d li utility tilit to t a server outside of Database Machine and centrally execute commands from that server. The dcli utility requires Python version 2.3 or later on the server running dcli. You can determine the version of Python by running the python -V command. In addition, dcli requires prior setup of SSH user-equivalence between all the servers. You can use the dcli utility initially with the -k option to set up SSH user-equivalence between a group of servers. Command output (to stdout and stderr) is collected and displayed after command execution is finished on the specified servers. The dcli options allow command output to be abbreviated to filter output, such as removing messages showing normal status. Exadata and Database Machine Administration Workshop 13 - 7

DCLI Examples $ dcli –g mycells date eidmcel01: Sun May 16 20:48:09 CDT 2010 eidmcel02: Sun May 16 20:48:09 CDT 2010 $ dcli –c eidmcel01,eidmcel02 cellcli –e list cell eidmcel01: eidmcel01 online eidmcel02: eidmcel02 online $ dcli –g mycells –x cellclicommands.scl $ dcli –g mydbservers –l root –x dbwork.sh

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

DCLI Examples The slide shows a series of examples using the dcli utiltiy. The first example shows dcli d li being used to execute the operating system date d t command. command The –g option specifies a file (mycells) which identifies the list of target servers to which the command (date) is sent. The servers can be identified by host names or IP addresses. The servers can be database servers or Exadata servers. The second example uses the –c option to specify the target servers (eidmcel01,eidmcel02) on the command line. It invokes CellCLI to report the cell status. The third example uses the –x x option to specify a command file file. The command file must exist on the server executing the dcli utility. The command file is copied to the target servers and executed. A file with the .scl extension is run by the CellCLI utility on the target server. A file with a different extension is run by the operating system shell on the target server. The file is copied to the default home directory of the user on the target server. Files specified using the –x option must have execute privileges or else dcli will report an error. The final example adds the use of the –l option to specify the user to connect as on the remote t servers. By B default, d f lt the th dcli d li utility tilit connects t as th the celladmin ll d i user. For more information, refer to the chapter entitled Using the dcli Utility in the Oracle Exadata Storage Server Software User's Guide 11g Release 2 (11.2). Exadata and Database Machine Administration Workshop 13 - 8

InfiniBand Diagnostic Utilities •

verify-topology – Verifies the presence of the expected Database Machine storage network links – Example: p

# cd /opt/oracle.SupportTools/ibdiagtools # ./verify-topology [ DB Machine Infiniband Cabling Topology Verification Tool ] Is every external switch connected to every internal switch..........[SUCCESS] Are any external switches connected to each other....................[SUCCESS] Are any hosts connected to spine switch..............................[SUCCESS] Check if all hosts have 2 CAs to different switches..................[SUCCESS] Leaf switch check: cardinality and even distribution distribution.................[SUCCESS] [SUCCESS] Check if each rack has an valid internal ring........................[SUCCESS]

•

infinicheck – Verifies connectivity across the Database Machine storage network links and checks bandwidth across the links Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

InfiniBand Diagnostic Utilities On each Database Machine server there are two useful InfiniBand diagnostic utilities, verify-topology verify topology and infinicheck, infinicheck which are located at /opt/oracle.SupportTools/ibdiagtools. You must be the root user or a member of the root operating system group in order to run the utilities. verify-topology conducts a series of topology-level tests to verify that the expected storage network links are cabled correctly. An example of the expected output for verifytopology is shown in the slide. You can modify the behavior and output of verifytopology through the use of optional arguments, however in most cases you will find this unnecessary. Execute E t verify-topology if l –h h for f a list li t off th the optional ti l arguments. t infinicheck conducts a series of tests that verify connectivity across each Database Machine storage network link. It also stress-tests each link to check that the available bandwidth for each link is as expected. infinicheck reports output to the terminal and logs the terminal output along with additional diagnostic information to a file named diagnostics.output. Execute infinicheck –h to view a list of the optional arguments gp page g shows and example p of the terminal output p from for infinicheck. The following infinicheck.

Exadata and Database Machine Administration Workshop 13 - 9

InfiniBand Diagnostic Utilities (continued) Here is an example of the terminal output from infinicheck. #

./infinicheck

Verifying User Equivalance of user=root to all hosts. (If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes) Verifying User Equivalance of user=root to all cells. (If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)

#### CONNECTIVITY TESTS #### [COMPUTE NODES -> STORAGE CELLS] (30 seconds approx.) [SUCCESS]..............Connectivity verified [ [SUCCESS]....... ] All ll h hosts can talk lk to all ll storage cells ll Verifying Subnet Masks on Hosts and Cells [SUCCESS] ......... Subnet Masks is same across the network Checking for bad links in the fabric [SUCCESS].......... No bad fabric links found [COMPUTE NODES -> COMPUTE NODES] (30 seconds approx.) [SUCCESS]..............Connectivity verified [SUCCESS]....... All hosts can talk to all other nodes

####

PERFORMANCE TESTS

####

[(1) Every COMPUTE NODE to its STORAGE CELL] (30 seconds approx.) [SUCCESS]........ Network Bandwidth looks OK. .......... To view only performance results run ./infinicheck -d -p [(2) Every COMPUTE NODE to another COMPUTE NODE] (30 seconds approx.) [SUCCESS]........ Network Bandwidth looks OK. ...... To view only performance results run ./infinicheck -d -p [(3) Every COMPUTE NODE to ALL STORAGE CELLS] (90 seconds approx.) (looking for SymbolErrors) [SUCCESS]....... No port errors found #

Exadata and Database Machine Administration Workshop 13 - 10

Database Machine Support Overview Support Offering

Key Features

Oracle Hardware Warranty

• Included with Database Machine, 1 year term • Any-time Web access, Local business hours phone access • 4 hour response during normal business hours • On-site response and parts exchange within 2 business days subject to availability and location

Oracle Premier Support

• Support for Oracle Database and Exadata software • Any-time Web or phone access • Software enhancements, fixes and upgrades • Proactive tools, including alerts and configuration guidance

Oracle Premier Support for Systems

• Support for server and storage hardware and firmware; includes Oracle Linux • Any-time y Web or p phone access • On-site hardware response with 2 hours if Database Machine is within a 2 hour service coverage area

Oracle Customer Data and Device Retention

• Provides replacements for failed disk drives • Customer retains the failed disk drives • Provides additional security for sensitive data

– See also http://www.oracle.com/support/policies.html Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Database Machine Support Overview Oracle provides a complete and integrated set of support offerings for Database Machine. Customers can use a single contact point to access all of the support services outlined on the slide. There is also a single point of accountability, meaning that issues will never remain unresolved while separate support organizations wait for each other to progress. The support services outlined in the slide are modular so that customers can choose the right level of support for their needs. Oracle Hardware Warranty is included with every Database Machine and is the minimum level of support. Typically, Database Machine is deployed for enterprise-scale business-critical applications, and in these cases Oracle recommends that customers t purchase h O Oracle l P Premier i S Supportt and dO Oracle l P Premier i S Supportt ffor S Systems. t O Oracle l Customer Data and Device Retention is recommended where security or privacy requirements compel customers to ensure that sensitive data never leaves their enterprise. In addition to the support offerings outlined here, Oracle provides installation and configuration services for Database Machine. These services are highly recommended to ensure an efficient and trouble-free start with Database Machine. Additional services exist to p customers with upgrades pg from a Quarter Rack to a Half Rack,, or to upgrade pg from a Half help Rack to a Full Rack. A specialized service is also available for customers who wish to interconnect multiple Half Rack and Full Rack Database Machines. Exadata and Database Machine Administration Workshop 13 - 11

Patching and Updating Overview •

There are three categories of software that must be maintained in Database Machine: – Software and firmware in the Exadata servers – Software and firmware in the database servers – Software and firmware for other components, like the InfiniBand switches for example

• • •

Compatibility between these different pieces of software needs to be maintained Patches and updates are “rolling” rolling in nature wherever possible Key information is maintained in My Oracle Support bulletin 888828.1

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Patching and Updating Overview There are three broad categories of software that must be maintained in Database Machine. That is is, software and firmware in the Exadata servers servers, database servers and other components like the InfiniBand switches. Compatibility between these different pieces of software is vital for Database Machine to function. Wherever possible, Database Machine patches and updates are “rolling” in nature, meaning that they are applied in a manner that facilitates ongoing system availability by “rolling” the patch across the environment one server at a time rather affecting all the servers at once. p information relating g to Database Machine p patches and updates p is maintained in My y Important Oracle Support bulletin 888828.1. This bulletin is constantly updated with new information. Oracle recommends that all Database Machine customers should sign up for an automated alert when this bulletin changes. The remainder of this lesson outlines some additional recommendations regarding Database Machine patching and updating.

Exadata and Database Machine Administration Workshop 13 - 12

Maintaining Exadata Software • •

Patches for Exadata are provided by Oracle as a patch bundle The patch bundle maintains consistency across all the Exadata software components p – – – –

• • •

Oracle Linux Exadata Storage Server software InfiniBand software Firmware

Patches can generally be applied online, online while the database is up and running Exadata automatically maintains firmware levels No additional software, RPMs or otherwise, should be installed Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Maintaining Exadata Software Patches for Exadata are provided by Oracle as a patch bundle. The patch bundle maintains consistency across all the Exadata software components. components Do not manually update Exadata firmware or software unless directed by Oracle Support. Exadata patches can generally be applied in a “rolling” manner, while the database continues to run. Note that the ability to perform rolling Exadata patches is not available if you use EXTERNAL redundancy disk groups which are not recommended on Database Machine. Patches may also include instructions for parallel installation on multiple Exadata servers. This approach generally requires down time but may be preferred if a scheduled maintenance window i d exists i t for f the th system. t As described in the lesson Exadata Performance Monitoring and Maintenance, Exadata automatically maintains firmware within the server. Firmware levels are periodically checked while Exadata is running and the correct firmware is automatically applied to components when the server reboots. Disk firmware is also automatically updated when a disk is replaced. Oracle recommends that no additional software, whether RPMs or otherwise, should be installed on Exadata Exadata.

Exadata and Database Machine Administration Workshop 13 - 13

Maintaining Database Server Software •

You can patch and update the database server software as you would for an Oracle Database server outside of Database Machine – Oracle Database p patches applied pp by y OPatch — —

Double-check compatibility of patches with Database Machine Oracle supplies Bundle Patches for Database Machine —

Periodic bundle of database patches recommended for Database Machine

– Operating system and firmware updated by Unbreakable Linux Network (ULN) —

•

Must maintain consistency with InfiniBand (OFED) software

Check Exadata patches for database server firmware and operating system updates

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Maintaining Database Server Software In essence, you can patch and update the database server software using the same techniques and approaches as you would for an Oracle Database server outside of Database Machine. OPatch is used to apply and manage Oracle Database patches. It is recommended that you raise a Service Request with Oracle Support to verify the compatibility of patches with Database Machine. In addition to regular database patches and updates, Oracle supplies periodic Bundle Patches for Database Machine. These bundle a series of database patches that are recommended for use in conjunction with Database Machine. Oracle recommends th t you should that h ld apply l Bundle B dl Patches P t h as partt off your patching t hi regime. i Operating system and firmware patches should be managed using standard Linux patching practices, such as using the Unbreakable Linux Network (ULN). My Oracle Support bulletin 888828.1 lists current constrains and requirements for operating system and firmware patches. The main consideration is to ensure that any patches or updates maintain consistency with the OFED software version which underpins the InfiniBand network. Some Exadata patches also include database server firmware and operating system updates updates. This may occur, for example, when the firmware for the InfiniBand HCA is updated on Exadata and the update must also be applied to the database servers. Exadata and Database Machine Administration Workshop 13 - 14

Maintaining Other Software •

Other components in Database Machine that have software or firmware: – InfiniBand switches: Software and firmware must be maintained in accordance with Oracle guidelines published in My Oracle Support bulletin 888828.1 – KVM module: May be updated according to customer requirements – Cisco Ethernet switch: May be updated according to customer requirements

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Maintaining Other Software Other Database Machine components that have software or firmware include: • InfiniBand switches: The software and firmware on the InfiniBand switches must be maintained in accordance with Oracle guidelines published in My Oracle Support bulletin 888828.1. • Keyboard, Video and Mouse (KVM) module: The KVM can be updated according to customer requirements. There are no specific requirements or constraints from Oracle. • Cisco Ethernet switch: The Cisco Ethernet switch can be updated according to customer requirements. q There are no specific p requirements q or constraints from Oracle.

Exadata and Database Machine Administration Workshop 13 - 15

Quiz ILOM enables you to actively manage and monitor an individual server independently of the operating system state. With ILOM, you can proactively: 1. Learn about hardware errors and faults as they y occur 2. Learn about Exadata software faults as they occur 3. Remotely control the power state of your server 4. View the graphical and non-graphical consoles for the host 5. View the contents of the database alert log 6. View the current status of sensors and indicators on the server 7. View information about the hardware configuration of the server

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answers: 1, 3, 4, 6, 7 ILOM does not generate Exadata software alerts, however a hardware fault may generate an ILOM alert and an Exadata alert alert. ILOM does not natively provide a view of the database alert log, however you can use ILOM to remotely access the console for a database server and using that console access you can log into the sever as a DBA user and view the database alert log.

Exadata and Database Machine Administration Workshop 13 - 16

Quiz Which of the following statements is a broad overview of the recommended approach for patching and updating Database Machine: 1. All updates p for Database Machine are specific p to Database Machine, and these are the only updates that should be applied to Database Machine 2. Use Unbreakable Linux Network (ULN) to update Linux on database servers and Exadata servers, and apply specific database and Exadata updates p for the remaining g software 3. Use Exadata-specific update bundles for the Exadata servers, and use normal database updating practices for the database servers

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Answer: 3

Exadata and Database Machine Administration Workshop 13 - 17

Summary In this lesson, you should have learned how to: • Describe the monitoring tools provided with Database Machine • Describe the support support, update and patching considerations that apply to Database Machine

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 13 - 18

Additional Resources •

Lesson Demonstrations (Viewlets) – Using the distributed command line utility (dcli) —

•

http://stcurriculum.oracle.com/demos/db/11g/r2/dbmach/131DBMDCLI/ 131dbmdcli_viewlet_swf.html

Documentation and Support Bulletins – ILOM Documentation —

http://docs.sun.com/app/docs/coll/ilom3.0

– Database Machine patches and updates —

My Oracle Support bulletin 888828.1

– Oracle Database Machine monitoring —

My Oracle Support bulletin 1110675.1

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 13 - 19

Practice 13 Overview: Using the distributed command line utility (dcli) In this practice you will use the distributed command line utility (dcli) provided with Database Machine to simultaneously execute monitoring and administration commands across multiple servers.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop 13 - 20

N New F Features t iin Update U d t Release R l 11.2.1.3.1 11 2 1 3 1

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives After completing this lesson, you should be able to: • Describe the new features of Exadata and Database Machine introduced in update release 11.2.1.3.1

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop A - 2

New Features Overview Release 11.2.1.3.1 includes: • Auto Service Request (ASR) – An automatic service requests is opened with Oracle Support when e a hardware a d a e failure a u e co condition d o is s de detected ec ed

•

Oracle Linux 5.5 – Includes security fixes not present in Oracle Linux 5.3

•

Enhanced operating system security – Unnecessary packages and services are removed – Firewall Fi ll ffunctionality ti lit is i implemented i l t d

•

Pro-active disk quarantine – Reduces the potential for data loss due to multiple failures

• •

Alerts for disk failures on database servers SNMP alerts from InfiniBand switches Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

New Features Overview The slide lists the main new features associated with Exadata software release 11.2.1.3.1. Each feature is further introduced in the remainder of this lesson lesson.

Exadata and Database Machine Administration Workshop A - 3

Auto Service Request (ASR) •

Automatically and securely opens service requests with Oracle for common hardware faults – Minimal data is collected and transmitted – No IP addresses are communicated – One-way (customer to Oracle) SSL encrypted communications are used

•

Enables fast and accurate resolution of the hardware faults – Improved availability, less downtime

•

Can be integrated with existing monitoring tools – ASR manager can send SR notifications via SNMP traps to existing monitoring tools

•

Included with hardware warranty and Oracle Premier Support for Systems Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Auto Service Request (ASR) Oracle Auto Service Request for Sun Systems is a secure, scalable, customer-installable software solution available as a feature of your Oracle or Sun hardware warranty warranty, and Oracle Premier Support for Systems or valid Sun support plan. The ASR software helps to resolve problems faster by using auto-case generation for Oracle's Sun server and storage systems when specific hardware faults occur. With the release of the Exadata software version 11.2.1.3.1, ASR functionality is extended to Exadata and Database Machine. ASR automatically opens service requests (SRs) with Oracle Support when specific hardware faults occur either in the Exadata servers or the d t b database servers. Note N t that th t ASR is i applicable li bl only l ffor hardware h d ffaults lt d detected t t d on th the following server components: CPUs, disk controllers, disks, flash cards, flash modules, InfiniBand cards, memory modules, system boards, power supplied, and fans.

Exadata and Database Machine Administration Workshop A - 4

The ASR process Customer Datacenter

Oracle Support Services

FRU replaced by Field Engineer

Oracle Support Engineer Field Replaceable Unit (FRU) dispatched by Support Engineer

Oracle Field Engineer

Fault occurs

Customer

SNMP trap sent to ASR Manager

ASR Manager

Service Request routed to Support Engineer

SR creation email notification to customer

Oracle Case Management System

ASR Service

Fault telemetry securely transmitted to Oracle

Service Request (SR) created

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

The ASR process The diagram in the slide illustrates the ASR process. It shows how a Service Requests (SR) is automatically opened by the ASR Manager after it receives an SNMP trap that is triggered by a hardware fault. Within Oracle Support Services the automatic service request is acknowledged via email and serviced according to normal procedures. Customers should note that there are occasions when a SR may not be automatically filed. This can happen due to loss of connectivity to the ASR manager, for example. Oracle recommends that customers continue to monitor their systems for faults and engage Oracle Support if they notice a fault but do not receive notice that a service request has been automatically t ti ll fil filed. d

Exadata and Database Machine Administration Workshop A - 5

ASR requirements •

ASR Manager requirements: – Qualified Sun server running Oracle Linux 5.3 (or higher) or Solaris 10 update 6 (or higher) – ASR Manager g version 2.7 or higher g – Connectivity to the management network of Database Machine – https connectivity to Oracle Support – Can upgrade an existing ASR Manager to monitor Database Machine – Can use a Database Machine database server as the ASR Manager, though not recommended

•

Technical system administrator contact for the Database Machine registered with Oracle Support

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

ASR requirements The slide lists the requirements for using ASR in conjunction with Database Machine. See http://www oracle com/asr for more information including a list of qualified Sun servers which http://www.oracle.com/asr can be used to run the ASR Manager.

Exadata and Database Machine Administration Workshop A - 6

Oracle Linux 5.5 • •

•

Upgrading Exadata to version 11.2.1.3.1 causes Oracle Linux to be upgraded from 5.3 to 5.5 as well For Database Machines shipping from the factory with the Exadata software version 11.2.1.3.1 the database servers are imaged with Oracle Linux 5.5 For existing Database Machines the database servers can be upgraded manually using standard Oracle Linux upgrade procedures.

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Oracle Linux 5.5 With Exadata software version 11.2.1.3.1, the operating system in the Exadata servers is upgraded to Oracle Linux 5 5.5. 5 This version of Oracle Linux includes security fixes not included in the previous version used in Exadata and Database Machine, Oracle Linux 5.3. Similarly, for Database Machines shipping from the factory with the Exadata software version 11.2.1.3.1, the database servers are also imaged with Oracle Linux 5.5. However, note that for existing deployments, when applying the Exadata 11.2.1.3.1 patch the operating system on the database servers is not be automatically upgraded to Oracle Linux 5.5. The database servers continue to use Oracle Linux 5.3 until the customer chooses to upgrade them manually, ll using i standard t d d Oracle O l Linux Li upgrade d procedures. d

Exadata and Database Machine Administration Workshop A - 7

Enhanced operating system security • • •

Unnecessary Linux packages are not installed Unnecessary Linux services are disabled Firewall functionality (using iptables) is available – Implemented with pre pre-configured configured rules on Exadata servers – Available but not implemented on database servers

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Enhanced operating system security In addition to upgrading the operating system on the Exadata servers to Oracle Linux 5.5, security has been enhanced by eliminating the installation of unnecessary Linux packages. packages Also, unnecessary services like nfs, ftp and telnet are disabled by default to minimize security vulnerabilities. In addition, firewall functionality (using iptables) with pre‐configured rules is implemented on the Exadata servers. For Database Machines shipping with version 11.2.1.3.1, or for freshly re‐imaged database servers, the number of Linux packages that are pre‐installed is also minimized and unnecessary services are disabled by default. Customers also have the ability to configure and d enable bl fifirewallll ffunctionality ti lit using i iiptables t bl on th the d database t b servers.

Exadata and Database Machine Administration Workshop A - 8

Pro-active disk quarantine •

If a disk or flash module fails, or is about to fail: – Associated grid disks are immediately dropped from corresponding ASM disk groups —

and from Exadata Smart Flash Cache for flash modules

– ASM re-creates lost data using mirrored extents sooner – Greater protection from potential data loss associated with multiple overlapping disk failures

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Pro-active disk quarantine With Exadata 11.2.1.3.1, if the cell server detects a failed disk or a disk predicted to fail soon, it will pro‐actively pro actively drop the associated grid disks from the ASM disk groups they belong to rather than waiting for the time specified in the DISK_REPAIR_TIME disk group attribute (3.6 hours by default). This ensures that ASM can immediately start to re‐create the data stored on the failed disk using the mirrored extents on other disks, thus preserving the data redundancy in the ASM disk group. This behavior improves the overall availability of the system because previously the loss of another disk during the DISK_REPAIR_TIME window could have resulted in the loss of data. Similarly, Si il l if th the cellll server d detects t t a flflash h module d l th thatt h has ffailed, il d or iis about b t tto ffail, il it d drops th the flash module from the Exadata Smart Flash Cache and drops any associated flash-based grid disks from the corresponding ASM disk groups.

Exadata and Database Machine Administration Workshop A - 9

Other new features •

Alerts for disk failures on database servers – SNMP traps can be configured to alert when a disk fails, or is about fail, on a Database Machine database server

•

SNMP alerts from InfiniBand switches – Upgraded InfiniBand switch software facilitates SNMP traps to enable improved monitoring of the switch

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Other new features In conjunction with the Exadata 11.2.1.3.1 update, it is now possible to configure and send SNMP traps when a disk fails fails, or is about to fail fail, in the Database Machine database servers servers. Also, in conjunction with this update the InfiniBand switch software in upgraded to version 1.1.3.2 which includes software ILOM functionality. This new functionality sends SNMP traps, about the overall health of the InfiniBand switch, which can be used to better monitor the InfiniBand switch.

Exadata and Database Machine Administration Workshop A - 10

Summary In this lesson, you should have learned how to: • Describe the new features of Exadata and Database Machine introduced in update release 11.2.1.3.1

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Exadata and Database Machine Administration Workshop A - 11