Optimizing Caches and Cfg Settings

TUNING BLOCK STORAGE ESSBASE Richard Sawa, Oracle Corporation

www.odtug.com

1

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Table of Contents Introduction....................................................................................................................................................................................................... 3 Audience............................................................................................................................................................................................................ 3 Essbase Cache Optimization.......................................................................................................................................................................... 3 Preamble ........................................................................................................................................................................................................ 3 Primary Areas for Optimizing BSO......................................................................................................................................................... 4 Block and Cache Review................................................................................................................................................................................ 4 BSO Cache Review..................................................................................................................................................................................... 4 CALC CACHE........................................................................................................................................................................................ 4 OS File System Cache/Essbase Data File Cache .............................................................................................................................. 4 Data Cache ............................................................................................................................................................................................... 6 Index Cache ............................................................................................................................................................................................. 6 Dynamic Calculator Cache.................................................................................................................................................................... 6 Dynamic Calculator Cache Compressed Buffer................................................................................................................................ 6 Measuring BSO Performance ........................................................................................................................................................................ 6 BSO Concept Review..................................................................................................................................................................................... 6 Outline Tuning ................................................................................................................................................................................................. 9 A Sparse-Dense Tuning Methodology in Brief...................................................................................................................................... 9 Ratio of Block Density to Block Count............................................................................................................................................ 10 Sparse/Dense Settings Revaluated..................................................................................................................................................... 10 The Outline ................................................................................................................................................................................................. 10 Member Tags and Data Storage......................................................................................................................................................... 11 Note on Overall Memory Use and the Block................................................................................................................................... 12 Aggressive use of Dynamic Calc Member Tags – A Cautionary Note....................................................................................... 12 Two forms of the Block....................................................................................................................................................................... 12 Other Basic Outline Considerations.................................................................................................................................................. 13 Handling Consolidation and Formulae ............................................................................................................................................. 13 Consolidation Types ................................................................................................................................................................................. 14 Natural Consolidations......................................................................................................................................................................... 14 Special Case Consolidations............................................................................................................................................................... 14 Summary ......................................................................................................................................................................................................... 14 Appendix A – Briefly Annotated ESSBASE.CFG File Settings........................................................................................................... 15 JVM and Authentication Modules.......................................................................................................................................................... 15 Shared Services.......................................................................................................................................................................................... 15 ASO Settings.............................................................................................................................................................................................. 15 ASO and BSO Settings............................................................................................................................................................................. 15 Other ASO/BSO settings.......................................................................................................................................................................... 16 BSO Settings.............................................................................................................................................................................................. 18

www.odtug.com

2

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Introduction This paper augments the Oracle Essbase presentation entitled Optimizing Caches and CFG Settings, delivered at the ODTUG user conference on June 18, 2008 in New Orleans. Together, they provide a near-complete though relatively high-level technical discussion of BSO cube optimization. Since its arrival to the market in 1993, Oracle Essbase has been providing end-users with “analysis at the speed-of-thought” functionality. Closely coupled with advanced analytic capabilities are full write-back support and an advanced calculation language. Essbase permits Organizations to instantiate for analysis nearly every conceivable business algorithm within a centrally managed multi-dimensional storage structure. One side effect of Oracle providing these capabilities was for users to overload data into BSO cubes. All multi-dimensional storage structures suffer from sparse matrix explosion, and it doesn’t take long for these structures to choke on sparseness. BSO Essbase is no exception to this. The BSO sword is double-edged. One edge provides users the combination of functionality uniquely offered by Essbase. On the other edge are sparse matrix management requirements that are exacerbated by sub-optimal cube designs. Left unchecked BSO cubes are difficult if not impossible to manage: data load and cube calculation times no longer meet batch window requirements, cubes become untenably large, and end-user response times degraded unacceptably. Tuning and optimizing BSO cubes bring together maximum performance characteristics with minimal storage requirements. These cubes provide the mo st efficient batch and administrative routines along as well as the fastest end-user response times. Once optimized, IT can be sure that their cubes deliver maximum end-user benefit while exhibiting maximum hardware utilization and performance.

Audience Reading this paper, IT Professionals and Managers will gain a high-level understanding of what is required to optimize a BSO Essbase or System 9 Planning environment. Technicians will develop a solid though basic understanding of the skills requisite to designing, optimizing and maintaining BSO cubes.

Essbase Cache Optimization Cache optimization essentially involves adjusting the Essbase memory buffers in support of reading and writing BSO cubes. Buffer tuning is not done in isolation and once for all. Like most important computer software tasks, tuning the buffers is an iterative activity. Altering one component to a cube, like, for example, adding members to a dense dimension, can immediately impact performance and require adjusting the caches. Hence, a discussion of tuning and optimizing BSO cubes really involves a discussion of many of the best practices for implementing BSO Essbase. BSO Tuning needs to optimize for three things: ? data loading and batch calc processing ? business rule/calc script processing ? end-user query responsiveness At times, optimization techniques that support one objective will conflict with the objective for another. An extreme example is bad practice of tagging many sparse members dynamic -calc in order to defray the amount of time it takes to execute business rules or calculation scripts. Deferring excessive block creation to query time can be expected to result in very poor query performance, and can possibly destabilize the Essbase ESSSVR application process. In order to lay a common foundation for a buffer and cache tuning and optimization methodology we will review some of the fundamental concepts of BSO Essbase. The review will not be exhaustive. It assumes at least a basic understanding of multidimensional concepts in general and of Oracle Essbase in particular.

Preamble Essbase developers are usually power-users, and are responsible for designing and implementing the Essbase model to support end-user requirements. Essbase Administrators are usually IT personnel and responsible for providing back-end software and possibly infrastructure support for these BSO applications. Some customer environments keep separate to a fault these two technical domains. Experience, on the other hand, regularly and sometimes painfully teaches us that the more

www.odtug.com

3

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

each of these domains understands of the other, the greater the chance for a successful Essbase implementation. Whereas having IT and Business Unit end-users in synch with each other does not guarantee success, failure is all but guaranteed by keeping them far apart from one another.

Primary Areas for Optimizing BSO There are three primary areas to the tuning of BSO Essbase: ? tune the outline (sparse/dense, member tags, formulae, etc) ? tune database settings (caches, essbase.cfg file, etc.) ? tune calculations (calc scripts and business rules) This document concentrates on tuning the outline as a prerequisite for optimizing database cache settings. It will touch briefly on calc optimization insofar as calc requirements might need to be supported by a special outline configuration. Outline sparse-dense configurations directly impact the size and number the data blocks. The data block and block index entry are the two of the fundamental units of storage for a BSO cube. Moving block structures in and out of RAM is what cache optimization benefits. For this reason we begin the discussion here.

Block and Cache Review The caches are the memory objects that Essbase uses to manage index and data pages. In order to understand how to configure these buffers, we must understand the concepts that surround the fundamental units of BSO storage: the data block and the index. Every important aspect of BSO Essbase is impacted directly by data block design decisions, and the caches are no exception. A thorough understanding of data block concepts that support tuning the caches underpin and enhance the additional activities required for outline and calculation script tuning. The data block is the structure that is comprised of the dimensions tagged as dense in the Essbase outline (.otl file). The block is defined by the intersections of all dense dimension member intersections. When data blocks and index entries are persisted to disk, the data blocks are compressed and stored within ESS*.PAG files and index entries are stored within ESS*.IND files. There are two views or types of the data block: ? the physical or stored block ? the fully expanded or logical block The physical block is compressed by Essbase before being persisted to disk. The logical block is a RAM object and includes the physical block cells plus all block members that are computed dynamically.

BSO Cache Review CALC CACHE The buffer called the CALC CACHE is an area of memory that is set aside by Essbase during aggregations (i.e. calculations that create parent data blocks) and is used to make block creation more efficient. The CALC CACHE does not contain data blocks or index entries. The CALC CACHE is a specialized memory cache that is provided in the PowerPoint presentation/slides entitled Optimizing Caches and CFG Settings. OS File System Cache/Essbase Data File Cache Fig. 1 depicts the caches that support Buffered I/O. When configured using Buffered I/O, the cache into which data is read, and from which data is written, is the OS File System cache. Disk I/O is handled by the operating system.

www.odtug.com

4

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Fig. 1 Buffered I/O Essbase

OS

Index Cache

Dynamic Calc Cache

OS File Cache

Physical Disk

Essbase Kernel

Data Cache

Dynamic Calc Cache Compressed Buffer

Fig. 2 depicts Direct I/O. Here, the OS file system cache is bypassed, and Essbase manages I/O directly using the data file cache. Data blocks are compressed in this cache. It is important to take into consideration the size of the block that is stored in each of the data buffers when allocating memory to the Essbase caches. Note that, when using Direct I/O, index pages bypass the data file cache and are written directly to disk by Es sbase. The data file cache represents an additional cache to configure if you decide to use Direct I/O.

Fig. 2 Direct I/O Essbase Index Cache

Dynamic Calc Cache

Physical Disk

Essbase Kernel

Data File Cache

Data Cache

Dynamic Calc Cache Compressed Buffer

www.odtug.com

5

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Data Cache The physical data cache holds the data block in its uncompressed form, and blocks are moved into and out of this cache on demand. The expanded block size is equal to 8-bytes multiplied by the number of stored cells in the block. Memory is allocated on demand as blocks are read, written or created into the data cache. Index Cache Index entries are moved into and out of the index cache on demand. Index pages are not compressed. In contrast to the data cache, the index cache is fully allocated in memory when the database is loaded. Dynamic Calculator Cache The data block is fully expanded (logical block) within the dynamic calculator cache to accommodate computing dynamic calc members. The fully expanded block size is equal to 8 bytes multiplied by the number of cells in the block. Dynamic calculations are optionally used in BSO. The list of dense dynamic calc functionality includes dynamic calc member tags as well as the use of dynamic time series. In cubes where any dense dynamic calc functionality has been implemented, every data block retrieved is fully expanded within the dynamic calculator cache. This is true regardless of whether a dynamic calc member has been requested. Whether Essbase computes every dynamic calc member upon retrieval, or only those requested, depends upon the nature of the functions within member formulae. The presence of functions that can only be resolved at runtime (e.g. @CURRMBR) can cause Essbase to compute every dynamic calc cell in the block upon retrieval. Dynamic Calculator Cache Compressed Buffer Administrators are provided the optional ability to manage server memory by configuring Essbase to be able to compress and move blocks from the dynamic calculator cache into the dynamic calculator cache compressed buffer. A compressed form of the fully expanded logical block can be temporarily held or set aside within this buffer.

Measuring BSO Performance Essbase provides access to a large number of cube statistics that can be used to measure and monitor cube performance. The decision of which metrics to use is left up to the individual developer or admin to decide. The methodology should be to correlate cube metric thresholds, like the average fragmentation quotient, directly with performance characteristics. Once having made the correlation, the statistic threshold can be used to initiate an appropriate admin process. In the final analysis then, the key performance indicator really is time. How long are things taking to do? Users concentrate on the following metrics: How long does it take to calculate a cube? How long does it take business rules to complete? How long does it take queries to complete? These are the most valuable performance metrics for IT also. Cache tuning and optimization seeks settings that enable data to be loaded, calculated and retrieved in the shortest possible time.

BSO Concept Review The BSO storage structure is an array (or matrix). In BSO, we talk about intersection points and cells, and we talk about data blocks as opposed to data rows. BSO Essbase essentially converts business elements to members of an array and stores floating point numbers at intersection points defined by one member from each dimension in the array as shown in Fig. 3.

www.odtug.com

6

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Fig. 3 BSO Array Visualization OBLIGATION

OHIO

R798A740UQHNGE000049744123

OBL ABSEUIESO

33.5 10/1992

JTPA TITLE IIA PROGRAM 106B

Arrays are inherently empty, or sparse, structures. What sparseness means is that the number of intersection points in the array that contain data is significantly smaller than the total number of intersections defined by the array. Let’s walk through a trivial example. We can easily conceive a company that sells every product in every region of operation. This very primitive ‘business model’ would currently contain only the two dimensions of product and geography. In terms of sales, they completely contain data: every product has sold in every region. When we start adding dimensionality to the business model, the concept of sparseness becomes manifest. If we track sales by the day, we suspect that we probably don’t sell every product, in every region, on every day. If we want to track sales for every customer by the hour, it becomes easier to see that no customer purchases every product, in every region, every hour, of every day. The point here is that by definition an array will contain a cell for every possible combination even when there won’t be anything to put in the cell. http://publib-b.boulder.ibm.com/abstracts/sg246138.html?Open is a link to a discussion to the IBM Redbook DB2 OLAP Server Theory and Practices that contains a discussion of sparse matrix management concepts and how they relate to Essbase.

Fig. 4 What is an array? array (10, 15, 3) The array (10,15, 3) produces a 3 dimensional array with 450 (10*15*3) cells, and so on…

The size of an array gets very large, very quickly. For example, consider a hypothetical cube matrix with the following dimensionality: Sample_Array (172, 21, 27, 32, 209, 32765)

www.odtug.com

7

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Whereas this array actually represents a rather small cube in Essbase terms, the six dimensions 21,370,660,375,680 intersection points (i.e. 172 * 21 * 27 * 32 * 209 * 32765). If each cell occupies eight bytes of memory, how do software applications model a matrix of this size? The solution the developers of BSO Essbase came up with was to modularize the matrix by mapping it in the index (sparse) and data block (dense) storage structures.

Fig. 5 Sparse vs. Dense Sparse

Dense

P r o du cts

Time X X X

Measures

Markets

X X X X X

X

X X X X X X X X

X X X X

X X X X

Fig. 5 tries to convey the notion that sparse dimensions have few intersection points containing data whereas dense dimensions contain more intersections with data. Fig. 6 shows that, in Essbase, dense dimensions comprise the data block and are materialized within ESS*.PAG files, and sparse dimensions comprise the index and are materialized in ESS*.IND files.

Fig. 6 Index and Block Review Dim Members Type Structure Disk ====================================================== Measures 172 DENSE Block ess*.pag Time 21 DENSE Block Mat 27 SPARSE Index Str 32 SPARSE Index ess*.ind Prod 209 SPARSE Index ORG 32765 SPARSE Index

• Block size: (21 * 172) 3612 cells * 8 28,896 bytes (~28.2 k)

bytes/cell

=

Dense dimensions define the unit of storage called the block. The size of the block in the above example is 21*172 = 3612 cells. In Essbase, each cell consumes 8 bytes, so this data block consumes 28,896 bytes. The computer is more able to move sections (block and index pages) of the matrix in and out of memory with efficiency. An important BSO corollary is that where no index entries exist, no data blocks are created. This is how BSO Essbase “squeezes” sparseness out of the model. The index nodes in Fig. 7 (e.g. Cola->East) are entries that would be found within the index (ESS*.IND) whereas the (cubelike) blocks (i.e. the Scenario->Measures->Time structures) would be found within the data (ESS*.PAG) files.

www.odtug.com

8

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Fig. 7 Data Block and Index Overview

Measures

Tim e

May Apr Qtr1 Mar Feb Jan

Index nodes

Profit COGS Margin Mkt Exp. Payroll Misc Tot. Exp. Sales

.

Actual Budget

May Apr Qtr1 Mar Feb Jan

Measures

.

Actual Budget

Var.

Var%

Scenario

C o l a-> N e w Y o r k

Var%

Cola -> E a s t

Ti m e

Ti m e

Measures

Profit COGS Margin Mkt Exp. Payroll Misc Tot. Exp. Sales

Var.

Scenario

May Apr Qtr1 Mar Feb Jan

Profit COGS Margin Mkt Exp. Payroll Misc Tot. Exp. Sales

.

Actual Budget

Var.

Var%

Scenario

Cola -> F l o r i d a

The most important decisions that impact overall cube performance are the dense/sparse settings. There are several methodologies for determining the optimal block configuration. Ultimately it will be left for the developer and admin to test possibly several block configurations using real data in order to determine which is the most efficient configuration for a given application. We will briefly outline one methodology below and briefly discuss perfectly valid reasons for choosing not to implement the most efficient block configuration.

Outline Tuning A Sparse-Dense Tuning Methodology in Brief Aggregations are generally the most time -consuming calculation activity for a BSO cube. Essbase permits loading data to any combination of stored levels in a cube. However, in the simplest scenario, data is loaded to the leaf level of the cube hierarchies causing the bottom or leaf level data blocks to be instantiated. An additional calculation procedure is then initiated whereby Essbase creates upper, or parent data blocks across all of the dimensional hierarchies. The number of upper blocks that are created is the Cartesian product of all of the members of all of the dimensions where data exists. The most efficient block configuration will result in least amount of I/O and result in: 1.

The fastest data load time

2.

The fastest aggregation time

3.

The smallest cube size

The objective is to isolate as efficiently as possible, the block configuration that results in the fastest aggregation time. The advent of 64-bit Essbase removes the practical limitations for block size that accrued to 32-bit platforms, and the number of block configurations that can be implemented increases in a 64-bit environment. The sparse-dense methodology briefly outlined here provides one way to determine which sparse dense block configurations are worth testing. There are six iterative steps: Step 1: For each block configuration, set all dense members that have children or that contain a formula to dynamic calc non-store. This not only reduces the block size and disk storage requirements and enables more dimensions to be considered candidates for inclusion within the block. Step 2: Load real and representative data to the database. By real is meant the requirement that the data is sourced from operational data stores. By representative is meant the requirement that the data set reflects either a fully loaded database or that the slice of data that is loaded has accurate and predictable growth characteristics. Deviating from either requirement renders these tests useless. Step 3: Retrieve from database statistics the average block density metric. Step 4: Retrieve from database statistics the number of blocks generated.

www.odtug.com

9

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Step 5: Calculate the ratio of block density to the number of blocks generated. Below is explained why the highest ratio indicates the configuration where the cube generates the fewest blocks that contain the most data. Step 6: Iterate through all appropriate cube sparse/dense configurations. The loaded cube is not aggregated during these tests. The assumption is that the number and density of the blocks generated at level-0 are representative of the number and density of blocks after aggregating the cube. This might not always be true especially in instances where data blocks are generated as a result of a business algorithm (i.e. calc script or equation). Not calculating the database enables you to reduce the amount of time that it takes to produce the ratio down to the amount of time it takes to programmatically iterate loading data through the set of possible database configurations. This has three effects: 1.

It drastically reduces the amount of time it takes to compute the ratios.

2.

The amount of disk space required is greatly reduced.

3.

The requirement that tests be performed on a development server is all but eliminated.

The design objective for sparse/dense settings is to enable the developer to reflect the denseness of the data within the data block, and eliminate sparseness through the index. This is done essentially to achieve the most efficient database configuration to support the default calculation (or aggregation) of the cube. We used to instruct developers to configure the cube so as to “create the fewest number of the most dense data blocks”. Optimal sparse/dense settings are associated with the most efficient use of auxiliary storage. Optimal disk storage infers minimal auxiliary storage requirements (i.e. disk I/O), which, in turn, requires the minimal time for database calculation processing. Ratio of Block Density to Block Count Step 5 above refers to computing a ratio of block density to block count. This ratio reflects the relation between the density of the data block and the number of blocks produced: as the density increases, the ratio increases, as the block count decreases, the ratio increases. For any given database then the highest ratio theoretically represents the most optimal sparse/dense configuration. The computation of this metric across different block configurations enables you to isolate the configuration that will result in generating the fewest and most dense blocks. Please note that the ratio is meaningful only in relation to the values computed for the ratio across different block configurations in the same cube. Inter-cube ratio comparisons are not useful. Sparse/Dense Settings Revaluated There are times when cube processing requires the manipulation only of specific slices of the cube. In these situations, it is regularly found that the amount of time that is spent performing work on slices outweighs the work performed to aggregate all data. In these instances, the designer will need to opt away from implementing the fastest database configuration supporting aggregations to select a configuration that provides support for managing cube slices. For example, scenario dimensions are generally associated with the density of the data, and “want” to be tagged dense. However, in a Planning-type cube, the developer will tag the scenario dimension sparse to support the requirement to copy data between different scenarios. BSO Essbase provides users with the ability to copy budget data to a new scenario in order to enable what-if analysis and forecasting on the new scenario. This is facilitated by the use of the FIX statement within a script to focus efficiently on a particular slice of the cube.

The Outline We have already referred to the potential impact of cube dimensionality. The questions now are: How many? How large? How deep? Essbase training will have taught cube designers to apply analysis rules when creating cubes. These are: 1.

Minimize the number of dimensions

2.

Examine dimension combinations

3.

Avoid repetition

4.

Eliminate inter-dimensional irrelevance

www.odtug.com

10

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

5.

Sawa

Examine consolidations: Dynamic Calc/Label Only

These practices lay the foundation for outline tuning. Member Tags and Data Storage It is an Essbase best practice to keep the size of the cube to a minimum by storing data only when necessary. In Fig 8, the block contains 6*4*3 = 72 cells:

Fig. 8 Member Considerations: The Block No Label Only and no Dynamic Members

Block size: 72 cells

marketing Actual Budget Forecast Var VarPer Scenario

cogs sales jan

feb

mar

q1

To minimize the amount of data that is persisted by Essbase use the Label-Only tag on navigational members wherever possible. Tagging Scenario Label-Only reduces the block size in Fig. 9 to only 5*4*3 = 60 cells:

Fig. 9 Using Label Only Block size: now 60 cells

marketing Actual Budget Forecast Var VarPer

cogs sales jan

feb

mar

q1

Whereas the label-only tag has some storage/performance overhead, the use of dynamic calc tags can have a dramatic impact on performance. The dynamic calc tag was introduced to remove the calculation of certain intersections completely from batch and procedural (i.e. scripted) calculations to be processed by users at query time. Strictly adhered to, there should be no

www.odtug.com

11

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

CPU cycles devoted to dynamic calc members during scripted calculations. Many BSO implementations unfortunately break this rule by invoking dynamic calculations in business rules or calc scripts. Strictly speaking, this is a bad Essbase practice. Using dynamic calc non-store tags within the example has reduced the Fig. 10 block size from 72 down to 3*3*3 = 27 cells, which is almost a 200% reduction in auxiliary storage requirements:

Fig. 10 Using Dynamic Calc Tags Upper value and formula members made dynamic Place holder members made label only

Block size now 27 cells marketing cogs Actual Budget Forecast

sales jan

feb

mar

Note on Overall Memory Use and the Block Fig. 9 depicts the 60-cell block that will be materialized within the dynamic calculator cache and Fig. 10 depicts the 27-cell block that will be materialized in the data cache. In this example, a block materialized in the data cache will consume 216 bytes (27*8), and the “same” block will consume 480-bytes (60*8) in the dynamic calculator cache. This “same” block occupies 696 bytes of RAM within Essbase caches, and that the true overall amount of RAM occupied by a block will also include overhead for Label-Only, etc. tags as well as the compressed block held either in the file system or data file cache. Aggressive use of Dynamic Calc Member Tags – A Cautionary Note An aggressive use of the dynamic calc tag can create a large difference between the size of the block in the dynamic calculator cache and the size of the block in the data cache. Successful implementations have seen a ratio of greater than 10:1. Dynamic calculations occur in the context of end-user querying, so it is very important to understand query behavior fully to optimize the dynamic calculator cache in support of query activities. Determining the average or worst-case working-set to support end-user query requirements is one of the best ways to quantify end-user query behavior. However, overall operational resource requirements are the combined impact of query and business rule/procedural calc activities. Resource requirements to process dynamic members within procedural scripts are unpredictable. Strictly speaking, these dynamic calc dependencies represent bad practice. Moreover, such dependencies should be expected to deliver very poor response times and can lead ultimately to application instability. Often Essbase development teams learn what the real resource requirements of an Essbase application are when too late and they are already in Production. Rigorous systematic UAT and stress testing are mandatory procedures for Essbase. And this is be nothing new for IT. Two forms of the Block There are two forms of the data block. The stored block contains data that is persisted to disk. This version of the block is compressed into the OS file cache or Essbase data file cache before it is written out to disk. The fully expanded version of the block requires additional memory to compute dynamically calculated members. This version of the block is held in the dynamic calculator cache and is compressed within the DCC compressed buffer when Essbase has been configured to do so.

www.odtug.com

12

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

Fig. 11 Two forms of a data block: Physical Block All stored cells + Overhead Found in Data Cache Compressed in Data File Cache

Fully Expanded Block Stored cells + dyn calcs + dyn time series + overhead Found in Dynamic Calculator Cache Compressed in DCC Compressed Buffer

Other Basic Outline Considerations We will briefly discuss other outline design issues here to the extent that they impact how much data is stored, and what data is dynamically computed. Organizational hiera rchies are embedded in the Essbase outline and most data ispre-computed and persisted to disk. How a cube performs will largely depend upon the inter-relation between how many dimensions there are, how large the dimensions are (i.e. total members) and how deep the hierarchies in the dimensions are. Business algorithms too can be embedded in the outline. Additional performance overhead accrues to cubes that need to embed business logic in formulae and calculation scripts. The performance characteristics of any given cube will be the combined impact of the use of member tags, outline consolidation (unary) logic, the use of formulae (and or calc scripts/business rules). Each of these components is impacted directly by dense/sparse settings. Query performance generally improves, as more of the cube is pre-computed. The dynamic calc member tag was one of the tools that Essbase programmers provided for designers to trade-off the cost of calculating data values on-the-fly for a reduced batch calculation. Planning and Planning-type applications are characterized by batch as well as end-user initiated calculations. In a Planning environment where users are as apt to be running calculations, as they are to be executing queries, the use of dynamic calc member tags needs to be carefully controlled. Many unstable Planning applications have been traced to the inappropriate use of dynamic calc tags. Handling Consolidation and Formulae Above we mentioned that as more data is persisted in the cube, query performance increases. The objective solely to guarantee optimal query performance is achieved by pre-calculating all database values. This would preclude the use of dynamic calc tags, and severely limit the overall number of members and dimensions in the cube. In a Planning-type cube environment, a pre-calculated solution is untenable because what is required for users is an environment where query response time is not more important than business rule response times. If either the queries or the business rules take too long, the end-user experience is diminished. Follow these general rules-of-thumb when implementing dynamic calc tags and member formula: 1.

Place member formulae in the outline (as opposed to calc scripts)

www.odtug.com

13

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

2.

Sawa

Try to ensure that inter-dimensional calculations are resolved within the data block. This objective will impact sparse-dense settings.

Both of these rules-of-thumb are difficult to follow rigorously in BSO applications. Planning-type BSO cubes require procedural calculations and the use of sometimes elaborate calc scripts.

Consolidation Types I would like to propose a way to conceive the consolidating of BSO cubes by dividing consolidation types into two general categories. The first category represents cubes that are ideal candidates for using the Aggregate Storage Option (ASO) of Essbase. Natural Consolidations A natural consolidation of an Essbase cube is invoked by issuing a CALC ALL (or the equivalent CALC DIM or AGG statement) consolidation. These can be of two types. First, the consolidation process performs only aggregations, and we can refer to it as Type-A. Type-A consolidations are supported by an outline that contains no member formulae. Block creation is regular and predictable as all business logic is expressed by using unary operators. The cube is calculated in outline order according to the default calculation algorithm. Type-A examples are most properly implemented using ASO Essbase. A Type-B natural consolidation occurs when an outline has dense member formulae that reflect some of the business logic. Ideally, any cross-dimensional references are resolved within the block and performed as they appear in outline order. The cube is calculated in outline order according to the default calculation algorithm. Block creation and consolidation is regular and predictable. A cube that can be supported by a Type-A consolidation is a very strong candidate for using ASO keeping in mind that some formula functionality using the classic BSO calculation language might not be efficiently reproducible using MDX formulae. These so-called natural consolidations represent optimal configurations for the computation of aggregations. Reporting cubes are excellent candidates for natural consolidations. It would be considered a best practice from an overall Essbase perspective to convert cubes that do not involve write-back or procedural calculation to the aggregate storage option using MDX to replicate any formula requirements. ASO cubes are very efficient at storing data. Special Case Consolidations Special case consolidations require different cube configurations than those that support the simply computation of aggregations. They require special sparse-dense configurations because of focused and potentially procedural complexities involved in calculations and aggregations. These cubes implement very advanced Essbase functionality and point to why optimizing Planning cubes for performance is a complex activity. Special case consolidations very often have formulae on sparse members that contain cross-dimensional references to other sparse member combinations. Block creation during the consolidation of these cubes is very often not regular and not predictable. In the final analysis, some cube configurations will be optimal not according the best sparse/dense configuration but according to requirements specific to the particular business model at hand.

Summary This paper provided a high-level technical discussion of the concepts involved in tuning and optimizing block storage option (BSO) Essbase cubes. We noted that every important aspect of BSO Essbase is impacted directly by data block design decisions, and that the caches are no exception. An understanding of the details of the BSO storage array is quintessential to understanding how to optimize BSO Essbase in general, and configure caches in particular. The impact of sparse-dense settings on cube aggregating, procedurally calculating and querying aspects of performance were briefly discussed and you should now be aware of effective use of outline member tags to enhance end-user experience and defray the cost of auxiliary storage I/O. Thank you very much for reading this paper. If you have any questions, I can be reached at [email protected].

www.odtug.com

14

ODTUG Kaleidoscope 2008

Appendix A – Briefly Annotated1 ESSBASE.CFG File Settings Essbase System 9.3.1 Configuration File Please refer to the Technical Reference and Database Administrator’s Guide sections of Infornation Map for detail descriptions of syntax and use.

JVM and Authentication Modules The following entry specifies the full path to JVM.DLL & external authentication location JvmModuleLocation /opt/hyperion/common/JRE-64/IBM/1.5.0/bin/classic/libjvm.so AuthenticationModule CSS http://"server:port"/interop/framework/getCSSConfigFile

Shared Services SharedServicesLocation - "server port" Shared Services User Management Server name & port allocation CSSREFRESHLEVEL - Specifies when Essbase & Shared Services refresh security info for all users, groups, & applications CSSSYNCLEVEL - Specifies when Essbase & Shared Services synchronize info for a user & any related groups SHAREDSERVICESREFRESHINTERVAL - Specifies periodic, automatic refreshes of Analytic Services security information from Shared Services.

ASO Settings MAXFORMULACACHESIZE - Specifies max size of ASO formula cahce to be made available for calcing members with formulas. Set by App/DB. PRELOADALIASNAMESPACE TRUE - Determines whether namespace for alias names is preloaded PRELOADMEMBERNAMESPACE TRUE - Determines whether namespace for member names is preloaded TARGETASOOPT - Optimize large queries from SS add-in, MDX, or report writer across a transparent partion with identical the source outline & target are identical defintion area.

ASO and BSO Settings AGENTDELAY - Specifies the number of seconds an Agent thread waits to perform a specific action AGENTTHREADS - The number of threads the Agent may spawn – varies by 32 or 64 bit platform AGTSVRCONNECTIONS - Specifies the maximum number of threads the Essbase Server can spawn to allow the first connection to an application and database, negotiated between the Agent and server. AGENTTHREADS - Specifies max number of threads Analytic Server can spawn to allow first connection to an app or db, negotiated between Agent & server SERVERTHREADS - Overrides the default value for numbr of threads an app may spawn. AGENTDISPLAYMESSAGELEVEL - Sets the msg type displayed in the server console, AAS. AGENTLOGMESSAGELEVEL - Sets the msg types to be written to Analytic Server log, essbase.log. AGENTPORT - Specifies the port that the Agent uses (default is 1423). 1

Grouped, orderd alphabetically and compiled by Paul Lilford, United Healthcare Group, December, 2007.

www.odtug.com

15

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

SERVERPORTBEGIN - This setting specifies the first port number essbase application process (ESSSVR) moved from 32K range SERVERPORTEND - The highest value for the ESSSVR process. If the value is unavailable, the app process will fail PORTINC - Specifies the value of the increment in between port numbers used by the Analytic Services agent process (ESSVR), default is 1 PORTUSAGELOGINTERVAL - Writes number of ports in use to Essbase.log every 60 minutes CALCPARALLEL - (default 1 for BSO / 2 for ASO) Number should be applied by app & db name, if used in cfg or use in calc scripts DLSINGLETHREADPERSTAGE - Instructs Essbase to use single thread per processing stage, or use values in two commands below. Used to impact load performance. DLTHREADSPREPARE - How many threads Essbase may use during data load prep stage, which organizes data in memory prep for storing to data blocks. DLTHREADSWRITE - How many threads Essbase may use during data load process that writes blocks to disk NETBINDRETRYDELAY - Specifies amount of time the app server retries on bind failure. NETDELAY - Specifies network request delay time. NETRETRYCOUNT - Specifies number of attempts for network connection before failing & reporting error. PIPEBUFFERSIZE - Sets the size of buffer used for communication between Spreadsheet Extractor & Report Writer

Other ASO/BSO settings AGENTDESC - When config utility is used to register Essbase Agent as Windows service, text entered in Service Name Identifier field is stored as AGENTDESC in Essbase.cfg. CLEARLOGFILE - Determines whether Essbase & app logs are overwritten CRASHDUMP - Sets whether or notEssbase saves core dump file with abnormal termination of agent or server process (UNIX only). DATAERRORLIMIT - Determines number of records that can be written to an error log during a dataload or dim build operation. DELIMITEDMSG - Seperate fields when writing logs files, use default (~) character. DELIMITER - Delimits Essbase & app logs using one of five allowed symbols (~ ^ * : &) DEPLOYMENTID - Uniquely identifies a specific deployment of Hyperion product componets. DISPLAYMESSAGELEVEL - Sets level of messages displayed in app console (AAS) DISABLEREPLMISSINGDATA - Instructs Essbase not to replicate #MISSING values to target partition, improves performance but may result in less accurate data. ESSBASELICENSESERVER - Specifies the location of the License Server or license file - removed need for by Oracle EXCEPTIONLOGOVERWRITE - Determines whether Essbase overwrites the existing exception log or creates a new exception log. GRIDEXPANSION - Improves performance for queries on transparent partitions. GRIDEXPANSIONMESSAGES - Sets whether grid expansion-related messagers are displayed to Spreadsheet Add-in users & written to app log. HAENABLE - Sets whether members can be retrieved from a Hybrid Analysis Relational Source HAMAXNUMCONNECTION - Sets max number of connections per db that Essbase can keep active against the relational db. HAMAXNUMSQLQUERY - Sets max number of SQL queries that can be issued against the fact table(s) in the relational db per Essbase query session. www.odtug.com

16

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

HAMAXQUERYROWS - Sets max number of rows that can be returned per SQL query issued on behalf of Essbase query. HAMAXQUERYTIME - Sets max time limit per query for SQL queries from a HA relational source (ignored against Oracle datasource, warning in log) HAMEMORYCACHESIZE - Determines amount of memory that is reserved to cache queried members from a HA relational source. HARAGGEDHIERARCHY - Enables support of null values in columns of dim tables that are used to creat dimensions for HA enabled outlines. HARETRIEVENUMROW - Sets max number of rows resulting from a SQL query to process at one time. HASOURCEDSNOS390 - Set to TRUE if the DB2 data source for HA resides on an OS/390 system. LOGINFAILUREMESSAGEDETAILED - Provides detailed error messages on user login faliure. LOGMESSAGELEVEL - Sets the level of messages written to app log (INFO,WARNING,ERROR). MAXLOGINS - Sets a limit on the number of user sessions that can be connected to Essbase at one time. MEMSCALINGFACTOR - Enables you to set data cache and data file cache sizes to values greater than 4GB. Applies when running Analytic Services on 64-bit platforms. Set by App/DB, use carefully. NETTCPCONNECTRETRYCOUNT - Specifies number of attempts a client will make to connect to a TCP/IP network before failing & reporting an error (default is 3) NODENAME - Identifies the host computer where Essbase application process runs. NOMSGLOGGINGONDATAERRORLIMIT - Prevents data load or dim build errors from being written to app log after the limit described by value of DATAERRORLIMIT is reached. NUMERICPRECISION - Sets number of percision digits use by Report Writer for numerical comparison. OUTLINECHANGELOG - Controls whether Essbase keeps a history of outline modifications. OUTLINECHANGELOGFILESIZE - Sets max file size of outline change log. QRYGOVEXECTIME - Sets max amount of time a query can use to retreive & deliver info before the query is terminated. REFERENCEC UBESIZELIMIT - In terms of cells, specifies max amount of memory to be set aside for all reference cubes loaded at the same time for a specific db. SECURITYFILECOMPACTIONPERCENT - Specifies percentage of obsolete space in the security file (essbase.cfg) that is a factor in triggering compaction of that file. SILENTOTLQUERY - Controls whether Essbase keeps a history of outline queries in the app log file. SQLFETCHERRORPOPUP - Controls whether Essbase error is generate when fetching data from a SQL db during data load or dim build. (Popup in AAS, & enables error handling using IFERROR in MaxL or ESSCMD). SSAUDIT - Enables spreadsheet update logging, appending to existing logs after archiving. SSAUDITR - Enables spreadsheet update logging, clearing the logs at the end of archiving process. SSLOGUNKNOWN - Controls whether Essbase logs error messages when it encounters an unknown member name during a spreadsheet operation. SSPROCROWLIMIT - Controls max number of spreadsheet rows Essbase processes on an Excel Add-in spreadsheet request. SUPNA - Controls whether the Supress #Missing Rows option is the spreadsheet interface suppresses the display of cells for which users has no access (along with suppressing #MISSING rows). TARGETTIMESERIESOPT- Globally sets query optimization across transparent partions for outlines that have a time dim w/ DTS members (requires identical source & target DTS). TIMINGMESSAGES - Controls whether Essbase logs the duration of each spreadsheet & report query in app log. TRIGMAXMEMSIZE - Specifies max amount of memory that Essbase can allocate to triggers feature. www.odtug.com

17

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

UNICODEAGENTLOG - Specifies whether the agen log is written in UTF-8 encoding or according to the locale of the system (set with ESSLANG variable).

BSO Settings AGGRESSIVEBLKOPTIMIZATION - Improves batch calc time for BSO cubes CALCCACHE - Specifies use of calc cache when calculating databases. CALCCACHEHIGH - Sets high value for calc script, use with SET CACHE HIGH command in calc script. Ensure that app data cache setting is set to effectively leverage setting. (max 200,000,000) CALCCACHEDEFAULT - Sets default value for calc script, use with SET CACHE DEFAULT command in calc script. Enusre proper setting of data cache to leverage. (default 200,000) CALCCACHELOW - Sets low value for calc script, use with SET CACHE LOW command in calc sript. Ensure proper setting for data cache to leverage setting. CALCLIMITFORMULARECURSION - Set to true to prevent the server from going beyond 31 formula execution levels. CALCLOCKBLOCKLOW - used with SET LOCKBLOCK LOW command in script. Sets number of blocks Essbase can fix (get addressability to) when calculating one block, use in context with data cache setting. CALCLOCKBLOCKHIGH - used with SET LOCKBLOCK HIGH command in script. Sets number of blocks Essbase can fix (get addressability to) when calculating one block, use in context with data cache setting. CALCLOCKBLOCKDEFAULT - used with SET LOCKBLOCK DEFAULT command in script. Sets number of blocks Essbase can fix (get addressability to) when calculating one block, use in context with data cache setting. CALCMODE - Enables global setting for formula execution mode, can be made app specific. CALCNOTICE - Sets completion notices about progress of calc (HIGH, DEFAULT, LOW) CALCOPTFRMLBOTTOMUP - Specifies whether Essbase optimizes calc of complex formulas on sparse dims in large db outlines. If enables, Essbase performs a bottom-up calc. CALCREUSEDYNCALCBLOCKS - Controls whether dynamically calced values are re -used during retrievals. CALCTASKDIMS - Specifies the number of sparse dims included in the identification of tasks for parallel calc, should be set by application along with calcparallel. If anchor has dymanic upper levels will be not leverage. CCTRACK - Controls whether exchange rates are tracked as Essbase calculates currency conversions. DATACACHESIZE - Defines initial value for data cache size for new dbs created after Essbase is restarted, should be set a db level. DATAFILECACHESIZE - Defines initial value for data file cache size for all new dbs created or migrated. DELAYEDRECOVERY - Determines whether Essbase delays free space recovery after an app crashes or terminates abnormally. DEXPSQLROWSIZE - Specifies number of rows inserted in calc export batch to a SQL files. Global, App, or DB level setting. DIRECTIO - Sets the file access mode to direct I/O instead of default buffered I/O. 6.2 or higher, should be used with caution! DISKVOLUMES

- Determines location of index & data files.

DYNCALCCACHEBLKRELEASE - Enables Essbase to create temp buffer for dynamic calcs in cases where the wait for space in dyn calc cache has exceeded the specified wait time. Set by app. DYNCALCCACHEBLKTIMEOUT - Specifies max time to wait for free space in dyn calc cache. Set by app (default is 10 seconds). DYNCALCCACHECOMPRBLKBUFSIZE - Specifies size of temp buffer for storing compressed blocks in order to make more space in dyn calc cache. Set by app (default is 1M). DYNCALCCACHEMAXSIZE - Specifies max amount of memory allocated for dyn calc cache. Set by app/db (default is 20M) DYNCALCCACHEONLY - Specifies whether dyn calcs can use memory outside the dyn cacl cache in case it is full. Set by app/db (defalut is false). www.odtug.com

18

ODTUG Kaleidoscope 2008

Tuning and Optimizing BSO Essbase

Sawa

DYNCALCCACHEWAITFORBLK - Specifies whether Essbase should wait for memory to be freed in dyn calc cache, or use outside memory. Set by app/db (default is false). EXCLUSIVECALC - Determines whether Essbase allows concurrent calcs (default is false, allows concurrent calcs). EXPORTTHREADS - Sets default number of threads that can be produced during a parallel data export. Can set by app/db (default is 1). IBHFIXTHRESHOLD - Controls how many IBH messages are returned to client or server log, relative to lev 0 blocks written to disk. Use carefully. IMPLIEDSHAREONMINUS - Deprecated. Specifies that single parent/child members with agg symbol MINUS are treated as implied share. Use only with App Manager. INCRESTRUC - Specifies whether incremental restructuring is enabled for a db. Can set by App/db. INDEXCACHESIZE - Defines initial value for index cache size for new dbs created after Essbase service restarted. LOCKTIMEOUT - Limits amount of time a spreadsheet add-in user can hold an exclusive lock. LROONSHAREDMBR - Specifies whether shared members have LRO's that are uniqued from those of their correspondign regular members. MULTIPLEBITMAPMEMCHECK- Enforces size limit for amount of memory that is used for calc cache when Essbase selects multiple bitmap cache option. PARCALCMULTIPLEBITMAPMEMOPT - Optimizes memory use when using multiple bitmap mode during parallel calc. QRYGOVEXECBLK- Sets max number of blocks that a query can access before the query is terminated. COPYMISSINGBLOCK - Sets whether the DATACOPY calc command creates #MISSING blocks during the coopy of data from a dense dim. (Default is on) UPDATECALC - COntrols whether intelligent cal is turned on or off by default. VLBREPORT - Enables Essbase to dyanmically determine the retrieval buffer size, between 100KB & 10MB, for retreivals from db w/o Dyanmic calc, attribute, or DTS members.

www.odtug.com

19

ODTUG Kaleidoscope 2008