The Internal Workings of Essbase: BSO and ASO Secrets Revealed
Edward Roske
[email protected] BLOG: LookSmarter.blogspot.com WEBSITE: www.interrel.com TWITTER: ERoske
Disclaimer These slides represent the work and opinions of the presenter and do not constitute official positions of Oracle or any other organization. This material has not been peer reviewed and is presented here with the permission of the presenter. This material should not be reproduced without the written permission of interRel Consulting.
We will send you a copy of the slides shortly after the presentation.
About interRel 2008 Oracle Titan Award winner - EPM Solution of the year 18 presentations at Collaborate 2009, 14 presentations at Kaleidoscope, 6 at OpenWorld 2008 2008 Oracle Excellence Award winner with Pearson Education One of the fastest growing companies in the world (Inc. Mag., ’08) We have two of the three Hyperion Oracle ACE Directors in the world Founding Hyperion Platinum Partner; now Oracle Certified Partner Focused exclusively on Oracle Hyperion EPM software Consulting Training Infrastructure and Installation Support Software sales 3
5 Hyperion Books Available:
Essbase (7): Complete Guide Essbase System 9: Complete Guide Essbase System 9: End User Guide Smart View 11: End User Guide Essbase 11: Admin Guide
eBooks available on Amazon Kindle
Coming Soon Hyperion Planning for End Users (September) Hyperion Planning for Admins
4
Copyright © 2007, Hyperion. All rights reserved.
To order, check out www.lulu.com
Who is this Edward Roske guy, anyway?
Began Essbase consulting in 1995 One of the first Essbase certified consultants in the world Founded interRel Consulting in 1997 to focus on Hyperion Speaker at annual Hyperion user conferences since 1996 President of ODTUG Hyperion SIG Essbase Domain Lead for OAUG Hyperion SIG Author of ―Look Smarter Than You Are with Hyperion Essbase‖ and most importantly…
Playwright and Lyricist of ―Eddie and the Consultants: A System 9 Musical‖ which performed to sold out audiences in Las Vegas and Walt Disney World in Orlando 5
Agenda
Introduction Internal Workings of Essbase: BSO Internal Workings of Essbase: ASO Question and Answer
Internal Workings of Essbase: Block Storage
7
Bob Earle Invented ―Sparse and Dense‖ How is data distributed?
SPARSE
DENSE
Products
Time Periods Accounts
Locations
X X
X X X
X X X X
X X X
X
X
X X X
X X X
X
X X
X X
8
Block Structure
Account (dense)
Var% Var Bud Act UnitsSold Headcnt Profit Expense OtherExp Marketing Salary Rev COGs Sales Jan
Feb
Mar
Q1
Q2
Q3
Yr
Period (dense) 9
Data Cells Within the Block
Actual -> Profit -> Jan
Cross-dimensional operator (means Actual BY Profit BY Jan)
Blocks An individual block is created for each combination of sparse stored members Cola->East
Cola->New York
Cola->Florida
Cola->Massachusetts
11
Where do we define database? The outline!
12
Storage and Compression
13
Storage PAG File
Contains the data (the blocks with a header for each) Contains up to 2 Gb of data in each PAG file (32-bit) Can be 1,024 different files Can be compressed and fragmented Can be stored on multiple drive locations
IND File Contains the list or pointers to the blocks (intersection of sparse dimensions) Contains up to 2 Gb of index in each IND file (32-bit) Can be 1,024 different files Can be fragmented Can be stored on multiple drive locations 14
Storage
15
Compression
16
Compression ―Simple‖ compression settings None zLib Index Value Pair Can’t assign directly Good for really large blocks with really sparse data
Following types use multiple compressions (one per block) Bitmap
Good for non-repeating data Will use Bitmap or IVP RLE = Run Length Encoding
Good for data with zeros and data that repeats (such as budgeting) Will use RLE, Bitmap, or IVP 17
Dimension Order & Compression
Dimension order affects compression First dense dimension determines your ―columns‖ in PAG file Compression is from left to right, top to bottom Move Time to first dense dimension and we get:
BUDGET
Jan
Feb
Mar
Apr
May
Jun
Sales
100
100
100
120
120
120
COGS
50
50
50
50
50
50
Margin
50
50
50
70
70
70
Exp.
30
30
30
30
30
30
Profit
20
20
20
40
40
40
Notice repeating values Time should be dense then Measures for RLE compression 18
Calculations
19
Calculation Process
Vermont -> Cola -> Actual Accounts Sales COGS Margin
Jan
Feb
Mar
Qtr1
124.71 119.43 161.93 406.07 42.37 82.34
38.77
47.28 128.42 80.66 114.65 277.65
Dense Calculation
After calc of Time dimension
Year Qtr1 Data load from table
XXXX XXXXXXX Jan ### XX Feb ### XX ### Mar XX
After calc of Accounts dimension Sales
COGS Margin Profit
Sparse Calculation Calculated blocks Upper-level blocks
East -> Cola
Input blocks Level zero blocks Vermont -> Cola
New York -> Cola
Calculation Order: All Dimensions
First, Accounts Second, Time Third, remaining dense dimensions Fourth, remaining sparse dimensions Two Pass Calculation
Calculations
Default Calc - simplest method Calc scripts Dynamic calculations Intelligent calculation
Commit Blocks Using Uncommitted Access When Commit Level is reached, blocks write to hard drive
Default is 3000 blocks Setting Commit Blocks to Zero Writes at completion of the entire transaction Will dramatically improve calculation time Will fragment your PAG file during a calculation
25
Retrievals
26
Index Cache
Index pages in index cache
New requests
Index pages on physical disk
Old requests
Memory
Disk
Data Cache New requests
Disk blocks in data cache
Disk blocks on physical disk
Old requests
Memory
Disk
Retrievals Sparse retrievals Sparse dimensions down the rows or across the columns Accesses multiple indexes and multiple blocks Slower retrieval times
Dense retrievals Dense dimensions down the rows and across the columns Accesses one index entry and one block Faster retrieval times
29
Internal Workings of Essbase: Aggregate Storage
30
BSO Limitations Financial applications are more densely populated so BSO works great in those instances BSO engine can handle sparse data but on a ―limited‖ scale Outline size limited Batch times required for loads and calcs
Aggregate Storage Option Remember all the concepts we just learned:
Dense / Sparse Index / page files Cache settings Block storage
X
32
Aggregate Storage Databases Similar to a BSO database – outline, dimensions, hierarchies… BUT Different method for storing/calculating databases New storage kernel built to handle ASO databases Calculates 10-100x faster Stores up to 252 dimension combinations
33
Aggregate Storage Databases ASO addresses the following types of databases: Read-only* Write back to level zero available in 9.3.1
―Rack and stack‖ Large dimensions
New Types of databases are possible Customer analysis—data is analyzed from any dimension and there are potentially millions of customers Procurement analysis—many products are tracked across many vendors Logistics analysis—near real-time updates of product shipments Market Basket analysis—products purchased along with other products
34
When to use ASO Database is sparse and has many dimensions, and/or the dimensions have many levels of members Database is used primarily for read-only purposes, with few or no data updates Calculation of the database is frequent, is based mainly on summarizing of the data, and does not rely on calculation scripts
Starting in Essbase 11x, ASO should be your default/starting idea
35
BSO vs. ASO BSO Outline
ASO Outline
36
End User Perspective End users won’t care whether their database is ASO or BSO The way users access ASO is the same as BSO Excel Add-in Financial Reporting, Web Analysis, etc. Smart View Add-in
Repeat… no differences (just more data/dimensions)
ASO Benefits from IT Perspective Faster load and calc times provide Lower hardware costs Lower maintenance costs Higher availability
Small disk footprints Efficient tuning for storage and query response
How does ASO work? Simple question with not so simple answer Asked greatest minds in the business how ASO works and got the same resounding answer: ―It’s a black box‖ or ―it's top secret and hard to understand‖ There must be a better answer!
ASO is Designed For… More dimensions and members Less time required for batches Fast aggregation of sparse data sets Faster loads Incremental loads
Reduction in database footprint
Key Concepts
Storage Sparse data Indexing Aggregation Nodes
ASO Storage (ROLAP in disguise) ASO can also be said to be ―ROLAP with a super fancy index scheme that rules.‖ Big difference between ASO and BSO and ROLAP is the ASO storage mechanism ROLAP stores data in a table and indexes a combination key between the rows Essbase stores the concept of a cube of data in multiple dimensions or rather multiple keys
ASO Storage (ROLAP in disguise) It’s a multidimensional index ASO takes it a step further and indexes the indexes in a way for more rapid aggregation of data Storage is no longer in "blocks" but in highly optimized aggregation nodes Visualize it as an asymmetric fractal Christmas tree flattened out and then indexed again
How does ASO work
Load Data only at level 0
Create aggregate views Algorithm selects and stores ―most taxing‖ queries Dynamic queries at runtime and increased speed by leveraging nearest stored view
ASO Concepts Concept of stored and dynamic hierarchies Stored hierarchies can only aggregate Dynamic hierarchies can utilize formulas and advanced unary operators
Formulas on dimension members use MDX syntax Pre-aggregated views can be defined to help query performance Aggregated design wizard helps you create aggregation scripts Outline paging helps you page portions of the outline in and out of memory to assist in performance You can convert BSO outlines to ASO outlines using a wizard
Storage and Compression
46
Directory Structure Directory structure differs in both content and purpose from BSO Tablespaces are utilized to store data and metadata Default – stores numeric data (.dat file) Log – records database activity Metadata – stores metadata information about the objects in the database Temp – temporary working space for the Essbase kernel
Tablespace Overview Defines the database storage in the form of file locations Each ASO application has 4 tablespaces
Default – database values Temp – temporary work space Log – transaction log files Metadata – database data structure
File location specifies a physical disk space for storing database files Each tablespace may contain one or more file locations Can span multiple physical drives and/or logical volumes 48
Manage Tablespaces
Sizing Tablespaces During the data load and aggregation process, data is stored in both the Temp and Default directories ASO will always build the full .dat file in the temp tablespace while the default tablespace still has the production .dat Hence, for your maximum drive size you have to plan on AT LEAST 3x your max bloated .dat size if you want to be "safe" (buffer to temp to default)
50
Compression Dimension ASO In old releases, the Accounts dimension enabled database compression Beginning in 9.3, the Accounts dimension is the default compression dimension BUT you can choose a different compression dimension Essbase helps you choose a compression dimension by estimating what the database size would be depending on which dimension is tagged as compression Time is an excellent candidate for compression dimension, especially if you have fiscal year as a separate dimension
Calculations
52
Aggregating ASO Data For ASO databases, after data values are loaded into the level 0 cells of an outline, the database requires no separate calculation step From any point in the database, users can retrieve and view values that are aggregated for only the current retrieval ASO databases are smaller than block storage databases, enabling quick retrieval of data values For even faster retrieval, you can precalculate data values and store the precalculated results in aggregations
Aggregations – The Down Side Lengthy process Requires extra disk space Sometimes yes, but it is rare that default aggs are more than 40 percent the size of the input data.
You want to balance query time and storage space
Intelligent Aggregations for ASO You can define hard restrictions for a dimension Default (no restriction for primary hierarchy, no aggregation for alternate hierarchies) Consider all levels Do not aggregate Consider top level only (you only query top level) Never aggregate to intermediate levels (you only query level zero or top dimension)
Level based weighting – provide levels to consider Process ASO considers hints when creating aggregations Attempts to create the most useful aggregations based on hints
55
Query Hints You can define ―soft restrictions‖ as a query hint Just select a representative member (any member) Essbase will take this into consideration when creating aggregation views
56
Query Hints
Design Considerations BSO Complex calculations and allocations Write back at upper levels from end users are required
ASO Large analysis applications with many dimensions and members Rolling up and analyzing large volumes of data
Both ASO and BSO Take advantage of the strengths of both database types
Consider Using Both ASO and BSO
Budget
Partition
BSO
Product SKU Analysis
ASO
Consider Using Both ASO and BSO – version 11
Partition
Product SKU Analysis
BSO
Budget
ASO
ASO vs. BSO Recap What are the similarities between ASO and BSO? Building dimensions Loading data (level zero only for ASO) Retrieving data
What are the differences?
Many dimensions, many members No calc scripts Use MDX member formulas Aggregations for improved query performance
Lots of improvements in System 9 and version 11 Understand the limitations in early versions of Essbase ASO Don’t miss the new features in 9.3.1 and 11x
61
Thank You!! Edward Roske
[email protected] BLOG: LookSmarter.blogspot.com WEBSITE: www.interrel.com TWITTER: ERoske
