Powering I.T I.T.. Empowering Business.
HFM Consolidation Demystified Jonathan Berry President & CEO
[email protected] 203.331.2267 Copyright © 2014, Accelatis. All rights reserved.
Agenda • Consolidation from a Business perspective • Consolidation from a Technical perspective • CPU Utilization • Intercompany • Database Utilization • Subcubes • Rules • Performance Tips
Who are are We? W e?
Jonathan Berry Founder / CEO
Rob Poirier Director Client Services
Tony Mitri Architect
Development and Support Team
Hyperion Solutions: HFM Development Team – Team – 1998-2008 1998-2008 Accelatis: Founded in 2008
22
Years of Hyperion / EPM Experience
20
Years of Hyperion / EPM Experience
80
Years of Combined Hyperion Experience
Our Focused Mission
Provide you the needed insight & control into Oracle EPM / Hyperion Hyperion to efficiently & consistently manage your system to deliver maximum user benefit….
Our Solution Manage All Moving Moving Parts
Monitor
Support
Across All Environments
Streamline Process
2. Bro road ad App ppro roac ach h to to APM
Enterprise Systems Resolve
Optimize
More Quickly
Eliminate Bottlenecks
Simulate User experience
1. Empo powe werr Bus Busin ine ess Users & IT
3. Deep Look into Underlying Systems
Consolidation Process 1. The rule is run for the the interse intersection ction of the the child child entity and the Value member member that stores the child entity’s default currency (Entity Currency). 2. If the child entity’s AllowAdjs attribute is set to Y, the rule is run for the intersection of the child entity and the Value Val ue member member that stores adjustments in the child entity’s default currency (Entity Curr Adjs). 3. If the child’s currency differs from the parent’s currency, the rule is run for the intersection of the child entity and the Value member member that stores amounts translated to the parent’s home currency (Parent Currency).
Consolidation Process (cont) 4. If a child’s currency differs from the parent’s currency and the rules file contains a Translation rule, Financial F inancial Management executes the Translation rule before step 3. parent’ s 5. If the child’s currency differs from the parent’s currency, and the child entity’s AllowAdjs attribute is set to Y, the rule is run for the intersection of the child entity and the Value member that stores adjustments translated to the parent’s home currency (Parent Curr Adjs). 6. If the parent entity’s AllowAdjFromChildren attribute is set to Y, the Logic rule is run for the intersection of the child chil d entity and the Parent Adjs value
Consolidation Process (cont) 7. The The rule rule is run run for the the inters intersec ectio tion n of the child entity and the Proportion value. 8. The The rule rule is run run for the the inters intersec ectio tion n of the child entity and the Elimination Elimination value.
9. If the parent entity’s AllowAdjFromChildren AllowAdjFromChildren attribute is set to Y, the rule is run for the intersection of the child entity and the Contribution Adjs Adjs value. 10. For each additional child child entity that contains previously unconsolidated unconsoli dated data, repeat steps 1 through 9. 11. The rule is run for the intersection of the parent entity and the Value member member that stores the parent entity’s default currency.
Consolidation vs Aggregation • Aggregation Aggregation is the process process of calculating calculating paren parentt total in wi thout any the HFM application server on the fly without action cti on by the the us er . Aggregation is performed within the HFM subcube. • Consolidation is the process moving data across the Entity and value dimensions by physically persisting the calculated data to the database, i t is a us er initia i nitiatted
action.
What Happens Inside HFM? 1. A consol consolida idatio tion n is ini initia tiated ted by by the user user 2. HFM HFM will will place place a syste system m lock lock to pre preven ventt metadata, rules, security, member lists from being loaded. 3. Syste System m preve prevents nts starva starvatio tion n 4. Chart Chart logic logic is perfo performed rmed on the the NONE NONE member member of the entity dimension. 5. Retrieve Retrieve all the base level level entitie entities s for for the the specified hierarchy and execute chart logic including prior periods with dirty calc status.
What Happens Inside HFM?(cont) HFM?(cont) 5. Start Start the conso consolid lidati ation on proc process ess using using a botto bottom m up approach. 1.
This This entity entity hierarch hierarchy y is locked locked to prevent prevent others others con consolid solidatio ations ns from writing reporting transactions to the DB
2.
Enumerat Enumerate e all the parents parents und under er the the speci specified fied hierarch hierarchy y starting starting with the bottom bottom most one.
3.
For every parent, parent, enumera enumerate te all the children children and and consolid consolidate ate them them into into the parent using the HFM consolidation process described in the beginning. 1.
Each Each child child is loc locke ked d by HFM HFM as as calcu calculat latio ion n are perfo performed rmed and and unloc unlocke ked d when done.
2.
Chart Chart logic logic and and trans transla latio tion n are exe execu cute ted d on all all the the chil childre dren n
3.
The The contri contributi bution on of all the children children is moved into the entity entity curren currency cy of the parent.
4.
Save Save the data data to to the databa database se in a transa transactio ctional nal way. way. Each Each subcub subcube e in a transaction.
What are my CPUs doing? Look familiar? f amiliar?
Simple Scenario o
o
o
o
o o
Consolidate hierarchy with 4 base entities, entiti es, each with with 100K records r ecords on a machine machine with 4 Cores Depending on the number number of threads t hreads that are defined for consol, consol , for this example let say that value is 4.
HFM will will load the 4 base entities into memory memory and will start running “Calculate” on them. They will all finish in roughly roughl y the same same time ti me due to the same number of records.
All 4 CPUs will have high utilization and then all drop HFM can now move up the hierarchy.
Less Simple Scenario o
o
o
o
o
o
o
HFM will enumerate those ID for those subcubes, they are not loaded into memory yet. Depending on the number number of threads t hreads that are defined for consol, consol , for this example let say that value is 4.
HFM will will load the first 4 subcube into memory from that list into memory memory and will start running “Calculate” “Calcul ate” on them. them. As soon as one finishes another will will be run from that list, li st, till that list is exhausted.
At this this stage all 4 CPUs should be fully utilized, but might might drop when a subcube subcube is loaded for calculation. Once the number number of subcube is below the number of threads then are assigned for for consol, then we we will start to see a drop in CPU utilization. HFM can’t move up the hierarchy till all the base enti ties are done.
What are my CPUs doing? (cont)
•
The HFM Application may have more impact on the performance profile than infrastructure.
• All children children in a hierarch hierarchy y need to be consolidated consolidated before consolidation moves up the hierarchy.
What are my CPUs doing? (cont) •
In a multi-CPU machine, children of a hierarchy are consolidated on separate threads and parent is processed when all children threads are complete.
•
Entities generally generally do not have equal amounts of data so some will be completed ahead of others. Most of the CPUs will will be utilized for a brief amount of time while while waiting for f or the entity with the most records to finish.
Intercompany o
Intercompany - Use rules or native nati ve
o
Open Data unit - efficiencies effici encies gained
o
Organizing rules rules to run IC elim by partner, not by account
Database Usage HF M is an a ana nallytica ytic al eng eng i ne, not a tr ans ac ti ona onall eng i ne o
This causes a lot of grief grie f for DBAs and Analysts who treat t reat it and try to analyz analyze e it like something that it is not. Ie: analyzing analyzing SQL statements.
A ll no non-ag n-ag g r eg ated ted data data points poi nts ar e r ead ead from fr om and wri wr i tten tten to the data databas bas e o o
o
A high-performance database is a critical. Verify that your database has the properly allocated allocated amount of memory and that your disk performance is optimal. Periodically run Oracle AWT or MSSQL SQL Profiler to help monitor system system performance.
Database Usage (cont) L arg e numbe numbers rs of records r ecords are freq fr equent uently ly adde added/ d/de delleted eted dur durii ng normal normal usa us ag e o
o
This effects indexes and thus query performance. Verify that your DBMS performs periodic index statistics updates and/or index rebuilding after high data load cycles to maintain mai ntain optimal performance.
If your database is stored on a NAS or other network storage device, the performance of the network will will directly affect your database performance. Periodically validate that your network network is not becoming overloaded or adversely affected by other issues (resets, re-transmits, etc).
Subcubes •
Subcube are units of memory inside HFM
• All data for a given Scenario/Year/Entity/Value Scenario/Year/Entity/Value are in a single subcube •
Get an understanding of the size of your subcubes
•
Understand the number of subcubes required r equired for your largest branch
•
Understand the intercompany process
•
If you can get all the subcubes in RAM to allow intercompany inter company to perform quickly quickly there th ere are performance benefits
•
Understand that 2 similar looking grids can perform very differently different ly
Subcubes Subcube Population If #records > 100,000 consider changing org structure SQL Example: SELECT lEntity, lValue, count(*) as numRecs FROM COMMA_DCE_1_201 COMMA_DCE_1_2011 1 GROUP BY lEntity, lValue
21
Capturing Subcube Populations select lEntity, lValue, COUNT (*) FROM group by lEntity l Entity,, lValue lValue Entity 5 5
Value 10 11
NumRecords 72 34
select lEntity, lParent, lValue, COUNT (*) FROM group by lEntity, lParent, lValue Entity 5 5
Parent 8 8
Value 10 11
NumRecords 72 34
•Need to convert IDs to Metadata Labels •Need to compare Before and After Consolidation
Rules Gotchas 1. Mul Multipl tiple e VBS VBScr crip iptt en engine gines s • Do not use global variables as counters • Total time calculated calculated may be more than actual elapsed time t ime due to multiple multiple threads 2. Perf Perfor orma manc nce e Imp Impac actt of Instr Instrum umen enta tati tio on • If tracking function time, write write data outside of timings timi ngs • If using Objects, make calls calls asynchronous so writing of data happens in queue on background thread. Do not create thread t hread for every call! 3. Elimina Eliminate te unnece unnecessa ssary ry calls calls to routine routines s from globa globall scope scope.. This can greatly increase load load time t ime and application startup time
Rules Gotchas (cont) 4. Writing Writing to file file from from mul multipl tiple e VBScrip VBScriptt engine engines s is danger dangerous ous.. • Each instance may block access to the file f ile and cause deadlocks as well well as major performance issues. Doing so effectively effectiv ely forces HFM HFM Consolidation into single si ngle • threaded operation. 5. Resol Resolutio ution n of timing timing using using the the Timer Timer function function in VBSc VBScript ript is >= 15ms. Using this built-in timing function can amplify amplify timing calculation calculation errors over many iterations.
Performance Performance Tips Tips
Profiling Rules The HFM Rules Profiler is an automated autom ated tool for instrumenting, inst rumenting, profiling and analyzing analyzing HFM Consolidation performance. perfo rmance. •
Overall Inclusive Function Time
•
Most Time Consuming Inclusive Inclus ive Functions
•
Overall Exclusive Function Time
•
Most Time Consuming Exclusive Functions
•
Function Call Frequency
•
Most Frequently Called Functions
Tip 1: OpenDataUnit
OpenDataUnit is powerful and efficient mechanism for retrieving retrieving data.
Tip 1: OpenDataUnit (cont) Resist using ‘for -next’ loops loops instead of OpenDataUnit to do calculation calculation on the account dimension. AccountList = HS.Account.List HS.Account.List ("", "") for each Account in AccountList ‘do work next
Tip 1: OpenDataUnit (cont) Use OpenDataUnit instead for the same task Set MyDataUnit = HS.OpenDataUnit("") lNumItems = MyDataUnit.GetNumItem MyDataUnit.GetNu mItems s for i = 0 to lNumItems-1 ‘do work next
Tip 2: Sub Calculate •
Sub Calculate is executed about 8 times for each entity and currency
•
Limit the rules execution on that function
•
Some data manipulation needs to occur only on the default currency, so adding that check in your if statement is a key to improving performance
Tip 2: Sub Calculate (cont) Resist logic against the whole whole Value dimension dimensi on If HS.Entity HS.Enti ty.IsBase .IsBase(“”,””) (“”,””) = True ‘do work End if
Tip 2: Sub Calculate (cont) Specify Entity and Value dimensions dimensi ons when possible If HS.Entity HS.Enti ty.IsBase .IsBase(“”,””) HS.Value.Member = (“”,””) = True And HS.Value.Member ‘do work End if
Tip 3: HS.EXP Resolution Resolution
HS.EXP is mechanism mechanism for copying blocks of data within a Subcube in an application.
Tip 3: HS.EXP Resolution (cont) Resist implicitly copying all 4 custom custo m data values from one account to the other. HS.Clear “ A#DestAccount” A#DestAccount” HS.Exp “ A#DestAccount A#DestAccount = A#SourceAcc A#SourceAccount ount”
Tip 3: HS.EXP Resolution (cont) Specify N custom data values between Accounts HS.Clear “A#DestAccount.C1#DestC1” HS.Exp “A#DestAccount.C1#DestC1 = A#SourceAccount.C1#SourceC1”
Tip 3: HS.EXP Data Explosion Explosion The following following causes data explosion by specifying specif ying Account on right side and not the left side. HS.Clear "A#Sales.C4#Movement" HS.Exp "C4#Movement = A#Sales.C4#Movement + A#Sales.C4#Increases A#Sales.C4#Decreases"
Tip 5: Global Logic •
Code at global scope is executed when applications load and when rules are loaded.
•
Errors in code at global scope will not be reported at load time and may go unnoticed until consolidation failures result Example (code in RED RED is is at global scope) ‘Create global arrays Global_Acclist_Statistical Global_ Acclist_Statistical Global_Acclist_Validations Global_ Acclist_Validations Global_Acclist_CAccounts Global_ Acclist_CAccounts
= HS.Account.L HS.Account.List("","Statistical") ist("","Statistical") = HS.Account. HS.Account.List("Validations","[Base]") List("Validations","[Base]") = HS.Account.L HS.Account.List("C_Validations","[Base]") ist("C_Validations","[Base]")
Sub NoInput() ' Make all input cells for the Budget Scenario, Year 2004, Sales account calculated (prohibit input) HS.NoInput HS.NoInput "S#Budget.Y#2004.A#Sa "S#Budget.Y#2004.A#Sa les" HS.NoInput HS.NoInput "S#Budget.Y#2004.A#Adminexpenses" "S#Budget.Y#2004.A#Adminexpenses" HS.NoInput HS.NoInput "S#Budget.Y#2003.A#Adminexpenses" "S#Budget.Y#2003.A#Adminexpenses" End Sub '*** c1 list fo r security security class C1_CORP Global_C1_List Global_ C1_List = HS.Custom1.List("TotC1","[Base]") For each StrMember in Global_C1_List If UCASE (HS.Custom1 (HS.Custom1.SecurityClass(StrMember)) .SecurityClass(StrMember)) "C1_CORP" then aC1_CORP = ExactFil ExactFilter(aC1_CORP,StrMember,False) ter(aC1_CORP,StrMember,False) End If Next Sub Consolidate() Set MyDataUnit = HS.OpenDataUnit("") HS.OpenDataUnit("") dPCon = HS.No de.PCon("") lNumItems = MyDataUnit.GetNumItems End sub
Tip 5: Global Logic (cont) Performance times to open an application and load rules on sample application affected greatly greatly by global logic to t o build member lists. 300 250
270
240
200 With
150
Without 100
50
30 10
0
App Load Time Time
Rules Load Time Time
Questions?
Jonathan Berry President & CEO
[email protected]
Rob Poirier Director of Client Services
[email protected]
