A New Twist to HFM Automation Usingto HFM Process Management Drive HFM Extracts Alice Lawrence – Walmart Samrat Chakraborty – Deloitte
HFM Implementation is complex
27 Countries
Multiple currencies
Sub Consolidations within some countries
HFM only used for Actuals
Essbase used for Management Reporting
GL
APPLICATIONS
REPORTS
Data Forms & Journals
SAP (Corporate Chart of Accounts
F D M
ALL BI FEEDS
H F M
1 2
(COA)) 3
SAP (Non Corporate COA)
FDM stands for Financial Data (Quality) Management
iSpark (INTL)
HOST/GL Extracts
A d j u s t m e n t s
O v e r r i d e s
External IFRS/USGAAP/ Local/FinPack Reporting
Management Reporting (INTL) Local Reporting
Country Cubes
KEY HFM (Summary Level) Interfaces
GL
ESSBASE (Detailed level)
Other Applications
1 2
Adjustments Extract HFM to Country Cubes Overrides Extract HFM to Country Cubes
Full Extract HFM to
3
ISpark
Need to Extract HFM Data on Process Status
Actuals and Plan needed for Mgmt Reporting
Business only wants to see data that is “Ready”
Don’t want to see data that with Journal Entries in progress
Year/Period Needs to Change each Month
4 Extracts/19 Country Rollups
Data will be extracted when Process Level in HFM is “Level 1”, “Level 2”, or “Promoted” Countries are “Promoted” a month later
We Reviewed Several Different Extract options Extended Analytics Templates
Essbase Analytics Link
EPM Maestro
SQL Table Extract
HFM API (Application Programming Interface ) for EA
Essbase Analytics Link
Designed to be more of a complete copy of what is in the HFM application Can be automated to be a “mirror” copy of what is in HFM Can create additional cubes that are “filtered” views of the data Can not create extracts based on process status
Extended Analytics most viable
Year and Period are variables so the same template can be used each month
Other Dims are hardcoded if they do not change.
EA Entity is Unique per Extract
EA had many challenges
19 Countries * 4 extracts each = 76 templates
Could not automate based on Process Status
Countries could run extracts ● Extra manual steps for countries ● Introduced risk and opportunity for failure ● Corporate would also need to run when they “approve”
Centralized extract at Corporate ● Additional coordination with countries ● Additional work/risk/opportunity for failure
HFM API
Advantages
Disadvantages
●
Extract on Process Level
●
Requires programming
●
Can Automate
●
API Knowledge necessary
●
Reduced Risk of Errors
●
Oracle Unix Platform rewrite could effect API and may cause rewrite of code
HFM API and Smart View VBA Selected
Solution was built within the Excel framework Smart View HFM VBA(Visual Basic Application) functions are used to refresh the Process Status HFM VBA API functions are used to extract the data Batch scripts and the windows scheduler are used to automate the job and perform other file tasks
DETAILED SOLUTION REVIEW
Extended Analytics Automation tool requirements • Primary Requirements:
• Automate extract generation based on process management
status • Generate 4 different type of extracts (diff set of entities, accounts, scenarios and C1 members) •The solution should rename the extract file as per specification and unzip it in a pre specified folder • Easy maintenance
Challenges • Comparing current to prior process management status
• 4 different extract types capturing different level of details • Compare process management status for multiple periods • Server cluster in PROD – need to know which server the extract gets generated
Extended Analytics Automation tool solution overview • Step 1: Obtain current process management status for the listed
entities • Step 2: Compare current process management status against prior • Step 3: List out the entities for which extracts needs to be generated • Step 4: Define extract POV to generate extracts • Step 5: Rename and move the extract file from the HFM server to the destination location • Step 6: Unzip the extract file and convert to ASCII • Step 7: Use windows scheduler to run the tool at regular intervals
Prerequisites • Excel is installed on the computer where this automation tool is run
• Smart View is installed • HFM Client is installed as it will use the relevant files required for this automation •The Financial Management application server files should be installed on the same computer or we must have read access to a file system on which the application server files are installed. • Access to the application server files is required so that the automation tool can refer the object model’s type libraries.
STEP 1: PROCESS OBTAIN CURRENT MANAGEMENT STATUS FOR THE LISTED ENTITIES
Master Control file contains Master data • A master control file is created as the only interface that needs to be maintained by admin (apart from code – as necessary) • Master control file contains the list of all entities for which extracts could be generated • Parent level entities – for process management • Base level entities – for extract generation • It also includes the environment information • Application name • DSN • HFM Server cluster name • Smart View connection name
EAMasterControl.xlsm stores master data HFM Environemt Info
HFM Entities that we are retrieving the Process Status of
EAMasterControl.xlsm entity Lists
HFM Entities that we are extracting; each column is a different country
Connect.XLSM Refreshes the Process Status
Separated to protect the HFM Admin password
Two main parts ● Excel tab ● VBA Code
Year/Period dynamically populated via VBA Smart View VBA used to Connect to HFM and Refresh the HFM Process Status
Define Workbooks and Refresh Entity List • Set the MasterApp and MasterBook variables Set MasterApp = New Excel.Application MasterApp.Visible = False Set MasterBook = MasterApp.Workbooks.Add(strFilePath & "\EAMasterControlFile.xlsm") MasterBook.Application.DisplayAlerts = False
• Copy over the Master Entities from MasterControlFile and update the Refresh worksheet (to refresh) Workbooks(RefreshBook).Worksheets("Refresh").Range(“A1:A100") = "" Workbooks(RefreshBook).Worksheets("Refresh").Range("A1:A100””).Value = MasterBook.Worksheets("MasterEntities").Range("A1:A100").Value
Check for HFM Connection • Obtain the period information for which the process management status is to be verified – based on current system date 'Store current period in tPer tPer =the Month(Date) 'Store the current Year in cYear cYear = Year(Date)
• Check if connection exists Syntax: Private Declare Function HypConnectionExists Lib "HsAddin.dll" (ByVal vtConnectionName As Variant) As Variant Example: isConnectionPresent = HypConnectionExists(FriendlyName)
Create Connection if it doesn’t Exist • If connection doesn’t exist – create it If isConnectionPresent = False Then Workbooks(RefreshBook).Worksheets("Refresh").Activate Syntax: Private Declare Function HypCreateConnection Lib "HsAddin" (ByVal vtSheetName As Variant, ByVal vtUserName As Variant, ByVal vtPassword As Variant, ByVal vtProvider As Variant, ByVal vtProviderURL As Variant, ByVal vtServerName As Variant, ByVal vtApplicationName As Variant, ByVal vtDatabase As Variant, ByVal vtFriendlyName As Variant, ByVal vtDescription As Variant) As Long Example: lRet = HypCreateConnection("Refresh", “", “", "Hyperion Financial Management", “", strHFMCluster, txtConnectionName, txtConnectionName, FriendlyName, "Hyperion Financial Management")
Establish the connection to HFM
Associate the worksheet with the HFM connection established Use HypConnect Workbooks(RefreshBook).Worksheets("Refresh").Activate ConnSheet = HypConnect("Refresh", “", “", FriendlyName)
Set Sheet Options to Process Status • Once connection is created refresh Smart View to obtain process management status • Set the Display to Process Management and not DATA / CALCULATION STATUS Syntax: Private Declare Function HypSetOption Lib "HsAddin" (ByVal vtItem As Variant, ByVal vtOption As Variant, ByVal vtSheetName As Variant) As Long Example: lSetOption = HypSetOption(15, 2, "Refresh")
Refresh Smart View and Disconnect • Refresh Smart View Syntax: Private Declare Function HypExecuteMenu Lib "HsAddin.dll" (ByVal vtSheetName As Variant, ByVal vtMenuName As Variant) As Long Example: RLGridRefresh = HypExecuteMenu("Refresh", "Smart View->Refresh")
• Disconnect the connection Syntax: Private Declare Function HypDisconnect Lib "HsAddin" (ByVal vSheetName As Variant, ByVal vLogoutUser As Boolean) As Long Example: ConnSheet = HypDisconnect("Refresh", True)
EAMasterControl.xlsm stores Process Status
HFM Entities from the 1st tab.
STEP 2: COMPARE CURRENT STATUS AGAINST PRIOR STATUS
Determining what to extract •
Challenge was to determine if the process status was one we wanted to extract
•
Ensure that we had not already extracted data based on that process status
•
Use excel to store the last process status retrieved from HFM
•
Use excel formulas to compare old vs new
Excel Formula handles comparison Formula in column F compares New Status vs Old Status and checks if the new Status is one we want to extract
STEP LIST OUT THE3:ENTITIES FOR WHICH EXTRACTS NEEDS TO BE GENERATED
Create Parent/Child lists for Entities Use the Entity Lists from the EAMasterControl.xlsm file to create Parent/Child Entity Lists
STEP 4: DEFINE EXTRACT POV TO GENERATE EXTRACT
Subroutines were Created for each Extract •
Four different type of extracts - different account and entity details - different C1 details
- different scenarios Example: Private Sub AllExtract(MasterEntity As String, ParentEntity() As String, ChildEntity() As String, tYear As String, tPeriod As String, nElements As Integer)
Create API connection Function ConnectHFM(txtConnectionName As String, strHFMCluster As String) As HsvSessionlib.HsvSession Dim oServer As HsxServer Dim oClient As HsxClient Dim oSession As HsvSession Set oServer = Nothing Set oSession = Nothing Set oClient = New HSXCLIENTLib.HsxClient oClient.SetLogonInfoSSO "", “", "", “" oClient.OpenApplication strHFMCluster, "Financial Management", txtConnectionName, oServer, oSession Set ConnectHFM = oSession End Function
Create HsvStarSchemaACM object Set oSession = Application.Run("'" & ThisWorkbook.Path & "\ConnectClient.xlsm'!Connect.ConnectHFM", txtConnectionName, strCluster) Now create schema object: Set cHsvSystemInfo = oSession.SystemInfo Set oStarSchema = oSession.CreateObject("Hyperion.HsvStarSchemaACM") oStarSchema.SetSession oSession
Create Function to retrieve Member Id Instantiate Metadata Object : Set oMetadata = oSession.Metadata Set the various dimension references Function GetMemberID(ByVal lDimID As Long, ByVal sMemLabel As String) As Long Dim cTreeInfo As HSVMETADATALib.IHsvTreeInfo Set cTreeInfo = oMetadata.Dimension(lDimID) GetMemberID = cTreeInfo.GetItemID(sMemLabel) End Function Used to return MemberId based on a member label
Some Dimensions are define as Long Year, Scenario, View – as Long: Dim lTotC1, lTotC2, lTotC3, lTotC4, lYearID, lPeriodID, lViewID, lValueID As Long lScenarioID = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONSC ENARIO, "Actual") lYearID = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONYE AR, tYear) lViewID = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONVIE W, "YTD")
Dimensions may require an Array Period ReDim lPeriodID(0) lPeriodID(0) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONPE RIOD, tPeriod) ICP ReDim lICPTop(0) lICPTop(0) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONICP, "[ICP Top]")
More array examples Custom2, Custom3, Custom4 ReDim lTotC2(0) lTotC2(0) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONCU STOM2, "TotalC2") ReDim lTotC3(0) lTotC3(0) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONCU STOM3, "TotalC3") ReDim lTotC4(0) lTotC4(0) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONCU STOM4, "TotalC4“)
Arrays with Multiple Values Select two currencies by defining the array to have two members, and getting MemberId for both
ReDim lValueIDs(1) lValueIDs(0) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSION VALUE, "") lValueIDs(1) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSION VALUE, "USD Total")
Selecting Base members of a Dimension
Set tempTreeInfo = oMetadata.Dimension(hfmconstantslib.tagHFMDIMENSIONS.DIM ENSIONCUSTOM1) tempTreeInfo.EnumBaseMemberIDs GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSION CUSTOM1, "{[Base]}"), True, lC1BaseIDs
Selecting Base Members of a specific member Set tempTreeInfo = oMetadata.Dimension(hfmconstantslib.tagHFMDIMENSIONS.DIM ENSIONACCOUNT) tempTreeInfo.EnumBaseMemberIDs GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSION ACCOUNT, "Financial Statement"), True, lAccBaseIDs
Obtain parent and child entity arrays ReDim lParent(nElements) i=0 For i = 0 To nElements lParent(i) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONENTITY, ParentEntity(i)) Next ReDim lEntity(nElements) i=0 For i = 0 To nElements lEntity(i) = GetMemberID(hfmconstantslib.tagHFMDIMENSIONS.DIMENSIONENTITY, ChildEntity(i)) Next
Extract data with CreateStarSchema Syntax: HsvStarSchemaACM>.CreateStarSchema bstrDSN, bstrTablePrefix, ssPushType, eaExtractType, vbExcludeDynamicAccts, varaScenarios, varaYears, varaPeriods, varaViews, varaEntities,varaParents, varaValues, varaAccounts, varaICPs, varaCustom1, varaCustom2, varaCustom3,varaCustom4 Example: oStarSchema.CreateStarSchema strDSN, txtConnectionName, ssCREATE, EA_EXTRACT_TYPE_FLATFILE_NOHEADER, True, lScenarioID, lYearID, lPeriodID, lViewID, lEntity, lParent, lValueIDs, lAccBaseIDs, lICPTop, lC1BaseIDs, lTotC2, lTotC3, lTotC4
STEP 5:
RENAME AND THE EXTRACT FILE MOVE FROM THE HFM SERVER TO THE DESTINATION LOCATION
Identify the HFM Server to copy extract from
Set oSession = Application.Run("'" & ThisWorkbook.Path & "\ConnectClient.xlsm'!Connect.ConnectHFM", txtConnectionName, strCluster) Set cHsvSystemInfo = oSession.SystemInfo tServerName = cHsvSystemInfo.GetServerName
Move only the latest files •
Obtain system time prior to extract generation varGetDate = Now() • Compare file timestamp against pre extraction system time EndFileName = MasterEntity & “" & "_" & tPeriod & "_" & tYear & ".dat.gz" Set objFSO = CreateObject("Scripting.FileSystemObject") Set objFolder = objFSO.GetFolder(strSource) For Each objFile In objFolder.Files If ob jFile .Date LastMo dified > v arGetDa te Then strFileName = objFile.Name objFSO.MoveFile strSource & "\" & strFileName, strDest & "\" & EndFileName End If Next
STEP 6: UNZIP THE EXTRACT FILE AND CONVERT TO ASCII
Unzip extract file and convert to ASCII using batch Unzip file using 7zip or another utility :DoExtract PATH=%PATH%;"C:\Program Files\7-Zip" SET DATAPATH= for /f %%D in ('dir /b %DATAPATH%\*.dat.gz') do 7z.exe e -y %datapath%\%%~nD.gz -o%DATAPATH% …and convert from Unicode to ASCII for /f %%D in ('dir /b %DATAPATH%\*.dat') do Type %datapath%\%%~nD.dat > %datapath%\%%~nD.txt
STEP 7: USE WINDOWS SCHEDULER TO RUN THE TOOL AT REGULAR INTERVALS
Windows Scheduler automation
Used Windows Scheduler and Batch file to execute file at specified intervals Call %~dp0\"WMTEAEXTRACT.xlsm“
Use Auto_Open within Excel file to automatically execute code
Sub Auto_Open() strFilePath = Workbooks("WMTEAEXTRACT.xlsm").Path Application.Run "'WMTEAEXTRACT.xlsm'!GenerateExtract.GenerateEAExtract"
Debugging and Maintenance
Created a log file to capture results of specific events
Set fObj = CreateObject("Scripting.FileSystemObject") Set fName = fObj.OpenTextFile(strPath & "\ExtractLog.log", 8, False) fName.WriteLine ("Extracting Data for Entity: " & MasterEntity & ", for Period: " & tPeriod & ", for Year: " & tYear & ", at " & Date & ", " & Time & " for AllData") fName.Close
Admin would have to maintain
The process mgmt. entity list
The extract entity list
Join the EPM Community at tonight’s
The EPM Carnival! 8 - 10 PM – Room 603 sponsored by:
Featuring: The Quest – Are you up to the challenge? Balloon Dart EPM Trivia Ace Director Bean Bag Toss Great Food and Flashy Prizes
QUESTIONS??
[email protected] [email protected]
APPENDIX
Functions to declare Private Declare Function HypConnect Lib "HsAddin" (ByVal vtSheetName As Variant, ByVal vtUserName As Variant, ByVal vtPassword As Variant, ByVal vtFriendlyName As Variant) As Long Private Declare Function HypInvalidateSSO Lib "HsAddin" () As Long Private Declare Function HypCreateConnection Lib "HsAddin" (ByVal vtSheetName As Variant, ByVal vtUserName As Variant, ByVal vtPassword As Variant, ByVal vtProvider As Variant, ByVal vtProviderURL As Variant, ByVal vtServerName As Variant, ByVal vtApplicationName As Variant, ByVal vtDatabase As Variant, ByVal vtFriendlyName As Variant, ByVal vtDescription As Variant) As Long
Functions to declare…contd. Private Declare Function HypMenuVRefresh Lib "HsAddin" () As Long Private Declare Function HypSetOption Lib "HsAddin" (ByVal vtItem As Variant, ByVal vtOption As Variant, ByVal vtSheetName As Variant) As Long Private Declare Function HypConnectionExists Lib "HsAddin.dll" (ByVal vtConnectionName As Variant) As Variant Private Declare Function HypDisconnect Lib "HsAddin" (ByVal vSheetName As Variant, ByVal vLogoutUser As Boolean) As Long
Functions to declare…contd. Private Declare Function GetCurrentProcessId Lib "kernel32" () As Long Private Declare Function OpenProcess Lib "kernel32" (ByVal dwDesiredAccess As Long, ByVal bInheritHandle As Long, ByVal dwProcessId As Long) As Long Private Declare Function TerminateProcess Lib "kernel32" (ByVal hProcess As Long, ByVal uExitCode As Long) As Long Private Declare Function HypExecuteMenu Lib "HsAddin.dll" (ByVal vtSheetName As Variant, ByVal vtMenuName As Variant) As Long Private Declare Function HypRemoveConnection Lib "HsAddin.dll" (ByVal vtFriendlyName As Variant) As Long
