228069314-Odi.pdf

Using Oracle Data Integrator with Essbase, Planning and the Rest of the Oracle EPM Products Edward Roske [email protected] BLOG: LookSmarter.blogspot.com WEBSITE: www.interrel.com TWITTER: ERoske

2

About interRel  2008 & 2009 Oracle Titan Award winner - EPM Solution of the year  2008 Oracle EPM Excellence Award  2009 Oracle EPM/BI Innovation Award  One of the fastest growing companies in the world (Inc. Magazine, ’08 & ‘09)  Two of the three Hyperion Oracle ACE Directors in the world  Founding Hyperion Platinum Partner; now Oracle Certified Partner  Focused exclusively on Oracle Hyperion EPM software  Consulting  Training  Infrastructure and Installation  Support  Software sales 4

 5 Hyperion Books Available:      

Essbase (7): Complete Guide Essbase System 9: Complete Guide Essbase System 9: End User Guide Smart View 11: End User Guide Essbase 11: Admin Guide eBooks available on Amazon Kindle

 Just out!  Hyperion Planning for End Users

 Coming Soon  Hyperion Planning for Admins (days now…)  Hyperion Financial Management (Q1 2010)  5

Copyright © 2007, Hyperion. All rights reserved.

To order, check out www.lulu.com

Disclaimer

 These slides represent the work and opinions of the presenter and do not constitute official positions of Oracle or any other organization.  This material has not been peer reviewed and is presented here with the permission of the presenter.  This material should not be reproduced without the written permission of interRel Consulting.

Agenda

      

What is ODI? A Real World Example – ODI and Planning ODI Demo A glimpse into the concepts Tips Getting Started Questions and Answers

What is ODI?

Oracle Data Integrator

Oracle Data Integrator (ODI) Overview

    

Integration tool that replaces HAL and DIM Used to move and transform information across systems Processes are based on metadata of existing systems Only metadata is stored in ODI, not data Sometimes referred to as Sunopsis

How is ODI used with EPM applications?

   

Interface metadata and data between source data systems Build dimensions Load data Transfer data across files and relational databases and the EPM applications  All supported with Hyperion Planning, Financial Management and Essbase  Not supported with EPMA

Oracle ODI to EPM Functionality Planning

Essbase

HFM

Metadata discovery & model creation

P

P

P

Load data

P P O O

P P P P

P P P P

Load Metadata Extract data Extract metadata Other

Refresh to Essbase

Calc, post and pre MaxL scripts

Consolidate

Marketing Message

 Three main points:  Extract-Load & Transform (E-LT) architecture  Removes ETL server from infrastructure  USP = Native target database performs transformations  Declarative Design  Developer define the rules, ODI:  Writes the data flows  Manages the data flow complexity  Writes the behind-the-scenes code

 Knowledge Modules  Similar to HAL adapters  Talks (and almost always writes) to almost everything

A Real World Integration with ODI 1

GL SAP

Accounts / Entities

DW Oracle

Planning Fcst Data

Consolidated Results

2

Accounts / Entities

6

HFM

7

3

Essbase

Accounts / Entities Sales by Customer DB

4

5

Customer Dimension Sales by Customer Data 8

MaxL – Run Allocations, Aggregate Data, etc

ODI Demo

A Glimpse

ODI Components and Concepts

ODI Components Source Systems

Target Systems

ODI (ELT) Extract–Load-Transform

ODI Components Topology Manager Configure Architecture / Infrastructure of source & target technologies

Security Manager

Designer

Master Repository

Work Repository

(Infrastructure)

(Interfaces)

Design & execute metadata models and interfaces to move data across systems

Maintain ODI Security

Operator Monitor Sessions Review errors

Agent Coordinates communication & scheduling

Metadata Navigator Web access to mettadata repository

Components GUI

Purpose

Users

Repository

• • • •

Administrators Developers Operators

Designer

• Reverse Engineer • Develop Projects • Release Scenarios

Developers Metadata Administrators

Operator

• Operate Production • Monitor Sessions

Production Operators

Topology Manager

• Physical and Logical Architecture of the infrastructure (servers, schemas and agents)

Infrastructure Administrators

Security Manager

• User privileges

Security Administrators

Configuration Info Metadata OLTP RDBMS Sunopsis Infrastructure

Deployment  Typically have 1 Master Repository that talks to multiple Work Repository (Dev, Test & Prod).  Can set the production work repository to execution only

Topology Security Versioning Master Repository (Infrastructure)

Work Repository \ (s) \ (Interfaces)

Models Projects Execution

Topology Manager

   

   

Representation of the Information System Connect to resources for integration purposes Contains: Physical Architecture  Data Server  Physical Schemas and Physical Agents Logical Architecture (logical schemas & logical agents) Contexts – Links logical and physical architectures Repositories – Master and attached work repositories Languages - different types of languages available

A Sample Topology Logical Architecture dbo.database MSSQL

Rep.App1 Essbase

Rep.App2 Essbase

PlanSamp. Consol Planning

FinOp.Plan 1 Planning

C:\ODI\File File

Physical Architecture Windows Machine X (Development Environment)

Essbase

DW dbo.databa se

Rep.App1

Source Files

Windows Machine Y (Production Environment)

Planning PlanSamp. Consol

C:\ODI\File Rep.App2

DW dbo.databa se

FinOp.Plan 1

Data Server

Physical Schema

Logical Schema

Physical Agents

Windows Machine Z (Production Environment)

Essbase

Planning

Rep.App1

PlanSamp. Consol

Source Files C:\ODI\File

Rep.App2 FinOp.Plan 1

Logical Agents

Languages

 Languages used to generate code for the specific technologies

Designer

 Designer includes:  Models – Metadata for your source and target systems  Projects - Integrations that move data from your source and target systems  Interfaces  Packages  Procedures  Variables  Knowledge Modules  The Designer module stores this information in a work repository, references the topology and the security information defined in the master repository.

Designing an Interface  Before you start building an interface you:  Import Knowledge Modules into the project  Create your Metadata Models  Then in the interface you:  Define your source and target  Define the mapping and translations (business rules)  Define the flow requirements  Execute  To automate execution, move on to:  Packages  Procedures  Variables  Sequencing

Knowledge Modules

 Knowledge Modules (KMs) are components with built in knowledge (functions and logic) that connect technologies to ODI.  Similar to HAL Adapters, but many more connection options.  KMs connect to the technology, extract data from it, transform the data, check it, integrate it, etc.  You can find a complete list of KMs at http://www.oracle.com/technology/products/oracle-dataintegrator/10.1.3/htdocs/documentation/oracledi_km_refer ence.pdf

Knowledge Modules 

Interfaces

 LKM (Loading Knowledge Modules) are used to extract data from the source database tables and other systems (files, middleware, mainframe, etc.).  IKM (Integration Knowledge Modules) are used to integrate (load) data to the target system.  CKM (Check Knowledge Modules) are used to check that constraints on the sources and targets are not violated.  .

Knowledge Modules  Models  RKM (Reverse Knowledge Modules) are used to perform a customized reverse-engineering of data models for a specific technology.  JKM (Journalizing Knowledge Modules) are used to create a journal of data modifications (insert, update and delete) of the source databases to keep track of the changes.

Knowledge Modules  Others  SKM (Service Knowledge Modules) are used to generate the code required for creating dataservices.

An Example: Hyperion Planning Model

Dimensions

Planning Metadata

An Example: Hyperion Essbase Model

Dimensions Essbase Metadata

Interfaces

 An ODI object which loads one target data store with data from one or more sources, based on business rules implemented as mappings

Inside the Interface: Mappings

More on Mappings

Execute On Expression Editor Target Column

Update

Multiple Sources, Joins and Filters

 Joins (similar to Joins in RDMS)  Links multiple data sources together  Uses SQL Clauses  Multiple sources in a single interface must be connected with a join.  Create joins by dragging and dropping columns together  Alternatively, you can use the expression editor to add them manually  Advanced joins  Connect models from different technologies or model / schemas. (Heterogeneous Join)  Connect 2 or more datastores

Join Definition

SQL Syntax Specify Join type

Use ISO-92 Syntax, Join Order, Autocalculate Order

Filter

 Reduce amount of data processed by the interface by applying a filter.  Funnel icon represents filter applied on the source  Use the Expression Editor or manually type in the SQL.  Test filter, right-click and select Data.

Expression Editor

Flow Tab

SUNOPSIS_MEMORY_ENGINE

 SUNOPSIS_MEMORY_ENGINE sits between source and target  Temporary staging area that is created during interfaces to facilitate mapping and translation  Not recommended for large data sets for performance reasons  Can be difficult to debug  We recommend configuring a separate staging area

Operator: Checking Status

Automation

 Packages - Group of steps to complete a workflow.  Can include:  Interfaces  OS Commands  Procedures  Workflow – can include error trapping and email status notifications  Scenarios – can schedule as a scenario or call a scenario from a batch file  Variables - can have variables defined to pass parameters (i.e. file name, current month, etc.)

Create Error Log Files in the Interface  Enable logging in the interface in the files to obtain more details or send information to a text file for use in debugging.

Errors

 Errors are not always ODI errors, they are typically being sent from other technologies (Oracle EPM in our case)  You will begin to see commonality of errors but they are not always descriptive or useful.  Plan extra time for debugging interfaces

Real Life Tips

Tips – General Information  Good sources of information:  interRel 3 day ODI and EPM training class (available this fall)  OTN Forum: http://forums.oracle.com/forums/forum.jspa?forumID=3 74  http://john-goodwin.blogspot.com/  Get the 10.1.3.5 release  No patching required for Essbase, Planning, and HFM  Available at: http://www.oracle.com/technology/software/products/od i/index.html

Design and Development Tips  Whiteboard the Topology before configuring in ODI  Standardize naming conventions for topology components  Incorporate as many error logs and error trapping processes as possible.  Consider using a staging area versus Sunopsis Memory Engine. This will allow performance optimization and error resolution.  If the source is a relational database, perform as many translations in a view as possible.

Design and Development Tips  Recommend against using comma as a file delimiter for source files. Use | or !.  Versus loading Essbase Data with ODI, write a MaxL to load data to Essbase and add it as a step in the ODI package.  Plan extra development time, learning curve is steep and it is more complex than HAL.

Project Tasks: Getting Started

          

Plan the scope, identify source and target Establish success criteria Identify the resources Define the Topology Define your Models Create Interfaces Unit Testing Automate with Packages Schedule with Scenarios Integration Testing Deploy

How long will take?

 No short answer, depends on the complexity of the interface. If this is the first time, allow ample time to debug issues.  Minimize risk by:  Working with infrastructure team for connectivity issues  Keeping data transformations simple or nonexistent in the initial development (prototype)  Then consider the opportunities to automate translations or business rules  Plan out your topology on a whiteboard first! It is critical to understand what links to what to avoid silly mistakes.  Use Relational Sources where possible.

OTHER

Security Manager

   

Separate from Shared Services Manage security for all ODI modules Create Users & Profiles Assign access  ODI Objects (data servers, projects, etc)  Methods (Edit, Create, Delete)  Servers

Metadata Navigator

 J2EE application that provides web access to the repositories  View Flow maps, trace the source of all data and drill down to field level  Possible to trigger and monitor processing jobs from a web browser using this tool  By default, installed on Jakarta Tomcat Application Server

Oracle Data Profiling and Quality

 Integrator extends data quality features of Oracle Data Integrator to provide more advanced data governance capabilities.  Data Cleansing  Duplicates  Parsing  Data Enrichment  User driven rules  Integrated with ODI

ODI and EPM Training

Coming Soon from interRel Training!

Introduction to ODI for EPM Administrators This class will be useful for administrators or developers that will design, develop and / or deploy data integrations that update Essbase, Planning or HFM with ODI. Includes several hours of hands on workshops !!!

Thank You.

Edward Roske [email protected] BLOG: LookSmarter.blogspot.com WEBSITE: www.interrel.com TWITTER: ERoske