Performance Optimization in BODS

Performance Optimization in BODS SAP Business Objects Data Services delivers a powerful, enterprise-class data integration and data quality solution. It provides one development, run-time, and management environment that allows you to integrate, transform, improve, and deliver data anywhere, at any frequency. With SAP BusinessObjects Data Services, you can: 

Respond to information needs and accelerate time to market



Deliver the most trusted and accurate information



Maximize operational efficiency with a single platform



Leverage a collaborative, complete, and open solution scalable for any project size Features and Functions: The comprehensive features and functions of SAP BusinessObjects Data Services can enable the following: Agility – Leverage features that can accelerate time to market, encourage reuse and collaboration, increase developer productivity, and create unified skill sets. Trusted information – Improve the reliability of your data with functions for data governance – including transparency, change management, security, and controls – global data support, and end-toend metadata management. Operational excellence – Gain a single data repository, platform, and environment so that costs associated with supporting multiple applications can be reduced; Leverage multi-platform support for 64bit, UNIX, Linux, and Windows. Enterprise class – Benefit from highly scalable performance with support for parallelization and grid computing, enterprise connectivity, and support for service-oriented architecture (SOA). It is recommended that the BODS jobs should be tuned accordingly to be able to complete in less time and utilize less resource. This document deals with the different Performance Optimization techniques needed to fine tune the BODS jobs there by enabling the BODS jobs to complete in less time with less rates of failure.

2

BODS Performance Optimization

BODS is an ETL (Extraction, Transformation and Load) tool that selects data from one data base, processes it in a job server by applying the transformation rules specified in the mapping and loads the data into the specified target database. There are 7 touch points that need to be considered for performance optimization. They are given below:

1.

Source Operating System.

2.

Source Data Base.

3.

Target Operating System.

4.

Target Database.

5.

Network.

6.

Job Server OS.

7.

BODS Repository database.

The BODS Job Server and the BODS Repository Database can reside on the same server. If installed on different servers, Network throughput configured between Job Server and BODS Repository Database will play a key role. The performance tuning methods for each identified touch point are given below.

2.1.1

Source Operating System

The Source Operating System should be tuned to quickly read data from disks. This can be done by the following methods: Set the read-ahead protocol to 64 KB to make the I/O operations fast. The size of read-ahead protocol is by default set to 4-8 KB.

2.1.2

Source Database

The source database should be tuned to make SELECTs as quickly as possible. This can be done by the following methods: Increase the size of Database I/O to match with the OS read-ahead protocol; otherwise it may cause bottle neck there by affecting the performance. Increase size of shared buffer to cache more data in the

database server.

Cache tables that are small enough to fit in the shared buffer by enabling cache at table level. Caching reduces the number of I/Os thereby improving the speed of access to the data. Turn off PARALLEL for small tables. Create indexes on appropriate columns by looking into the dataflow to identify the columns used in the join criteria.

Create Bitmap indexes on the columns with low cardinality as the index does not take much space in the memory, there by resulting in faster SELECTs.

2.1.3

Target Operating System

The Target Operating System should be tuned to quickly write data to disks. This can be done by the following methods: Turn on the asynchronous I/O to make the Input/output operations as fast as possible.

2.1.4

Target Database

The Target Database should be tuned to perform INSERTs and UPDATEs as quickly as possible. This can be done by the following methods: Turn off archive logging. Turn off redo logging for all tables. Tune rollback segments for better performance. Place redo log files and data files on a raw file if possible. Increase the size of the shared buffer.

2.1.5

Network

Even if the Source Database and Target Database are tuned if the network band width is small, there is possibility for the occurrence of bottle neck which hampers the performance. Adjust the size of the Network buffer in the database client so that each client request completely fills a small number of network packets, there by reducing the number of round trips across databases which in turn improves the performance.

2.1.6

Job Server OS

Data Services jobs are multi-threaded applications. Typically a single data flow in a job initiates one ‘al_engine’ process that in turn initiates four threads. For maximum performance benefits: Consider a design that runs one ‘al_engine’ process per CPU at a time. Tune the Job Server OS so that the Data Services threads spread to all available CPUs.

The above performance optimization methods need to be implemented during the environment and infrastructure preparation of BODS components.

2.1.7

Data Services Jobs

The following execution options can be used to improve the performance of BODS jobs: Monitor Sample Rate: If the job processes large amount of data set the ‘Monitor Sample Rate’ to a higher value (maximum being 50,000, default is 1000) to reduce the number of I/O calls to the log file there by improving the performance. If virus scanner is configured on the BODS JobServer, exclude the Data Services log from the virus scan. Otherwise the virus scan scans the Data Services log repeatedly during the execution, which causes performance degradation. Collect Statistics for self-tuning: BODS has a self-tuning capability to determine the cache type by looking into the statistics of previous job executions. The Collect Statistics option needs to be selected during the first execution of the job. BODS collects the statistics for that job and stores in the metadata for that job. In the next execution select the ‘Use Collected statistics’ option to allow BODS to decide the type of cache to be used to execute the job, there by improving the performance of the job. Set the data flow properties like Degree of Parallelism depending upon the number of CPUs available for processing and set the cache type to in-memory if the data volume is less. If source tables are from same schema and using same Data Store, identify the joins in the early phase so that the join can be pushed to the database. Create synonyms for the tables in other schemas to pushdown join conditions of tables belonging to different schemas in a database. Use data base links to connect different schemas in different databases to pushdown the joins of tables available in these schemas to database. Use data transfer (type = ‘TABLE’) to pushdown the complex logic in a dataflow to the database. Do not use advanced features like Run Distinct as Separate process etc available in ‘Advanced’ tab in Query transform, as it starts multiple sub-processes which causes heavy traffic between the processes and there by lead to the termination of job. Do not use Data Transfer transform unless required. (Use table type if required as it is more reliable.). SAP suggests that Data Transform is not a reliable transform and hence recommends to not using it unless required. Turn off the Cache option for the tables with larger amounts of data. Cache is turned on by default for every source table. Make sure that there are indexes created on key columns for these tables on which cache is turned off. Do not use BODS functions like job_name(), instead initialize a variable in a script and use that variable for mapping in query transforms. Use Join where ever applicable in place of look up transform as the look up transform has to access the table for each and every record which increases the number of I/O operations to database. Use Query transforms to split the data in place of Case transforms as it is a costly transform in BODS.

SAP Business Objects Data Services Target Table Options: created by Ankit Sharma on Sep 18, 2013 11:37 AM, last modified by Ankit Sharma on Sep 23, 2013 12:48 PM Version 3

This will help you to understand how to make use of the different options available in the target table while loading data.

Below is the descriptions of all the options with the sample data.

Auto Correct Load:

Auto correct loading ensures that the same row is not duplicated in a target table

Input Data Screen Short:

Target Option Window: Following steps need to be done

Uncheck the Delete sata from Table before Loading

Before Updating in Update Control:

After Updating:

Now go to Source Data and add two Duplicate rows Before Updating:

After Updating:

After Creation of Jobs in Data Services Designer

Now Apply Auto Correct Load on Target Tables

Note: After apply Auto Correct Load Duplicate values are removed from Target Tables Now Execute the Job

Output:

Ignore Column Option:

If a matching row exists, the row is updated depending on the values of Ignore columns with value, and Ignore columns with null:

Ignore Columns Options are Worked with Auto Correct Load:

• When the Ignore columns with null option is set to Yes and the column data from the source is NULL, then the corresponding column in the target table is not updated. Otherwise, the corresponding target column is updated as NULL since the source column is NULL.

Now Go To Target Table Properties:

Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

View of Source table After Null Values Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

Ignore Column with Value Option:

If a matching row exists, the row is updated depending on the values of Ignore columns with value, and Ignore columns with null:

• When the column data from the source matches the value in Ignore columns with value, the corresponding column in the target table is not updated. The value may be spaces. Otherwise, the corresponding column in the target is updated with the source data.

Now Go To Target Table Properties:

Before Update in the Target Table Properties:

Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

Update Key Columns in Update Control:

Now Go To Target Table Properties:

Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information: Before Update:

After Update in Source Table:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

Sending IDOC from BODS to SAP

Document Purpose This document provides steps required to load data in SAP using one of the SAP interfaces. This document will be used at the time of proof of concept to demonstrate the ability of Business Objects Data Services (BODS) to invoke IDOC interface to load data in SAP.

The main objectives are:

 

Loading data in SAP using IDOC



Display required configurations step-by-step & invocation



Demonstrate actual sample load using BODS job

Demonstrate how BODS can be used to load data using IDOC

Overview: This document provides systematic approach for the data load in three sections: 1. 2. 3.

Configuration Required at SAP Side: Configuration Required at BODS Side: Demonstration Sample Load using BODS Job: Each of above sections provide information as in how to configure the specific parameters required to get both SAP & BODS ready to communicate with each other. Screen shots provided for better understanding of the parameters & their configuration.

Configuration Required On SAP (ECC 6.0) Side

Following is the required configuration on SAP side.

1.Display IMG [ /nsale ] Connect to SAP System using SAP GUI Use Transaction Code (TCODE) /nsale to “Display IMG”

Screenshot 1: Display

IMG screen

Expand Logical Systems tree & click on the Clock icon in front of “Define Logical System”

2.

Defining Logical System

Screenshot 2: Defining

Logical System screen

Click on the “New Entry” Button

For SAP Inbound IDOC One Logical system (FRMDICL100) is required Create a Logical system as shown below

Screenshot 3: Overview of added entries

Click on the save button to save both the Logical System.

Verify that the LS FRMDICL100 is present in the SAP system

Screenshot 4: Overview of Logical Systems

3. Create Partner Profile [ /nwe20 ] Use TCODE /nwe20 to configure partner profile.

Screenshot 05: Partner Profile screen Click on Create button to create new partner profile Create partner profile as shown below

Note : 1. 2.

Provide partner no. “FRMDICL100 “ i.e same as BODS Logical System. Agent 50010120 “EDI Department” is necessary for communication with non-SAP systems like BODS. Without this agent IDOC communication will not be successful.

Screenshot 06: Partner Profile deatil screen

Click on the Save button to save the partner profile After saving; the “Create Inbound parameter” will be activated in “Inbound paramtrs”

Screenshot 07: Partner Profile create Inbound paramtrs screen

Click on “Create Inbound parameter” Green Plus button in “Inbound paramtrs” section

Screenshot 08: Partner Profile Inbound paramtrs screen Note : In Inbound options tab, select appropriate process code relevant to message type. (In this case; for MATMAS message type, process code is MATM)

Screenshot 09

In "Post processing : permitted agent" tab provide details as shown above. Save "Partner profile: Inbound parameters" using Save button After saving verify Partner Profile & Inbound Parameter added.

Screenshot 10

Configuration Required On BODS Side

Following is the required configuration on BODS side

1. Create SAP Datastores

Create SAP Datastore by providing required SAP Application Server Information & creadentials

Screenshot 11: SAP Datastore in BODS

Right click on IDOCs and select “Import By Name…”

Screenshot 12: IDOC list under SAP datastore in BODS

Screenshot 13: Import IDOC screen

Provide Type and Name as shown above and click on “Import” to import IDOC in datastore

2. Generating input XML schema To create the input XML structure with the correct datatypes quickly drop the IDOC as message source onto the dataflow, go into this object and rightclick the schema. There, you can ask the Designer to generate the entire structure as XML schema, thus you get all columns, schemas including the datatypes.

Screenshot 14: MATMAS01 IDOC screen

“MATMAS01_OUT” this XML file source is created using generated schema in“File Format” tab.

Screenshot 15: XML Schema screen

3. Build Batch Job Build a Batch Job with a DataFlow inside… DataFlow is as Follows

Screenshot 16: Batch Job screen Double click on source XML File Source configure it as shown below. Note: For demonstration purpose we have used XML file that was output of the Real-Time Job designed in article "Receiving IDOC In BODS from SAP".

Important : Make sure that in XML file Sending Partner setting i.e. should be same as BODS logical system name. (FRMIDCL100) as shown below.

Screenshot 18: IDOC out XML file strucutre screen

Unnest by right clicking on MATMAS01_OUT schema in "Query" as shown below

Screenshot 19: IDOC structre screen Drag the IDOC MATMAS01 from SAP DataStore as shown

Screenshot 20: IDOC in SAP Datastore Note: Make sure to select option "Make IDOC Message Target" Configure “IDOC Message Target”

Screenshot 21: BODS side IDOC configuration

Receiving IDOC in Business Objects Data Services (BODS) from SAP created by ROSHAN CHHAJED on Jul 16, 2012 10:51 AM, last modified by ROSHAN CHHAJED on Jul 16, 2012 5:43 PM Version 1

Document Purpose

This document provides steps required to load data in BODS from SAP using IDOC interface. This document will be used at the time of proof of concept to demonstrate the ability of Business Objects Data Services (BODS) Real-Time job to receive data using IDOC interface from SAP.

The main objectives are:

Loading data in BODS from SAP using IDOC Demonstrate how BODS can be used to receive data using IDOC Display required configurations step-by-step & invocation Demonstrate actual sample load using BODS Real-Time job

Overview:

This document provides systematic approach for the data load in three sections: 1. 2. 3.

Configuration Required at SAP Side: Configuration Required at BODS Side: Demonstration Sample Load using BODS Job:

Each of above sections provide information as in how to configure the specific parameters required to get both SAP & BODS ready to communicate with each other. Screen shots provided for better understanding of the parameters & their configuration.

SAP [ECC 6.0] Configuration for Outbound IDOC Communication

1. Display IMG [/nsale]

Connect to SAP System using SAP [ECC 6.0] GUI Use Transaction Code (TCODE) /nsale to “Display IMG”

1.0 Display IMG

2. Defining Logical System

Expand Logical Systems tree & click on the Clock icon in front of “Define Logical Systems”

2.0 Logical Systems: overview

Click on the “New Entries” Button

For SAP Outbound IDOC, two Logical Systems are required First Logical System for SAP (ID4CLNT100) Second Logical System for BODS (ID100CBODS)

Create these two Logical System as shown below

2.1 New Entries for Logical System

Click on the "SAVE" Button to save both the Logical Systems. Confirmation message “Data was Saved” will be displayed as shown below.

2.2 Data Saved Conformation

Verify two (2) Logical Systems are present in the SAP

2.3 Verifying Logical System

3. Configure RFC Connection [/nsm59]

Use TCODE "/nsm59" to configure RFC Connection

3.0 Configurations of RFC Connection

Select TCP/IP Connection as shown above & click on the Create Button to configure new RFC Connection.

3.1 Creation of New RFC Destination: Technical Settings

Provide required information as shown above

Note: We are using registered Program ID as “DI_RFC_Client”. This could be any simple name, but the same name should be used while configuring “RFC Clients” using BODS Administrator. (Ref: 2.Client Interfaces, page no 21)

Click on the MDMP & Unicode tab

3.2 Creation of New RFC Destination: MDMP & Unicode

Note: We have selected “Unicode” radio button because BODS Target RFC Client is Unicode. Important: There are two buttons “Connection Test” and “Unicode Test” we will test RFC Connection after “Configuring RFC Client” using BODS Administrator in Steps provided later in this document. (Ref: 3. Testing RFC Client from SAP, page no 23)

4. Create Transactional RFC Port [/nwe21] Use TCODE "/nwe21" to configure the RFC Port

Expand Ports tree to go to “Transactional RFC”

4.0 Creation of New Transactional RFC Port

Select Transactional RFC and click on “Create” Button.

4.1 Transactional RFC Port Name

Provide Name "DI_PORT" as shown above. Click on Continue button

Port Name "DI_PORT" will appear in the list as shown below.

4.2 Transactional RFC Ports Overview

Click on the DI_PORT to provide Description & RFC destination (“DI_RFC_DESTINATION” ) that we have created in earlier section (Ref: 3.Configure RFC Connection, page no 7)

5. Create Partner Profile [/nwe20]

Use TCODE "/nwe20" to configure Partner Profile.

5.0 Creation of New Partner Profile

Click on Create button to create new partner profile Create partner profile as shown below Note : 1. Provide partner no. “ID100CBODS “ i.e same as BODS Logical System.

2. Agent 50010120 “EDI Department” is necessary to communicate with non-SAP systems like BODS. Without this agent IDOC communication will not be successful.

5.1 Partner Profile Permitted Agent

Click on the Save button to save the partner profile After saving the "Create outbound parameter" will be activated in “Outbound paramtrs”

5.2 Create outbound paramtrs.

Click on “Create outbound parameter” button in “Outbound paramtrs” section.

5.3 Partner Profile: Outbound parameters

Provide the Message Type "MATMAS" & IDoc Basic type "MATMAS01" Save "Partner profile: Outbound parameters" using Save button Confirm Partner Profile & Outbound Parameters added.

5.4 Outbound Parameters Overview

6. Change Distribution Model [/nbd64]

Use TCODE "/nbd64" to configure Change Distribution Model

Screen Shot 6.0 Change Distribution Model

Click on the “Create Model View”

Note: If this button is inactive then select the Model View in left hand tree and click on Edit button to enable “Create Model View” button & active.

6.1 Create Model View

Provide Short text & Technical name as follows: Tech name same as short text.Click continue button

The DI_MODEL will be added to the list as shown below

6.2 Distribution Model Changed overview

Select DI_MODEL and click on “Add Message Type” button

6.3 Add Message Type for Changed Distribution Model

Sender is SAP Logical System & Receiver is BODS Logical System. Provide all information as shown above and click on continue button. The Distribution Model created is shown below.

6.4 over View of Distribution Model

Check if IDOC distribution is turned on.

Run transaction "/nbd61" and make sure the checkbox is set.

6.5 Activation Status

Create Real Time Job in BODS

1. Create SAP Data store Go to Data stores tab in BODS Designer, right click on blank area to create "NEW". Create SAP Datastore by providing required SAP Application Server Information & creadentials

1.0 Creation of SAP Data Store

Expand the "SAP_TTL_Client100_DS" datastore Select the IDOCs and Right click , click on “Import By Name…”

1.1 IDOC'S importing

1.2 IDOC Import by Name

Provide Type & Name as shown above & click on “Import” to import IDOC in data-store

2. Build Real Time Job

Build a Real Time Job with a DataFlow inside…

2.0 Real Time Job Overview.

Data Flow is as Follows

2.1 Data Flow

Query is used to Unnest as follows

2.2 Query Unnest

Note: You can unnest by right clicking on MATMAS01 as shown below

2.3 Unnest Option

The Target XML file configuration is a follows :

2.4 Target XML file Configuration

Note: Make sure “Make Port” checkbox is checked.

Configuring RT Services and RFC for IDOC through Web Administrator

1. Configuring Real Time Service & Service Provider Open BODS Web Administrator Expand Real-Time tree and select Real-Time Services to do Real-Time Service Configuration Provide required information as shown below:

1.0 Web Administrator: Real Time Services Configuration

Job Name is Real Time Job name that you have created in earlier stage in designer (Ref: 2.Build Real Time Job, page no 18) Browse the Job and select Job name In service provider settings - Click on Add button to add Job Server and make it enabled.

Once the Service is configured you can start the service by selecting the service & click on start button.

1.1 Real Time Services Status

2. Client Interfaces

Using BODS Web Administrator Expand Real-Time tree and select Client Interfaces to do RFC Client Configuration Provide required information as shown below

2.0 RFC Client Configurations

Click on Apply button

Note: RFC Program ID must be same as register Program ID given while configuring RFC Connection. (Ref: 3.Configure RFC Connection, page no 7) Go to “List of Supported IDOC” tab & then click on “Add” button to add IDOC.

2.1 Supported IDOC

Provide IDOC Type which is BASIC IDOC TYPE in SAP. (Ref: Screen Shot 5.3 Partner Profile: Outbound parameters, page no 13) Service Name must be same as Real Time Service we have created & click on Apply button to get the IDoc Information registered. List of Supported IDOC will be displayed as shown below

2.2 Lists of Supported IDOC's

Note: You may add more than one iDOC type per RFC Client Once the RFC Client configured you can start RFC Client. Make sure RFC Client is started & status is “Waiting for Idoc” as shown below:

Shot 2.3 RFC Client Interface Status

3. Testing RFC Client from SAP Use TCODE "/nsm59" to go to Configuration of RFC Connections Open TCP/IP Connections & locate DI_RFC_DESTINATION Double click & open DI_RFC_DESTINATION Click on “Connection Test” button

3.0 Testing RFC Client from SAP

If everything configured properly, you will see success message as shown below.

3.1 Connection Test Success

Note: If Connection test failed, make sure that the RFC client on BODS side started properly with green icon.

Sending Material from SAP [/nbd10] 1.

Sending Material MATMAS from SAP

Use TCODE "/nbd10" to "Send Material"

1.0 Sending Material from SAP

Click on the match code button

1.1 Material Numbers / Material Description

Click on “Start Search “button for finding the entries for material numbers.

1.2 Selections of Material and Material Description

Select any Material number from the list displayed above and click on OK button. Selected material number will appear in front of "Material" on Send Material screen. Provide required fields as shown below & make sure the "Logical system" is BODS Logical System ID100CBODS

1.3 Send Material

Click on Execute button. We will get couple of following messages.

1.4 IDOC Information after material sent Click on “OK”

2. Check Received MATMAS IDOC in BODS Using BODS Web Admin Go to BODS Web Admin, click on Real Time Services to check that one IDOC request received as shown below

2.0 Received Requests in Web Administrator: Real Time Services

1. Checking the output data in XML File Go to BODS Designer Open the Real-Time Job Dataflow Go to the target MATMAS01_OUT.xml & click on view data to confirm the data received Generated output XML file data as follows