Ab Initio EME Concepts

Ab Initio EME EME Training Course – Sivasakthipriya

Course Content EME (Enterprise Meta Environment) • Features of EME • Projects, Sandbox & Parameters • Reports, Versioning and Tagging • Checking out Projects, Sandbox, graphs and files • Editing the Sandbox, graphs and files • Checking in Projects, Sandbox, graphs and files • Dependency analysis • Air commands

May 18, 2010

Features of EME • What is EME • Why EME • Interfaces to the EME • EME Data Store Settings

July 6, 2010

What is EME? •Enterprise Meta Environment is high performance object oriented storage system that inventories and manages various kinds of information associated with Ab Initio applications which may range from design information to operational data. •It acts as the hub for data and definitions. •In simple terms, it is a repository, which contains data about data – metadata. •It consists of two basic types of metadata ¾ Technical metadata – application related business rules, record formats and execution statistics ¾ Business metadata – user defined documentation of what you deem important to track in your company , such as logical models, data dictionaries, roles and responsibilities. May 18, 2010

Why EME? • Avoid setup scripts, use EME parameters – more standardisation • Source Control • Dependency and impact analysis for the graphs in the repository Interfaces to the EME •GDE •EME Web interface to Technical Metadata •EME Web interface to Enterprise Metadata

May 18, 2010

EME Data Store Settings •EME Data store is a specific instance of EME in the environment. • This is a repository where different versions of code and its related data like the record formats, transformations etc are maintained. • At any point of time a user can connect to only one such EME repository instance. •To access an EME Data store, go to Settings in the GDE Menu and click on to Enable EME Connections which is shown below.

May 18, 2010

• After clicking the Enable EME Connections , The GDE is connected to the EME data store. It is shown below.

May 18, 2010

Projects in EME • What is a Project • Project Structure • Structure of project in EME • Different Types of Project ¾ Public project ¾ Private project

July 6, 2010

What is a Project ? •Physically, each project in the EME data store is a collection of graphs, plans & related files stored in a well described structure. •Logically, a project is a fairly self–contained and largely independent content area , a suite of graphs and associated files that accomplishes a business goal.

Project structure •The directory tree of a project contains subdirectories for graphs, scripts, sqls , transforms etc. •/Projects – default directory under which specific projects reside.

May 18, 2010

Structure of a project in EME /Projects

Project1

mp

Sub directory for AI Graphs

run

Sub directory for deployed shell scripts

xfr

Sub directory for transform files

dml

Sub directory for record format files

db

Sub directory for database interface files

bin

Sub directory for tools and utilities

sql

Sub directory for sql queries

Project2

May 18, 2010

Different Types of Project Public Project • A Public project shares their data and metadata with other projects • A Project that is included by other Projects is termed as a Public Project

Private Project • A Private project are not expected to be shared with other projects • . Projects including public Projects are known as Private Projects.

May 18, 2010

Sandbox • What is a Sandbox • Structure of Sandbox • Sandbox vs Projects in EME • Environment Structure • Creating a Project by creating a Sandbox

July 6, 2010

What is a Sandbox? •Sandboxes are the user work areas in order to work on files in EME Project •To work on Projects, they must be checked out to a user work area on the file system where development and modification of code can happen. •This working area on the file system is known as Sandbox. •After the changes are complete the code is checked in from the user sandbox area to the EME data store •This action creates a new version of the code in the EME Data store.

May 18, 2010

Structure of Sandbox •Sandbox has the same structure of Projects in the EME but it lies in different file system. It is shown below. /Mysandbox

Project1

mp

Sub directory for AI Graphs

run

Sub directory for deployed shell scripts

xfr

Sub directory for transform files

dml

Sub directory for record format files

db

Sub directory for database interface files

bin

Sub directory for tools and utilities

sql

Sub directory for sql queries

Project2

May 18, 2010

Sandbox vs. Projects in EME •

• •

Sandboxes are work areas used to develop, test or run code associated with a given project. Only one version of the code can be held within the sandbox at any time. The EME Datastore contains all versions of the code that have been checked into it. A particular sandbox is associated with only one Project where as a Project can be checked out to a number of sandboxes.

Check-in

Check-out

May 18, 2010

Environment Structure

Common Sandbox (Optional) /Common/Project1/mp

EME User Sandboxes /Mysandbox/Project1/mp

Repository /Projects/Project1/mp

check in Check out inCheck in

16 May 18, 2010

Creating a project by creating a Sandbox •Create an empty sandbox in the GDE. Select Project>Create sandbox dialog. •Host : server name •Directory : The sandbox path •Location parameter : leave default or give some name •Click ok and the sandbox will be created in user area. •Check in the sandbox and put it as Project in EME.

May 18, 2010

Parameters • What is a Parameter • Graph Parameter • Sandbox Parameter or Project parameter

July 6, 2010

What is a Parameter? •A parameter is a name-value pair with some additional attributes that determine when and how to interpret or resolve its value. •Parameters are used to provide logical names to physical location and should always be used instead of hard coded paths in graphs. •Parameters are used to change the behaviour of graphs and projects in a controlled and uniform way •There are two types of parameters. ¾ Graph Parameters ¾ Project Parameters or Sandbox Parameters

May 18, 2010

Graph parameters •Graph parameters are specific to the individual graphs and are private to them. They affect execution of the graph for which they have been defined. •Graph parameters can be defined by navigating to Edit>Parameters in the GDE which opens the graph parameters editor.

May 18, 2010

Project Parameters or Sandbox Parameters •Project parameters are inherited by all the graphs in the Project and are accessed from the GDE by the sandbox parameter editor in Project>Edit Sandbox>Parameters. •This shows a dialog box prompting to enter the sandbox path. Choose the correct host and the sandbox path and press OK to open the sandbox parameter editor, which exactly like the graph parameter editor shown as above.

May 18, 2010

• Reports • Version control & Tagging

July 6, 2010

Reports •Reports present summary EME data • View the report using EME Web Interface •The below list of reports can be taken from EME ¾ Graph reports ¾ Dataset reports ¾ Job report ¾ Directory report ¾ Project report

May 18, 2010

Version Control and Tagging • Each object under EME source control, which may be a file, a directory or a Project, exist as a series of versions, each of which is a representation of what was checked in by some user. • It can optionally have a textual description attached to it called as a tag and a description as a comment. • Each version is separately numbered and can be accessed by either the version number or the tag attached to it. • Version numbers, which are integers and tags, are global to the whole EME Datastore. • Tags are the basic units during migration of code across EME Datastore instances.

May 18, 2010

Checking out Projects, Sandbox, graphs and files •Check out •Check out Process in GDE

July 6, 2010

Check out • The Ab Initio GDE provides wizards to check out code from the EME to sandbox. • Check out updates the sandbox with the particular version of code that is being checked out from the EME. By default the latest version of any object is checked out. • Even any version of code can be checked out.

Check out Process • Select Project>Checkout. Specify the Sandbox directory name and EME path. • Specify the EME name, Project path in EME and Sandbox path in user area. It is shown below.

May 18, 2010

•

After specifying all details click on to check out button.

•

It will check out all the things from EME to sandbox

•

To check out a project in existing sandbox , right click the sandbox name in the sandbox view and select check out

•

To check out only the current graph or plan , click the check out current document icon( ) on the GDE menu bar

•

To check out any file in the project, right click the file in sandbox view and select check out.

May 18, 2010

Editing the Sandbox, graphs and files •Need for Locking •Locking an Object •Lock Button

July 6, 2010

Need for locking •Lock is used to edit the graph or files. •The lock button on the GDE indicates the status of an object with regard to locking. •Locking the graph prevents other users from changing it at the same time you are working on it. •Otherwise , conflicts can occur between sandbox and data store version of files.

Locking an Object •Clicking the lock button on the GDE button •Choose file> lock on the GDE menu bar •Right click a file on sandbox view and select lock

May 18, 2010

Locking Button •The lock button’s appearance identifies the locking status of the object you are viewing •If an object has never been checked in , there is no lock associated with it. The lock button on the tool bar is an open gray lock on gray back ground. •When a graph or file is unlocked , the lock button on its toolbar is an open yellow lock against a gray back ground. A unlocked object is read only and cannot be edited in the GDE. •Sometimes an object lock button indicates that another user already owns the lock , which means you cannot edit the object . In this case , the lock button is a closed lock on red back ground. •You can click the lock button and display the owner of the lock.

May 18, 2010

•

There is no way in the GDE to break another users lock.

•

Once you set the lock , you are the owner. You can modify , save, and check in the locked object. The lock button is now a closed recessed in green back ground

•

Locking when sandbox copy of the object is out of date. It will show below dialog box.

•

Releasing a lock without checking in changes , it will displays a dialog. Save changes or release lock without reverting

May 18, 2010

Checking in Projects, Sandbox, graphs and files •Check in •Check in process in GDE

Internal and

July 6, 2010

Check in •Once you have edited your sandbox files, you have to check them in to put the new versions under source control in the EME datastore and make them available to other users.

Check in process in GDE •To check in an entire sandbox , right click the sandbox name in the sandbox view and select check in , or select Project>check in on the GDE menu bar. •To check in only the graph , you are currently editing , click the check in document icon ( ) on the GDE menu bar. •To check in any sandbox file, right click the file in the sandbox view and select check in.

May 18, 2010

Dependency Analysis

July 6, 2010

Dependency Analysis •Dependency Analysis is a process by which the EME examines a project in its entirely and traces how data is transformed and transferred, field by field , from component to component within and between graphs. • Using Dependency analysis we can observe the following things: • How operations that occur later in a graph(“downstream“) are affected by components earlier (“upstream”) in the graph • What data is operated on by each component, and how. • All details of every component in the project. • What happens to each field in a dataset throughout a graph. • All the graphs that use a particular dataset.

May 18, 2010

There are four ways in which the dependency analysis can be controlled : 1) It can be directly invoked in the GDE, by selecting Project > Analysis……. To start the Analysis wizard. 2) It can be directly invoked from the command line, by running the air project analyze- dependencies command. 3) It can be invokes implicitly at check-in, depending on the settings you select in the Check-in wizard’s Advanced Options under the Analysis tab. 4) You can also create an analysis_level parameter and use it to specify the amount of dependency analysis you wish to take place during check-in. The setting of this parameter(if it exists) restricts all other analysis level settings.

May 18, 2010

AIR Commands •Command line syntax of air command •Air Sandbox group •Air Project group •Air object group •Air Lock group •Air repository group •Air Branch group

July 6, 2010

Command line syntax of Air command: •The air utility is a command-line or shell-script interface for administering and managing the EME. • All air commands have the following command-line syntax. air[-root root] [-version version | tagname][-remote] ¾ -root root specifies root of the data store.(Optional) ¾ -version version (Optional) specifies the version number of the data store to be accessed by the air command. The version number can be specified as an integer or as a tag. ¾ A tag is a symbolic name that you can associate with a specific version when you check in a project. ¾ -remote specifies that the air utility should start the EME server remotely even if a direct connection is possible.

May 18, 2010

Air Sandbox group The air sandbox commands perform operations on sandbox. •

air sandbox create – creates the sand box in specified path. syntax- air sandbox create path ¾ Path – specifies a directory. ¾ [ -template path ] – creates the sandbox from a specified template sandbox at path. ¾ [-prefix prefix] – Add prefix to the predefined sandbox parameters. ¾ [-replace s1 s2] – change one thing to another thing

May 18, 2010

Air Sandbox group •

air sandbox diff –displays difference between two graphs or two files syntax- air sandbox diff {path | path1 path 2} ¾ [Path] – specifies a directory in which file is located . ¾ { -version version path | -version version path1 –version version path2} – specifies the version which has to be compared ¾ {-text} – forces text differencing , regardless types of files.

• air sandbox find – identifies root of sandbox

syntax- air sandbox find [path] ¾ [Path ]– specifies a directory in which to start looking . ¾ {-up|-down}- upward and downward search , towards and downwards of the directory specified. ¾ { -sandbox} – specifies sandbox found should be listed ¾ {-project} – specifies EME project the sandbox is associated with should be printed. May 18, 2010

Air Sandbox group • air sandbox info –displays info about the files associated with sandbox

syntax- air sandbox info {path} ¾ [Path] – specifies a directory. • air sandbox lock – performs lock operation on sandbox objects

syntax- air sandbox lock path ¾ [Path ]–the path to file checked in sandbox ¾ [-set|-release|-break|-release].- set locks in a file, releases lock , breaks another user lock , resets the lock broken by another user •air sandbox parameter–creates sandbox parameter syntax- air sandbox parameter [pset] parameter [interpretation] [value] [flag] ¾ [Pset]–specifies the location of parameter ¾ [-parameter].- name of the parameter ¾ [interpretation] – interpretation used for value. May 18, 2010

Air Sandbox group •

air sandbox pset – creates or modifies input value for graph. syntax- air sandbox pset pset-path ¾ [pset-path] –path to input values. ¾ [-create] – creates an input value set.

• air sandbox revert – reverts sandbox object to versions in data store.

syntax- air sandbox revert –basedir sandbox ¾ [-basedir]– path to sandbox ¾ [-files path ].- sandbox relative paths of object to be reverted •air sandbox run – runs a graph or file. syntax- air sandbox run path ¾ [Path ]– path to a graph (.mp ) file, graph task or plan or .pset file

May 18, 2010

Air Project group This manages the datastore projects. air project create – creates a new project and specifies various project attributes.

•

syntax- air project create project name ¾ Project-name – name of the project ¾ [ -location location ] – name of the project parameter that specifies the location of the project. ¾ [-prefix prefix] – prefix for default project parameters created automatically.

May 18, 2010

Air Project group • air project export –check out or copies projects from datastore into a sandbox.

syntax- air project export project-path –basedir sandbox-root-dir ¾ Projectpath –project path and name in datastore ¾ [ -basedir ] – specifies an existing or new sandbox • air project import – check in or copies sandbox files into an existing datastore project.

syntax- air project import project-path –basedir sandbox-root-dir ¾ Projectpath – path to the project to which you want to check in files ¾ [ -basedir ] –path to sandbox

May 18, 2010

Air Project group •

air project mkdir – creates a directory in specified project. syntax- air project mkdir [-q] project-name [-no-export] dir-name ¾ -q – supress error if directory already exists ¾ [-no-export] – the directory is not exportable

• air project add – Manually adds one or more files to a datastore project’s internal list of files.

syntax- air project add [-q] project-name relative path ¾ -q – supress error if relative path already exists ¾ Relative path – relative path of file to be added

•air project modify – edits the specified project location parameter. syntax- air project modify project-name [-location loc-parameter] ¾ Project-name – name of the project ¾ [-location loc-parameter] – Name of the project parameter

May 18, 2010

Air Project group • air project move –Moves a project from one place to another within datastore.

syntax- air project move old-name new-name ¾ Old-name – datastore path specifying the original location ¾ New-name – the path which project to be moved • air project parameter – deals with project parameter.

syntax- air project parameter proj-path [-export] parameter [interpretation] value ¾ Proj-path – path to project ¾ ¾ ¾ ¾

-Export – exported to environment Parameter – the parameter name you want to add or edit Interpretation – determines how value is interpreted Value – the new value of the parameter .

May 18, 2010

Air Project group •

air project revert – reverts one or more objects in a Project syntax- air project revert proj-path –version ver-path relative-path ¾ Proj-path –path to the project containing objects you want to revert ¾ -version ver-path – version number ¾ Relative-path – relative path of the object to be reverted.

• air project set-executable –ensures that a file will be marked executable when it is exported.

syntax- air project set-executable proj-path relative-rpath[-not] ¾ Proj-path – path to project containing the file ¾ relative-rpath – relative path of file within the project ¾ -not – specifies than an executable file be marked not executable.

May 18, 2010

Air Project group • air project set-type – overrides the mime type assigned to a single datastore project

syntax- air project set-type proj-path -force relative-path mime-type ¾ Proj-path –path to the project in which object resides ¾ -force – by default you cannot set . You have to force. ¾ Relative-path – relative path to the object whose MIME type you want to change. • air project show – displays project attributes.

syntax- air project show proj-path ¾ Proj-path – path to project

May 18, 2010

Air Object group It manipulates object and directories in the datastore and control permission to access objects.

•air object access– To test whether all permissions exists for the project.

syntax- air object access rpath [r|w|x|f] ¾ Rpath – path to a data store project ¾ -r – test whether you can read the specified object. ¾ -w - test whether you can write the specified object ¾ -x - test whether you can execute the specified object ¾ -f – test whether you can see the existence of the specified project. • air object show – displays project attributes.

syntax- air project show proj-path ¾ Proj-path – path to project

May 18, 2010

Air Object group air object chown – Changes the ownership of one or more datastore objects. Only the EME administrator can perform this operation.

•

syntax - air object chown[-Rf]user[.group] rpath...... ¾ -R

¾ -f

-

Optional. Recurses into rpaths if they refer to a directory. Optional. Attempts to continue even if the operation fails

for one or more rpaths. ¾ user

- New owner expressed as a valid user name.

¾ .group - Optional. Group designation, preceded by a period. ¾ rpath - Path to a datastore object, or object ID, whose ownership you want to change. You can specify multiple paths.

May 18, 2010

Air Object group •

air object cp – Creates a copy of the object known as rpath1 in a

location within the datastore. The command copies rpath1, retaining links to any other objects in references. syntax - air object cp rpath1 [-R] rpath2 ¾ rpath1

Path to a datastore object, or object ID, to be copied.

¾ -R Optional. If specified, command copies the specified object rpath1 and all objects referred to by rpath1. If not specified, the specified object rpath1 is copied, but objects referred to by rpath1 will be linked but not copied. ¾ rpath2

New datastore path for the copied object.

May 18, 2010

Air Object group •

air object create-parameter-set – Creates a parameter set file

from the parameters associated with a datastore object. syntax- air object create-parameter-set file graph_path ¾ file

-

Name of the parameter set file. It should have an extension of .pset ¾ graph_path - Path to the graph in the datastore. ¾ switch The name of a switch parameter. ¾ binding A value specified for switch.

May 18, 2010

Air Object group •

air object delete – Marks one or more objects, and other objects in

the same domains, for permanent removal from the datastore. syntax - air object delete –R –whole–domain [-force] rpath [ rpath2…] ¾ -R

- If rpath is a directory, all of its directory child and grandchildren and all objects in the descendants’ domains will be marked for deletion. ¾ -whole-domain - All objects in the same domain as the object specified by rpath will be similarly removed. ¾ -force Do not prompt for confirmation of deletion. ¾ rpath rpath2… Datastore paths to the objects you want to delete.

May 18, 2010

Air Object group •

air object In - Creates an additional name , rpath2, for the object

rpath1. syntax - air object In rpath1 rpath2 ¾ rpath1 ¾ rpath2 •

Path to a datastore object, or an object ID. New name for the existing object.

air object mv - Renames the object rpath1 to rpath2. syntax - air object mv rpath1 rpath2 ¾ rpath1 ¾ rpath2

Path to the datastore object you want to rename. New name for the object rpath1.

May 18, 2010

Air Object group •

air object load - Loads objects from a file in portable

interchange format. This file was created by running the air object save command. The objects are loaded to the datastore location corresponding to the ones in which they were saved. syntax - air object load [-table-of-contents] file-name ¾ file-name - Standard UNIX file-name of the file storing the objects you want to load. If file-name is -, then the data is read from standard input. ¾ -table-of-contents - Specifies that nothing be loaded, but instead print a listing of the file’s contents.

May 18, 2010

Air Object group •

air object revert - Reverts one or more objects to an earlier

version. Objects newer than specified version are deleted from the datastore. syntax - air object revert [-revert-links] version rpath……

¾ -revert-links - Optional; if specified, then objects linked to rpath will also be reverted. If not specified, no action is taken on linked objects. ¾ version Version number (integer) or tag (string) of the datastore that you wish to back out to. ¾ rpath…

-

Path to an object, or object ID.

May 18, 2010

Air Object group •

air object mkdir - Creates a datastore directory. Parent directories

are created as necessary. syntax - air object mkdir rpath ¾ rpath

•air object

Path of the directory you want to create. unlink - Unlinks a directory pathname of an object.

syntax - air object unlink[-f] [-no-project-remove] rpath ¾ -f

- Specifies that no error be generated if rpath does not exist. ¾ -no-project-remove - Specifies that the object be removed without removing it from source code control. ¾ rpath - Datastore pathname of the directory.

May 18, 2010

Air Object group •

air object rm - Removes an entry from a directory. If the entry is in a

datastore project, then the system performs an implicit air project remove. syntax - air object rm[-f][-r][-no-project-remove] rpath…..

¾ -no-project-remove - Specifies the object be removed without removing it from source code control. ¾ -f - Specifies that no error be generated if rpath does not exist. ¾ -r - Specifies that subdirectories of rpath be removed recursively. ¾ rpath - Path to the directory entry you want to remove.

May 18, 2010

Air Object group •

air object save

- Saves datastore objects to a file in portable

interchange format. You can subsequently load the file by running air object load. You can use air object save to migrate objects from one datastore to another. syntax - syntax - air object save file-name rpath… [-include{common| local| rpath…}]… [-external { common| local| erpath…}]… [-format {1| 2| 3}] [-export epath…] [-settings common| project…] [-save-jobs] [-analyzed] [-no-default] [-tag tag] [-comment comment] [-Stag tag] [Scomment comment] [

May 18, 2010

Air Object group •

air object versions - Displays the revision history of all tagged

versions or all versions of the specified object or the datastore. History includes version number, date, user, tag, and comment for the specified object. syntax - air object versions[-verbose][rpath] ¾ -verbose - Optional. Displays all versions for an object or the entire datastore. (If nor specified, only tagged versions will be displayed.) ¾ rpath

- Optional. Path to the object about which you want version information. If not specified, displays information for the datastore as a whole.

May 18, 2010

Air Lock group It manages locks to prevent file collision. • air lock set – sets or modifies a lock on datastore objects or projects.

syntax- air lock set [-force] [-breakable | -unbreakable] [-autorelease | manual-release] [-modify attribute] [-object object | -project project] ¾ Force –breaks another lock and sets a lock for you ¾ [ - breakable | -unbreakable] – specifies whether other user can break the lock ¾ [auto-release | manual-release] – the lock is automatically has to be released automatically or not after check in ¾ [-modify attribute] – modifies the breakability and release attributes of an existing lock ¾ -object – object is the path which you want to set lock ¾ -Project – Project is the path which you want to set lock

May 18, 2010

Air Lock group air lock release – sets or modifies a lock on datastore objects or projects. syntax- air lock release [-object object] [-project project] ¾ -object – object is the path which you want to set unlock ¾ -Project – Project is the path which you want to set unlock air lock break – break users lock on a given datastore object. syntax- air lock release [-object object] [-project project] ¾ -object – object is the path which you want to break ¾ -Project – Project is the path which you want to break air lock show –displays a listing of locks for a given user , object or project.

syntax- air lock show [-user username] [-object object] [-project project] ¾ -object – object is the path which you want to display ¾ -Project – Project is the path which you want to display May 18, 2010

Air Repository group The air repository commands operate on the storage system of the datastore

•air repository create - creates a datastore and installs certain default system objects . •air repository chmod –change permissions on datastore. •air repository chown –change ownership of the datastore. •air repository rename – renames all files that makes up an EME data store. •air repository restart –shuts down and restarts the EME process. •air repository start – starts the EME process. •air repository shutdown –Shuts down the EME process. •air repository backup –it is used to backup of datastore

May 18, 2010

Air Branch group The air repository commands operate on the storage system of the datastore

•air branch create - creates a new branch with the name new-branch from an existing branch . •air branch delete – deletes object on the branch. •air branch freeze –freezes or unfreezes a branch . •air branch list – displays information about objects on the branch. •air branch modify – allows you change the title of the branch .

May 18, 2010

QUESTIONS ?

May 18, 2010

THANK YOU

May 18, 2010

66