39189489 Beyond the Basics 4

Beyond the Basics

Page 1 of 76

This section contains topics that go beyond the basic information needed to build Ab Initio graphs, grouped as follows: Working in Sandboxes Using Parameters Using Phases and Checkpoints Using Subgraphs and Macros Creating Custom Components Using Conditional Components Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

This section contains the following topics: What is a Sandbox? Using a Sandbox Using Sandbox Parameters Sandbox Parameters and Graph Parameters Adding Subdirectories to a Sandbox Sandboxes and the Ab Initio Environment Sandboxes and the EME Sandboxes without the EME Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

A "sandbox" is a special directory (folder) containing a certain minimum number of specific subdirectories for holding Ab Initio graphs and related files. These subdirectories have standard names that indicate their function. The sandbox directory itself can have any name; its sandbox properties are recorded in various special and hidden files that lie at its top level (the my_project directory, in the following illustration):

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 2 of 76

For information on how to create a sandbox, see Create Sandbox Dialog. See also Creating a Sandbox. If you are using the EME, you can create a sandbox during checkout of a project. See Checking Out Projects, Graphs, and Files. If you are creating sandboxes in an Ab Initio Environment, the sandboxes will havd additional subdirectories. See Sandboxes and the Ab Initio Environment.

A sandbox is not limited to containing only the default subdirectories shown here. You can add any number of additional ones. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

Sandbox subdirectories are used to organize and store Ab Initio graphs and the files that are associated with them. Graphs and their associated files are all considered to be metadata (as opposed to the data which the graphs operate on), and sandboxes normally hold only metadata. Consider the following simple graph:

Viewed as a collection of various kinds of files, this graph consists of the following: Data files — the Transactions input dataset and the Transaction Data output dataset. If these are multifiles, the actual datasets will occupy multiple files. Record formats — there are two separate record formats in the graph, one for the input file, the other for the output. These record formats could be embedded in the Input File and Output File components themselves, but here (for the sake of the example) we will assume that they are stored in separate record format (.dml) files.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 3 of 76

Transforms — the Reformat component performs various changes on the format of incoming records, as specified by the statements in its transform function. The transform function can be embedded in the component, but (as with the record formats) we will assume that it is stored in a transform (.xfr) file, for the sake of our example. Graph file — the graph itself is stored complete as an .mp file. Deployed script — if deployed as a script, the graph will also exist as a .ksh file, which has to be stored somewhere.

Without a sandbox, you have to decide how to organize all these pieces of your graphs, and you have to take care to maintain your organizational conventions in all your applications—not doing so will quickly lead to chaos. But with a sandbox, you automatically have a set of standard subdirectories into which everything in a graph can be put. Sandboxes also help make your applications portable. Graphs reference their location- and platform-dependent information through sandbox parameters, rather than having it separately hard-coded in the separate components. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

Here is how a simple graph and its pieces would be stored in a sandbox:

Every piece of the graph has a specific subdirectory in which to go. Metadata such as record format files (stored in the dml directory) or transforms (stored in the xfr directory) can easily be shared among graphs, as they are all located in the same relative place within the sandbox. If the graph employed database components, the db directory would be used for the database configuration file or files.

! Notice that data files are not usually stored in sandboxes. Sandboxes are for organizing and storing metadata, not the data itself. Typically, you create sandbox parameters to refer to any data files or directories used by your applications, and you use these parameters in the graph components to refer to the data. See Creating Additional Sandbox Parameters.

To place yourself "in" a sandbox, you must open (in the GDE) a graph that is in that sandbox's mp directory, or save the graph you are working on in the mp directory of the sandbox you wish to work in. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 4 of 76

" Graphs refer to the sandbox subdirectories by using sandbox parameters.

" Sandbox parameters are variables which are visible to any component in any graph which is stored in that sandbox. Here are some examples of sandbox parameters: $PROJECT_DIR $DML $RUN For the difference between sandbox parameters and graph parameters, see Sandbox Parameters and Graph Parameters.

# " For example, within an Input File component, you might specify $DML/transactions.dml to reference a record format file for the transactions dataset. Here is what this would look like as specified in the Ports tab of the Input File properties dialog:

$DML is one of the default parameters present in all sandboxes, and its value is always the full path of the dml subdirectory in that sandbox. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

" The default sandbox parameters in a GDE-created sandbox are these six: PROJECT_DIR — absolute path to the sandbox directory DML — relative sandbox path to the dml subdirectory XFR — relative sandbox path to the xfr subdirectory RUN — relative sandbox path to the run subdirectory DB — relative sandbox path to the db subdirectory MP — relative sandbox path to the mp subdirectory These six parameters are automatically created (and assigned their correct value) whenever you create a sandbox.

" However, you can create additional sandbox parameters yourself, and you can use them to store values other than the paths of sandbox subdirectories. In fact, you can use sandbox parameters to store any value you wish to make available, for whatever reason, to graphs in that sandbox.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 5 of 76

For information on editing sandbox parameters and creating new ones, see Sandbox Parameters Editor. Although data files are not usually stored in sandboxes, there is no reason why you cannot create sandbox parameters to point to the data locations outside the sandbox. For example, you might have a transactions data file located at /my_data_path/transactions.dat You could define a sandbox parameter called DATA_PATH, set it to the value my_data_path, and reference the dataset like this:

To test the validity of a new sandbox parameter that specifies a path, simply type the parameter's name in a component's URL box (for example, as shown in the illustration above) and click Browse. If the parameter is successfully resolved, the Select Data File dialog will appear, displaying the contents of the directory referenced by the parameter's value. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

The difference between sandbox parameters and graph parameters is: Graph parameters are visible only to the particular graph to which they belong Sandbox parameters are visible to all the graphs stored in a particular sandbox

In the figure above, the arrows show how the values of the sandbox parameters $DB, $DML, and so on, are propagated to the individual graphs in the sandbox. The arrows have been omitted from the second graph for the sake of clarity.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 6 of 76

The individual graphs also can have their own parameters, some of which are shown, as examples, in the figure: $graph1_parm1 and $graph1_parm2 in the first graph, and $graph2_parm and $graph2_parm_2 in the second graph. These parameters are completely separate from and unaffected by the sandbox parameters (although you can use sandbox parameter values in the definitions of graph parameters). Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

At runtime, the graph parameters are set after the sandbox parameters. This means that, if the graph has a parameter with the same name as a sandbox parameter, the graph parameter value will override the same-named sandbox parameter value.

In the diagram above, Graph 1 has a parameter called PARM_X, and there is also a sandbox parameter that happens to have the same name. As a result, Graph 1 will "see" the value of the graph parameter called PARM_X, not the value of the sandbox parameter PARM_X. Graph 2, on the other hand, will see the sandbox parameter PARM_X. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

$

The following diagram shows what happens when a sandbox parameter employs in its definition another sandbox parameter, which happens to have the same name as a graph parameter. The sandbox parameter's value is overwritten by the same-named graph parameter, as already explained. But now, suppose this sandbox parameter is used by some other sandbox parameter in its definition. In that definition, the sandbox value of the parameter will be used, and it is this defined value of the parameter that will be seen by a graph. The following diagram illustrates this situation.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 7 of 76

For further information about graph parameters, see Graph Parameters and the Graph Parameters Editor. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

Adding a subdirectory to a sandbox consists of two steps: 1. Creating the subdirectory 2. Creating a sandbox parameter to reference the new subdirctory Creating the directory should be a straightforward operation. For information on creating sandbox parameters, see the Sandbox Parameters Editor. See also Creating a Sandbox and Testing the New Sandbox Parameters.

%

&

If your sandbox is attached to a project, creating a new subdirectory in the sandbox will probably mean that you want to add the subdirectory (and its contents) to the project. To do this, simply create the subdirectory and check the project in. The Checkin Wizard will detect the presence of the new subdirectory and prompt you to specify whether you wish it to be checked in or not. If you do not want to add the subdiectory to the project, you can specify that at this point.

%

%

#'

"

In an Ab Initio Environment, you can edit the project templates so that any projects subsequently created will have the additional subdirectories by default. For further information, see The Ab Initio Environment. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 8 of 76

%

#'

"

If you create your sandboxes using the Ab Initio Environment, they will by default have two subdirectories additional to the five you get in an "ordinary" sandbox. Here's what the default form of an Ab Initio Environment sandbox looks like:

The two additional subdirectories are: bin — for executable files Usually this directory is used only in public projects, where it is used to hold macro scripts and third-party executables for Run Program components. A "public project" is an Ab Initio Environment concept: it is a common project, intended to be accessible to other projects. Public projects are the Ab Initio environment's mechanism for sharing data among projects. sql — for SQL files Used to hold .sql files used by database components, in particular by the Run SQL component. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

#(# You can have sandboxes with projects, or sandboxes without projects.

A sandbox that is not associated with a project is simply a special directory. See What is a Sandbox? for more information.

)

&

A sandbox that is associated with a project is by definition under source control in an Enterprise Meta>Environment datastore. The project is the "master copy" of the sandbox contents; you check out copies of files in the project to work on in your sandbox, and check in the altered files to update the project copies under source control. The directory structure of the project in the datastore is exactly the same as the sandbox directory structure in your filesystem. The sandbox now contains only some set of particular versions (usually the latest) of the files that make up its contents. The project contains all the versions of the files.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 9 of 76

& A project and a sandbox are thus essentially the same thing, seen from different points of view. A sandbox is a copy of a project. There can be (and usually are) many such copies. Multiple users can set up sandboxes based on the same project, thus creating multiple copies of it. Each sandbox can be associated with only one project, but a project can be (and usually is) associated with multiple sandboxes. Any user working on the project will have his or her own sandbox in which to work. When you check a sandbox in to the EME, the contents of the sandbox are copied into the EME datastore project area. When you check the project out, the filesystem copies of the project contents are updated in your sandbox. To turn a non-EME sandbox into a project in the EME, all you have to do is check the project into the EME; the project will be created as you do so. See Creating a Project. See also Sandbox and Project Parameters. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

)

#(#

A standalone (non-EME) sandbox is mainly an organizational structure designed to hold Ab Initio applications. Its contents are metadata: graphs themselves (.mp files) and deployed graphs (.ksh scripts), along with files containing other metadata used by graphs: database configuration, record format, and transform files. See What is a Sandbox? for more information on this. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

%

#'

"

The Ab Initio Environment is a collection of functionality that allows you to set up a framework in which to organize your Ab Initio development, testing, and production work. It uses EME projects, common projects, and common data areas to build an open-ended hierarchy that can be as simple or complex as you require. Note that the term "Ab Initio Environment" can be a little ambiguous in certain contexts. It can be used to refer to an instance of an organizational framework that you have created, or to the set of Ab Initio executables that you use to create it. In these paragraphs, "Ab Initio Environment" signifies the thing(s) you create with the software: the collection of projects and sandboxes, the common data areas that are associated with them, and so on.

( The main characteristics of an Ab Initio Environment are the following: Every sandbox (and project) in an Ab Initio Environment has by default an expanded set of subdirectories In an Ab Initio Environment, you must create new projects either by running the install-environment script (when you create the new environment), or by running the create-project script. Every Ab Initio Environment is by default set up with ready-to-use data areas for serial and multifile data, as well as logs, error reporting, and the like Each of these areas is common to all projects in that environment, although each project has its own separate subsection of it; these can be further subdivided into user-specific sub-areas. You create an Ab Initio Environment by running the install-environment script. Every Ab Initio Environment has a single central common project (called the "environment project") whose parameter definitions are included in all other projects in that environment; these values allow the individual projects to access the data areas and any other values common to the environment The environment project is created automatically by install-environment.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 10 of 76

Including a given environment project is what puts a given project "in" a specific Ab Initio Environment. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

" This section contains the following topics: Generic Parameter Information Graph Parameters Graph Parameters Editor Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

%

"

A parameter specifies some aspect of graph, subgraph, or component behavior. For example, the Filter by Expression component requires an expression to choose which records are written to the out port. This expression is specified in the select_expr parameter. A parameter is a name-value pair, with a number of additional associated attributes that describe when and how to interpret or resolve the value. Parameters are used to change the behavior of graphs and sandboxes in a controlled and uniform way. You can assign values to parameters on the Parameters tab of the Properties dialog or in the parameters editors. Required parameters that do not have values assigned are indicated by yellow boxes

.

Interpretation Graph Parameters Graph Parameters Editor Adding a Parameter in a Parameters Editor Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

" A graph is completely defined by the totality of its parameter values. A parameter is a name-value pair, with a number of additional associated attributes that describe when and how to interpret or resolve the value. Parameters are used to change the behavior of graphs and projects (groups of graphs in a sandbox) in a controlled and uniform way. Graph parameters are associated with individual graphs and are private to them. They affect the execution only of the graph for which they are defined. All the specifiable values of a graph, including each component's parameters (as well as other values such as URLs, file protections, record formats, and so on) comprise that graph's parameters. Graph parameters are considered to be part of the graph with which they are associated. Usually you use the Graph Parameters Editor when you want to parameterize a graph—that is, make it reusable either as a deployed script or as a graph run in the GDE—with various runtime settings and values specified on the command line or in the GDE when it is invoked. For much of your graph development you may never need to use the graph parameters editor. For example, you might wish to define a parameter called RUN_DATE for many of your graphs. To do this, you would use

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 11 of 76

the graph parameters editor.

Formal Graph Parameter Adding a Formal Input Parameter to a Graph Exporting Component Parameters to the Top Level of a Graph Order of Evaluation of Parameters Adding a Parameter in a Parameters Editor Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

+ "

"

A formal graph parameter is a parameter you substitute for a path and/or filename when you create a graph. This allows you to specify the value of that parameter at runtime. For example, you might use a formal graph parameter to indicate a working directory. You can then specify one working directory when you run the graph while it is still in development or testing, and another directory when you run the graph in production. You can also use a formal graph parameter to defer until runtime the location and/or name of a dataset. This lets you run the same graph using different datasets that share the same record format.

#, ( -# Suppose that you enter $DAILY_TRANS as the path to a dataset. You can give DAILY_TRANS a value in several ways: If the graph is deployed as a script, the production person who runs the script can first use the Korn shell to enter a command like: export DAILY_TRANS=absolute_path where absolute_path is the absolute path (including filename) to the dataset. You can edit the host setup script to give DAILY_TRANS a value using something like the above command. If you are using the EME, you can include a common project whose project parameters include DAILY_TRANS. The sandbox must be located on the runtime run host, the project must be checked out at runtime, and its parameters must have the Export option selected. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

+ " %

"

Suppose you have created a graph that performs some task, and you want to use a different path to an input file dataset each time you run the graph. The best way to accomplish this is to define a formal graph input parameter of type string for the graph, and then reference that graph parameter from the Layout parameter of the Input File component in the graph, using $ substitution. To define the formal input parameter for the graph: 1. On the menu bar of the GDE, choose Edit > Parameters to open the Graph Parameters Editor. 2. In the Editor's grid, click in the row underneath the row containing the name of the graph. This row will typically contain the name of one of the components in the graph. 3. On the menu bar of the Editor, choose Edit > Insert Parameter. A new row appears with a local parameter icon and noname next to it in the Name column.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 12 of 76

4. Replace noname with a name for the parameter. For the purposes of this example, use input_file_url. 5. Right click anywhere in the row, and choose Details from the shortcut menu to open the Parameter Details Window. 6. In the Scope row of the Parameter Details window, click the arrow and choose Formal. Although the default for Scope is Local, you need to use Formal for a parameter that accepts user input. 7. In the Kind row, either leave the default value, or select one of the other choices, depending on the way in which you want the parameter to receive its value. For the purposes of this example, leave Kind set to the default, positional. 8. In the Type row, leave the type set to its default, string. 9. In the Parameter Details window, click OK. 10. In the Value column of the Graph Parameters Editor, change the value for the Input File component's Layout parameter to $input_file_url. 11. Click OK. If you run this graph from the GDE, the GDE will prompt you to supply a value for the input_file_url parameter. $ substitution will substitute this value for $input_file_url in the Layout parameter of the Input File component. If you run the deployed script for the graph, you must pass the value for input_file_url as the first argument on the command line used to invoke the script. $ substitution then proceeds in the same way it does when you run the graph from the GDE. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

#

"

"

* -'

You can export any number of component parameters to the top level of a graph or subgraph, thus making them Formal graph parameters. When you run the graph in the GDE, you will be prompted for values for the parameters. If you run the graph as a deployed script, you will have to specify values for the parameters on the command line (if the parameters' Kind has been declared as Keyword or Positional), or supply the values in properly-set environment variables that have the same names as the parameters (if the parameters' Kind has been declared as Environment Variable). This technique allows you easily to partially parameterize a graph for testing or any other purpose. To remove the parameterization, you simply delete the top-level parameters in the graph parameters editor. To export a component parameter to the top level of a graph or subgraph, do the following: 1. Open the parameters for the component whose parameters you wish to export. You can do this in either of two ways: (in the GDE graph display) Open the Properties dialog for the component and click the Parameters tab. (from the graph parameters editor) Right-click on the row of the parameter you wish to export, and select the Goto Parameter parameter name item. This will close the graph parameters editor, open the component's Properties dialog, and put you in its Parameters page. 2. Select the parameter you wish to export. For example, let's assume you want to export the key parameter of a Join component in a graph. In the Join component's Parameter Properties page, you select the key parameter:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 13 of 76

Notice that, in the lower portion of the dialog, the parameter's Kind is shown in a grayed-out text box. As long as the parameter's Kind is not Implicit (which means that the parameter's value is calculated internally by the GDE), it can be exported. 3. Click the Export button, which is located just above the Kind box. The Export as Parameter dialog appears.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 14 of 76

4. Fill in the fields with the appropriate values. Make sure the parameter's Kind is Formal. When you are finished, click OK. You will be returned to the component's Properties dialog. Click OK. At this point, if you re-open the graph parameters editor, the parameter is displayed at the top tree level, if it is a top-level parameter. If the parameter is in a component in a nested subgraph, it is exported up one level, not to the top level of the graph.

Now, when you run the graph, you will be prompted to supply a value for the Join component's key parameter:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 15 of 76

To un-export the parameter, simply delete it in the graph parameters editor. to mean command substitution. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

.

#'

"

When you run a graph, parameters are evaluated in the following order: 1. The host setup script is run. 2. Common (that is, included) sandbox parameters are evaluated. 3. Sandbox parameters are evaluated. 4. The project-start.ksh script is run. 5. Formal parameters are evaluated. 6. Graph parameters are evaluated. 7. The graph Start Script is run. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

#

To display the parameters editor for a graph, open the graph in the GDE, and then select Edit > Parameters from the GDE menu bar. Use the Graph Parameters Editor to: Add parameters to a graph by specifying information about them in the Graph Parameters Editor Columns. Edit parameters in a graph, including the parameters of any subgraph or component in the graph, by changing the information in the Editor's columns. Use the Graph Parameters Editor Menus and Graph Parameters Editor Toolbar to perform related tasks. The graph parameters editor serves the same purpose as the individual components' Properties dialogs. It gives you comprehensive access to all the parameters in a graph (and its subgraphs) at once, rather than forcing you to go back and forth among the separate components to edit their parameters and other values. In short, every graph value is represented in the graph parameters editor, regardless of whether it is called a "parameter" or not in a component (see Name). Anything that you can do in a component's Properties dialog — set parameters, edit DML,

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 16 of 76

edit transforms, and so on — you can also do in the graph parameters editor.

#

"

Each row in the editor represents a parameter. Each column in the grid represents a different aspect of a parameter. You click on a cell to edit it. To select an entire row, click in the shaded box to the left of the row. You edit parameters' Value cells by clicking the Edit Button.

"

!

To get a complete display of all the column values for a particular parameter, click in the parameter's Name cell, or click the button to the left of the parameter's row, then right-click and select Details from the floating menu.

Generic Parameter Information Graph Parameters Exporting Component Parameters to the Top Level of a Graph Order of Evaluation of Parameters Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

#

"

By default, the Graph Parameters Editor displays the Name, Value, and Description columns. In addition to these three, the following columns can appear in the grid of the Editor: Scope Kind Type Location Value Edit Button Interpretation Required Export to Environment To determine which columns appear in the grid of the Editor, choose View > Customize on the Editor's menu bar, select the check boxes for the columns you want to appear, and clear the check boxes of those you do not want to appear. To edit the values for the above columns without having the columns appear in the Editor, do one of the following: Choose View > Details on the Editor's menu bar. Right click in the row of the parameter you want to edit, and choose Details from the shortcut menu. The Parameter Details window opens, and you can perform your edits in it. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 17 of 76

"

#

(

The Graph Parameters Editor offers five menus.

+%-#(#/ The File menu of the Graph Parameters Editor contains the following items: Save (Ctrl + S) — Saves the current changes in the Graph Parameters Editor. Validate — Checks the selected parameter, or the parameters of the selected graph, subgraph, or component to make sure there are no errors in syntax, and that all $ references can be resolved. Print — Opens the Print dialog, so you can print the grid of the Graph Parameters Editor. Close (ESC) — Closes the Graph Parameters Editor.

#!%*(#/ The Edit menu of the Graph Parameters Editor contains the following items: Cut (Ctrl + X) — Removes the selection and places it on the clipboard. Copy (Ctrl + C) — Copies the selection and places it on the clipboard. Paste (Ctrl + V) — Inserts the contents of the clipboard at the insertion point. Insert Parameter — Adds a new parameter above the selected row in the center grid. Delete Parameter — Removes the selected parameter. Find (Ctrl + F) — Searches for the string specified in the Find What box of the Find dialog. In the Search list, select one of the following: By Columns to search the current column and then continue with the next column By Rows to search the current row and then continue with the next row Replace (Ctrl + H) — Replaces the string specified in the Find What box with the string specified in the Replace With box of the Replace dialog. In the Search list, select one of the following: By Columns to search the current column and then continue with the next column By Rows to search the current row and then continue with the next row Find Next (F3) — Searches for the next occurrence of the string specified in the Find dialog. Script Start — Opens a text editor that lets you write shell commands that run before the execution of the graph to manage tasks such as moving or creating files. Script End — Opens a text editor that lets you write shell commands that run after the execution of the graph to manage tasks such as renaming output files.

0%# (#/ The View menu of the Graph Parameters Editor contains the following items: Customize — Opens the Customize Columns dialog so you can select which columns you want the Editor to

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 18 of 76

display. Details — Opens the Parameters Details dialog. Use to view or edit all columns regardless of which ones are visible in the Editor. Reset Columns — Resets the columns displayed by the Editor to the default selection: Name, Value, and Description. Expand Sub-record — Opens the parameter sub-tree for the selected row. Collapse Sub-record — Closes the parameter sub-tree for the selected row. Expand All — Opens all levels of the parameter tree. Collapse All — Closes all levels of the parameter tree except the top level. Show Parameters — Displays the parameters of the graphs, subgraphs, and components in the active graph. To display the parameters of a graph, subgraph, or component: a. Select Show Parameters. b. Click the plus sign next to the graph, subgraph, or component. Alternatively, instead of step b you can select the desired row and then choose View > Expand Sub-Record.

+%-*#1 (#/ Use the Filter menu of the Graph Parameters Editor to select which components or parameters appear in the grid of the Editor, according to the criteria expressed by the following menu choices: Clear — Removes any filtering that is in effect. By Parameter Name — Displays only parameters with the specified name. By Parameter Type — Displays only parameters with the specified type. Select the type from the submenu. By Value — Displays only parameters with the specified value. Explicit — Removes from view any parameters with values propagated from other components. By Component Name — Displays only components with the specified name. By Component Type — Displays only components of the specified type. Select the type from the submenu. By Phase — Displays only components in the phase of the graph you specify. By Layout — Displays only components with the layout you specify.

2#- (#/ The Help menu of the Graph Parameters Editor contains the following items: Editor Help — Opens Help for the Graph Parameters Editor. Help Topics — Provides access to all Ab Initio Help topics. About Ab Initio — Displays information about the active version of the GDE. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

#

*

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 19 of 76

The Graph Parameters Editor displays a toolbar with navigation arrows and shortcuts for commonly used menu items. Validate checks the selected parameter, or the parameters of the selected graph, subgraph, or component, to make sure there are no errors in syntax, and that all $ references can be resolved. Print opens the Print dialog, so you can print the grid of the Parameters Editor. Cut removes the selection and places it on the clipboard. Copy copies the selection and places it on the clipboard. Paste inserts the contents of the clipboard at the insertion point. Insert Parameter adds a blank parameter above the selected component, subgraph, or parameter. Delete Parameter removes the selected parameter. Move Down moves the selected parameter downward through the grid. Move Up moves the selected parameter upward through the grid. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

#

To add a parameter in one of the parameters editors: 1. Open a Parameters Editor. 2. In the Editor's grid, click in the row above which you want to insert the new parameter. 3. Do one of the following: On the Editor's menu bar, choose Edit > Insert Parameter. On the Editor's toolbar, click the

button.

A new row appears with a local parameter icon and noname next to it in the Name column. 4. Replace noname with a name for the parameter. 5. Make entries in the rest of the Editor's columns to define the parameter you want. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

)

*

"+

This section contains these topics: About Transform Functions The Transform Editor Rules in Transform Functions Statements in Transform Functions

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 20 of 76

Local Variables in Transform Functions Comments in Transform Functions Specifying Where a Transform Is Stored Making Recursive Calls in Transforms The GDE Expression Editor The DML Functions Palette The DML Operators Palette Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

"+

A transform function (or transform) is the logic that drives data transformation — most commonly, transform functions express record reformatting logic. In general, however, you can use transform functions in data cleansing, record merging, and record aggregation. To be more specific, a transform function is a collection of business rules, local variables, and statements. The transform expresses the connections between the rules, variables, and statements, as well as the connections between these elements and the input and output fields. Transform functions are always associated with transform components; these are components that have a transform parameter: the Aggregate, Denormalize Sorted, Fuse, Join, Match Sorted, MultiReformat, Normalize, Reformat, Rollup, and Scan components. Each component that has a transform parameter: Determines the values that are passed to the transform function Interprets the results of the transform function Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

* *

"#

This section includes these topics: About Creating Transforms Displaying the Transform Editor in Grid View Transform Editor (Grid View) Displaying the Transform Editor in Text View Transform Editor (Text View) Using Text Alternatives to the Transform Editor Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

"

To create a transform, you create rules, variables, and statements and describe the relationships between them. The Transform Editor has two views: Grid and Text. In the Grid View, you create a rule by dragging and dropping. In the Text

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 21 of 76

View, you create a transform using DML. Most graph developers use a combination of these two techniques to create and modify transforms — switching back and forth between Grid and Text Views as needed. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

*

"#

0 )

The Grid View of the Transform Editor provides a graphical interface for building and editing transforms:

To display the Transform Editor Grid: 1. Double-click a graph component that has a transform parameter. This displays the Properties dialog. 2. On the Parameters tab, double-click the transform parameter (or select transform in the Parameters list and click New or Edit). The Transform Editor opens and a dialog appears displaying the question: Do you want the system to generate a default transform? 3. Click Yes or No. If you click No, you see a blank Transform Editor. If you click Yes, the system generates a default transform function that you can edit. 4. If the Transform Editor opens in Text View, select View > Grid. NOTE: You can also display the Transform Editor without going through the Properties dialog: Shift-double-click a component that has a transform parameter. You can also use the Transform Editor in Text View to create transforms using DML. To do so, select View > Text View and see Creating Transforms.

Transform Editor (Grid View) Rules in Transform Functions Statements in Transform Functions Local Variables in Transform Functions Comments in Transform Functions Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

"#

3

0 )4

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 22 of 76

Use the Transform Editor to work with transform functions by creating or modifying business rules, statements, and local variables. You can also set rule priority here.

You can also use the Transform Editor in Text View to create transforms using DML. To do so, select View > Text View and see Creating Transforms .

Creating Rules in the Transform Editor Grid Working with Rule Priority in the Transform Editor Grid Entering Statements in the Transform Editor Grid Declaring Local Variables in the Transform Editor Grid Initializing Local Variables in the Transform Editor Grid Deleting Local Variables in the Transform Editor Grid Adding Comments in the Transform Editor Grid Transform Editor (Grid View) Menus Transform Editor (Grid View) Toolbar Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

*

"#

*

0 )

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 23 of 76

The Text View of the Transform Editor is similar to other text editors:

To display the Transform Editor in Text View: 1. Double-click a graph component that has a transform parameter. This displays the Properties dialog. 2. On the Parameters tab, double-click the transform parameter (or select transform in the Parameters list and click New or Edit). The Transform Editor opens and a dialog appears displaying the question: Do you want the system to generate a default transform? 3. Click Yes or No. If you click No, you see a blank Transform Editor Grid. If you click Yes, the system generates a default transform function that you can edit. 4. If the Transform Editor opens in Grid View, choose View > Text View on the menu bar of the Editor. You can also use the Transform Editor Grid View to create transforms. To do so, select View > Grid View and see Transform Editor (Grid View).

Using Text Alternatives to the Transform Editor Rules in Transform Functions Statements in Transform Functions Local Variables in Transform Functions Comments in Transform Functions Transform Editor (Text View) Menus Transform Editor (Text View) Toolbar Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

"#

3*

0 )4

Use the Transform Editor in Text View to enter or edit transforms using DML.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 24 of 76

To view the Transform Editor in Text View, see Displaying the Transform Editor in Text View. For detailed information on working with transforms using DML, see Transform Syntax, Rule Syntax, and the topics listed in About Statements. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

'

*

"#

Instead of entering DML in Text View of the Transform Editor, you can do one of these: You can enter the DML into a standalone text file which you edit using the text editor of your choice. Then, you include the standalone text file in the transform's package. If the DML for the transform function is fairly short, you can enter the DML code into the Embedded box on the Parameters tab of the component Properties dialog. Most graph developers use a combination of these two techniques to create and modify transforms — switching back and forth between Grid and Text Views as needed. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

1

*

"+

This section includes: About Rules Creating Rules Prioritizing Rules Working with Null Values In Rules Generating Default Rules Working with DML Wildcards in Rules Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

1 A rule, or business rule, is an instruction in a transform function that directs the construction of one field in an output record.

Creating Rules Prioritizing Rules Working with Null Values In Rules Generating Default Rules Working with DML Wildcards in Rules Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 25 of 76

1 This section includes these topics: Creating Rules in Grid and Text View Creating Rules in the Transform Editor Grid Working with Rules Using DML Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

1

*

0 )

You create transform functions using one or both of these methods: You can use drag and drop techniques in Grid View of the Transform Editor.

NOTE: When you create transforms in the Grid View, the transforms are automatically translated into DML code. To view the generated code, select View > Text from the Transform Editor menu bar. For example, the transform shown above in Grid View looks like the illustration below when viewed in Text View. You can also use the Transform Editor in Text View to create transforms by entering DML code directly. To do so, select View > Text View and see Creating Transforms .

Using Text Alternatives to the Transform Editor Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

1

*

"#

To create a rule: 1. If it is not already displayed, display the Transform Editor Grid. 2. Click the Business Rules tab if it is not already displayed. 3. Drag an input field from the Inputs pane into a cell on the Business Rules tab. Arrows indicate the flow of data. 4. To use another input field in the rule, drag another input field into the cell. 5. To alter the data in the fields you have dragged into the cell, do one of the following: Type an expression incorporating the field names.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 26 of 76

Right-click the rule and choose Edit Rule from the shortcut menu to open the Expression Editor, and then click Help. 6. Drag from the Rule Port to an output field as follows:

NOTE: To quickly copy a field from the input to the output with no changes, drag from the Inputs pane to the Outputs pane. See also Generating Default Rules.

Working with DML Wildcards in Rules Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

)

1

!(-

To create a rule using DML: 1. If it is not already displayed, display the Transform Editor in Text View. 2. Enter statements using the syntax described in Creating Rules. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

5

1

This section contains these topics: About Prioritizing Rules Working with Rule Priority in the Transform Editor Grid Working with Rule Priority Using DML

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 27 of 76

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

5

1

You can set the order of evaluation of rules in a transform function by assigning priority numbers to the rules. The rules are attempted in order of priority, starting with the assignment of lowest-numbered priority and proceeding to assignments of higher-numbered priorities, then finally to an assignment for which no priority has been given. You set rule priority using one or both of these methods: Using the Transform Editor Grid as described in Working with Rule Priority in the Transform Editor Grid. Using DML code as described in Working with Rule Priority Using DML. Most graph developers use a combination of these two techniques to create and modify transforms — switching back and forth between Grid and Text Views as needed. The best way to view all your rule priorities at once is to view the Transform Editor in Text View.

Working with Rule Priority in the Transform Editor Grid Working with Rule Priority Using DML Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

)

1

*

"#

To set the priority of a rule in the Transform Editor Grid: 1. Do one of the following to display the Assignment Priority dialog: Select the rule and choose Edit > Priority. Right-click a rule and choose Set Priority from the shortcut menu. 2. Click Help in the dialog.

Working with Rule Priority Using DML Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

)

1

!(-

To set the priority of a rule: 1. If it is not already displayed, display the Transform Editor in Text View. 2. Enter the priority as described in About Prioritizing Rules, Creating Transforms, and Prioritized Wildcard Rules.

Working with Rule Priority in the Transform Editor Grid Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 28 of 76

)

/

0

%1

There is no graphical mechanism for working with NULLs in rules. Therefore, the best way to work with NULLs is by using DML code in Text View of the Transform Editor. For more information on NULLs in rules, see the following topics: Null Results from Input Expressions Making a Transform Succeed with Null Input Values Intentionally Using Nulls in Output Fields Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

1

This section contains these topics: Generating Default Rules in the Transform Editor Grid Add Default Rules Dialog Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

1

*

"#

If you have input fields that are not referenced explicitly in other rules, you can add default rules to have a transform assign the values of these input fields to output fields. To generate default rules in the Transform Editor Grid: 1. If it is not already displayed, display the Transform Editor Grid. 2. Click the Business Rules tab if it is not already displayed. 3. Select Edit > Add Default Rules. 4. Click Help. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

1

!

Use this dialog to generate default rules for input fields that are not referenced explicitly in other rules: the values of these input fields will be assigned to output fields that have the same name as the input field.

To generate one default rule for each unreferenced input field, select Match Names. For example, this is the result if you select Match Names for the input and output fields shown here:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 29 of 76

The resulting rules in DML code are: out.custid :: in.custid; out.title :: in.title; ... To generate one default rule for all unreferenced input fields, select Use Wildcard (.*) Rule. If the input and output fields are the same as those shown above, this is the result:

The resulting rule in DML code is: out.* :: in.*; Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

)

!(-

1

In addition to generating default (wildcard) rules in the Transform Editor Grid, you can also use wildcards when you create rules using DML code. For detailed information, and an explanation of how wildcards in DML-coded rules are expanded and propagated to subfields, see Using Wildcards in Rules. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

*

"+

This section includes these topics: Entering Statements in the Transform Editor Grid Creating Statements Using DML

The Assignment Statement

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 30 of 76

The If Statement The While Statement The For Statement The Block Statement The Switch Statement The Expression Statement Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

#

"

*

"#

Transform functions can contain several kinds of statements, although statements are not required. Statements are comprised of programming logic that affects the flow of control in a transform. Typical statements are the if statement and the while statement. You can use the Transform Editor Grid to enter statements, although the Grid does not provide drag and drop methods for building statements; you can only type DML statements directly into the grid. If you enter statements using the Transform Editor Grid, they will end up in the proper place when the transform function is automatically converted to pure DML. Alternatively, you can enter DML statements in Text View of the Transform Editor. See Creating Statements Using DML. NOTE: If you work with statements in Text View, you must be sure to place the statements after any local variable declarations and before the rules of the transform function. To create a statement in the current transform function: 1. If it is not already displayed, display the Transform Editor Grid. 2. Click the Statements tab. 3. Enter statements in the lines of the grid using the syntax described in Statements. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

!(-

To create a statement in the current transform function: 1. If it is not already displayed, display the Transform Editor in Text View. 2. Enter statements using the syntax described in Rule Syntax, Local Variable Declaration and Initialization Syntax, and the topics referenced in Statements. NOTE: Be sure to place the statements after any local variable declarations and before the rules of the transform function. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

-

0

*

"+

This section includes these topics: About Local Variables Declaring Local Variables

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 31 of 76

Initializing Local Variables Deleting Local Variables Editing Local Variables Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

-

0

You can often simplify or consolidate rules in a transform function by using a local variable. A local variable is a named storage location in an expression or transform function. You declare a local variable within the transform function in which you want to use it. The local variable is reinitialized each time the transform function is called, and it persists for one single evaluation of the transform function. You declare, initialize, and delete local variables inside transforms. You can do this using drag and drop techniques in the Transform Editor Grid: To declare local variables, use the Variables Editor. See Declaring Local Variables in the Transform Editor Grid To initialize local variables, use drag and drop techniques on the Variables tab:

Or, you can enter DML code in the Text View of the Transform Editor:

See also Using Text Alternatives to the Transform Editor. Most graph developers use a combination of these two techniques to create and modify transforms — switching back and forth between Grid and Text Views as needed. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

-

0

You can declare local variables using either the Transform Editor Grid or using DML code.

!# - 1%/ -. -0 1% -# %/ *2#*1 / +.1( #!%*.1 1%! To declare a local variable in the Transform Editor Grid: 1. If it is not already displayed, display the Transform Editor Grid. 2. Choose Edit > Local Variable from the menu bar of the Transform Editor to display the Variables Editor. 3. Click Help or see the Variables Editor topic.

!# - 1%/ -. -0 1% -#

%/ !(-

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 32 of 76

To declare a local variable using DML code: 1. If it is not already displayed, display the Transform Editor in Text View. 2. Enter local variable syntax as described in Creating Local Variables. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

%

5

-

0

You can initialize a local variable in the Transform Editor Grid or using DML code.

%/%*% -%6%/ -. -0 1% -# %/ *2#*1 / +.1( #!%*.1 1%! To initialize a local variable in the Transform Editor Grid: 1. If it is not already displayed, display the Transform Editor Grid. 2. Click the Variables tab. The rightmost pane becomes the Variables pane, displaying the Variables in the active transform function.

3. In a cell in the Variables tab, enter the expression that results in the desired value for the variable. 4. Drag from the Variable tab's port the value.

to the name of the variable (in the Outputs pane) to which you want to assign

NOTE: Local variable initialization is executed in order; therefore, later variables may depend on values in earlier variables.

%/%*% -%6%/ -. -0 1% -#

%/ !(-

To initialize a local variable using DML code: 1. If it is not already displayed, display the Transform Editor in Text View. 2. Enter the appropriate DML code as described in Local Variable Declaration and Initialization Syntax. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

-

0

You can delete a local variable in the Transform Editor Grid, or using DML code.

!#-#*%/ -. -0 1% -# %/ *2#*1 / +.1( #!%*.1 1%! To delete a local variable using the Transform Editor Grid: 1. If it is not already displayed, display the Transform Editor Grid. 2. Display the Variables Editor.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 33 of 76

3. Click the gray area at the beginning of the cell containing the variable you want to delete. 4. Press Delete. 5. Delete or edit any rules or statements that reference the deleted variable.

!#-#*%/ -. -0 1% -#

%/ !(-

To delete a local variable using DML: 1. If it is not already displayed, display the Transform Editor in Text View. 2. Display the Variables Editor. 3. Select View > Text. 4. Delete the DML code that declares the variable. 5. Delete or edit any rules or statements that references the deleted variable. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

#

-

0

This section includes these topics: About Editing Local Variables Displaying the Variables Editor Variables Editor Setting Attributes of Local Variables Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

#

-

0

You can edit local variables and set their attributes using the Transform Editor Grid or using DML code. To edit a local variable using the Transform Editor Grid: 1. If it is not already displayed, display the Variables Editor. 2. Select Help > Editor Help from the Variables Editor menu bar. To edit a local variable using DML code: 1. If it is not already displayed, display the Transform Editor in Text View. 2. Edit the variable as described in Creating Local Variables. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

0

#

To display the Variables Editor: 1. If it is not already displayed, display the Transform Editor Grid. 2. Do one of the following to display the Variables Editor: Choose Edit > Local Variable.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 34 of 76

Right-click a cell on the Variables tab, and choose Edit Local Variable from the pop-up menu. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

0

#

Use this dialog to create or edit local variables.

Field Name. Enter or edit the name of the local variable in this column. Field names must be unique within a variable. Data Type. Click the arrow to choose or change a data type. User-defined types are included in this list. Length. Enter or edit the length or delimiter for the field, as follows: For fixed-length fields, the length must be an integer representing the total number of bytes the field holds, for example 10. For variable-length fields, the delimiter must be enclosed in double-quotes, for example "end". For decimal fields, the length must be an integer representing the total number of bytes in the field, and optionally, a period and an integer representing the number of decimal point places the field stores. For more information on describing decimals, see Decimals. For dates, open the Attribute pane (as described in Setting Attributes of Local Variables), and set the Format attribute to a format string. For more information on date formats, see Details. The Status window across the bottom of the Editor displays messages about incorrect or incomplete entries. A yellow square at the beginning of a row indicates an incorrect or incomplete entry in that row. Placing the cursor in a cell in that row displays a message in the Status window if the entry in the cell is incorrect or incomplete. Click the

button to expand the status window into a text editor for reading long messages.

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

-

0

Use the Attributes pane of the Variables Editor to set the attributes of local variables. To display the Attributes pane: 1. If it is not already displayed, display the Transform Editor Grid.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 35 of 76

2. To display the Variables Editor, choose Edit > Local Variable, or right-click a Variables tab cell and choose Edit Local Variable from the pop-up. 3. In the Variables Editor, select View > Attributes from the menu bar. Use the following columns to set the selected variable's attributes: Attribute displays the name of the attribute Value provides a place to enter a value. Edit displays the attribute.

icon for certain attributes. Click this icon to open an appropriate editor for working with the

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

""

*

"+

You can add comments to a transform either in the Transform Editor Grid, or using DML in the Transform Editor Text View.

""

*

"#

You can add comments at the beginning of a transform, or at the end of a particular field. To add comments at the beginning of the current transform function: 1. Put the cursor in any field of the center pane of the Transform Editor. 2. Make sure the Documentation tab is selected at the bottom of the center pane. 3. Type your comment in the Documentation box. To add comments at the end of a particular field of the transform function: 1. Place the cursor in the field of the center pane where you want to add a comment. 2. Make sure the Details tab is selected at the bottom of the center pane. 3. Type your comment in the Details box.

""

!(-

You can add comments at the beginning of a transform, or at the end of a particular field. Add comments either between /* and */ patterns or between double slashes and the end of the line: /* comment in the style of C and Java */ // comment in the style of C++ and Java Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

"%

You can specify where a transform is stored. You do this on the Parameters tab of the component Properties dialog; you cannot do this using DML code. There are four options: Embedded

Specifies that the transform is embedded as part of the component. Using this option reduces the number of small files you need to save outside the graph. However, an embedded transform is not as easy to reuse or share as a transform saved as a file.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 36 of 76

Host

Specifies that the transform be saved as a file on the computer running the Co>Operating System.

Local

Specifies that the transform be saved as a file on the computer running the GDE.

EME Datastore

Specifies that the transform be saved as a datastore object at the specified path.

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

(

1

'

*

"

You can make recursive calls in transform functions only by using DML code. You can do this using the Transform Editor in Text View, or — if the transform is in a file — you can use any text editor to edit the file. For detailed information, see Making Recursive Calls. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

* + The Fields pane displays the input record formats available for use in an expression. To find the type of a field, right-click the field name. The type appears at the bottom of the shortcut menu. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

* + The Functions pane displays the built-in DML functions. To get information about a function, right-click the function name and choose Definition from the shortcut menu. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

* . The Operators pane displays the built-in DML operators. To get information about an operator, right-click the name of the operator and choose Definition from the pop-up menu. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

"+

!(-

To work with transforms using DML, see these topics: Creating Transforms About Transforms Transform Syntax Transform Syntax Details Creating Rules About Rules

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 37 of 76

Prioritized Rules Rule Syntax Rule Syntax Details Evaluation Results Examples of Rules Without Priorities Examples of Rules with Priorities Working with Null Values In Rules Using Wildcards in Rules Creating Local Variables About Local Variables Local Variable Declaration and Initialization Syntax Setting Whether a Local Variable is Nullable Statements About Statements The Assignment Statement The If Statement The While Statement The For Statement The Block Statement The Switch Statement Switch Statement Examples The Expression Statement Making Recursive Calls Adding Comments to Transforms Storing Transforms Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

) This section includes these topics: About Packages Viewing Packages Including Packages Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 38 of 76

This section contains these topics: What Is a Package? A Note About Include Files The Global Package Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

% Packages are an Ab Initio feature for use by GDE graph developers who need to collaborate and share transform functions, type definitions, and variables. A package is a named collection of related DML objects. A package can hold types, transform functions, and variables, as well as other packages. Packages provide a means of locating in one place DML objects that are needed more than once in a given graph, or needed by multiple developers. Packages allow developers to avoid redundant code; this makes maintenance of DML objects more efficient. Packages are very useful in these types of situations: The record formats of multiple ports use common record formats and/or type specifiers Multiple components use common transforms

7 # 1#

. % *#! %*2 .( ./#/* /! .1*

Because packages are always associated with components and ports, you can access packages only via the Properties dialog of a graph component. To display a packages in a tree view, you use the GDE's graphical editors (Package Editor, Transform Editor, and Record Format Editor). Packages can be stored in standalone files or can be embedded in a component. Package files have .xfr or .dml file extensions.

7 # /! ( -*%8 * # .( ./#/* The multi-stage transform components require packages because, unlike other transform components, they are driven by more than single transform functions. These components each take a package as a parameter and, in order to process data, look for particular variables, functions, and types in that package. For example, a multi-stage component might look for a type named temporary_type, a transform function named finalize, or a variable named count_items. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

/

%

+

The term include file is sometimes used to refer to a standalone file of DML code, even if that code does not include an actual "collection" of related DML objects, and may, in fact, include as little as one DML object. However, there is really no need to think of include files as technically different from package files. Therefore, for the sake of clarity, we use the term package file to refer to any DML in a file that is incorporated into a graph via an include statement. We use the term package to refer either a package file, or to an package embedded in a component. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

* file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 39 of 76

The global package is the package that contains all the built-in DML elements such as the built-in functions and data types. By default, every port and transform function in a graph is associated with this global package. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

0 ) When you work with packages, you can use drag and drop techniques in the tree view of the Package Editor, you can enter DML in text view of the Package Editor, or you can switch back and forth between the two techniques. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

0 )

*

0 )

You can use the tree view of the Package Editor to work with the DML objects in a package. To display the Package Editor in tree view, do one of the following: In the Properties dialog, select a transform parameter on the Parameters tab or a record format on the Ports tab, and click New or Edit. Double-click a component port on a component in the GDE. If the Record Format Editor opens, make sure it is in Grid Mode, and then choose View > Package from the Record Editor menu bar. Shift-double-click a component. If the Transform Editor opens, make sure it is in Grid Mode, and then choose View > Package on the Editor menu bar. This illustrations shows a package named utils as viewed in the Package Editor. It has three named types and a transform:

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

0 )

*

0 )

You can use DML code to create or edit the DML objects in a package. You can do this by viewing a package file in a text editor or by looking at the package in Text View of the Package Editor. To display the Package Editor in Text View: 1. Do one of the following:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 40 of 76

In the Properties dialog, select a transform parameter on the Parameters tab or a record format on the Ports tab, and click New or Edit. Double-click a component port on a component. If the Record Format Editor opens, make sure it is in Grid View, and then choose View > Package from the Record Editor menu bar. Shift-double-click a component. If the Transform Editor opens, make sure it is in Grid Mode, and then choose View > Package on the Editor menu bar. 2. Select View > Text View from the Package Editor menu. The following illustration shows a package named utils as viewed in Grid Mode of the Package Editor:

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

% To include a package: 1. Open the Package Editor for the component that will use the included package. 2. Select the Includes folder. 3. On the menu bar of the Package Editor, choose Edit > Insert. 4. Do one of the following: Click Browse, navigate to, and double-click the package you want to include Type the path to the package in the File name box and click Open. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

2

Page 41 of 76

+

This section includes these topics: Creating Helper Functions Modifying Helper Functions Reusing Helper Functions Deleting Helper Functions Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

2

*

"+

Transform functions get record definition information from their input or output ports. This allows the transform functions to be independent of data definitions and therefore reusable. In some cases, you may want to override this data type independence by creating a helper function. Create a helper function in any of these situations: When you need to operate on a part of a complex record format containing hundreds of fields When you have complex, repeatable transformations that you want to treat as a function The GDE associates helper functions with a package. You use the Package Editor to add, modify, or delete helper functions.

Creating Helper Functions Modifying Helper Functions Reusing Helper Functions Deleting Helper Functions Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

2

+

You create helper functions in two phases: Create the function, its arguments, and parameters. Associate data records with the new function. To create a helper function: 1. Select the transform to whose package you want to add the function. 2. Press Shift and double-click the transform to open the Transform Editor. 3. Select View > Package to open the Package Editor. 4. Select the Functions folder. 5. Select Edit > Insert to open the Function Definition dialog. 6. Create the function. For more information, click Help in the Function Definition dialog.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 42 of 76

7. (Optional) Use the Move Up or Move Down arrows to reposition the function in the package. The package executes functions in sequential order. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

1

)

2

+

To associate data records with a helper function: 1. In the Package Editor, double-click the helper function to open the Transform Editor. 2. Use the Transform Editor to add data records and transformations. Exit the editor when done. The helper function is now ready to use. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

(

2

+

You can modify the name, inputs, outputs, and record definitions of helper functions. To modify a helper function: 1. In the Package Editor, double-click the helper function that you want to modify. The Transform Editor appears. 2. In the Transform Editor, select Edit > Definition to open the Function Definition dialog.

Creating Helper Functions Reusing Helper Functions Deleting Helper Functions Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

1

2

+

You can use helper functions in more than one package by using the Include function. To include a package containing a helper function in another package: 1. Select the transform to which you want to add an existing helper function. 2. Shift-double-click the transform to open the Transform Editor. 3. Select View > Package to open the Package Editor. 4. Select Edit > Insert. 5. In the Open dialog, select the package or DML file containing the helper function you want to add. 6. Click Open. The Package Editor adds the package to the Includes folder. For more information, refer to The Include Statement and Including Packages.

Creating Helper Functions Modifying Helper Functions

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 43 of 76

Deleting Helper Functions About Helper Transform Functions Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

2

+

To delete a helper function: 1. In the Package Editor, highlight the helper function that you want to delete. 2. Press Delete. CAUTION: The system will not prompt you to verify the deletion. Use delete with caution.

Creating Helper Functions Modifying Helper Functions Reusing Helper Functions About Helper Transform Functions Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

This section contains the following topics: What is a Phase? What is a Checkpoint? Setting Phases and Checkpoints Viewing Phases Recovery Deleting Phases and Checkpoints Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

A phase is a stage of a graph that runs to completion before the start of the next stage. By dividing a graph into phases, you can save resources, avoid deadlock, and safeguard against failures. To protect a graph, all phases are checkpoints by default. When you divide a graph into phases and run it, the components upstream of the phase break (the boundary between phases) write temporary files in the layouts of the components downstream of the phase break. When the execution of the graph successfully passes the following checkpoint, the Co>Operating System deletes the files. NOTE: You must allow enough space in the layouts of the downstream components to hold all the data passing through the phase break or the graph will fail. The following diagram shows a graph that is divided into three phases:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Phase numbers appear beside each component. White phase numbers inside blue boxes checkpoint at its downstream boundary; blue phase numbers against a white background

Page 44 of 76

denote a phase that has a denote phases that do not have

checkpoints. When there are phase breaks inside a subgraph, the phase number shows the range of phases inside. When you insert an Intermediate File component into a graph, it automatically creates a phase break. Blue bars across flows indicate phase breaks that are checkpoints, unfilled blue bars across flows breaks that are not checkpoints.

that occur

indicate phase

1#+#1#/ %/ +%-# #* ##/ 2 # In Co>Operating System versions earlier than 2.12.2, the Co>Operating System checks all Input Files for existence and creates all Output Files at the beginning of graph execution. This means that you cannot, for example, reference a file in phase 1 that the graph creates in phase 0 — the graph will fail because, when it checks for the existence of the phase 1 file at the beginning of graph execution, it will not find it. To solve such problems, use an Intermediate File, or insert an m_touch command with the path to the file as an argument in the start script. Beginning with Co>Operating System version 2.12.2, the Co>Operating System checks Input Files for existence and creates Output Files at the beginning of the phase in which they are written or read. Be aware that if you set the AB_COMPATIBILITY configuration variable to an earlier version, the Co>Operating System maintains the earlier behavior.

Setting Phases and Checkpoints Viewing Phases Deleting Phases and Checkpoints Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

All phase breaks are checkpoints by default. When you include checkpoints in a graph, you can recover completed stages of the graph if it fails.You can remove the checkpoint from a phase break by clicking the Toggle Checkpoint

button.

When the execution of a graph passes a checkpoint the Co>Operating System saves all the information it needs to roll the graph back to the state it was in at the checkpoint. The Co>Operating System retains this information until the next checkpoint completes successfully, and then deletes it. By assigning phases with checkpoints to a graph, you can recover completed stages of the graph if failure occurs.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 45 of 76

For information about checkpoints in continuous flow graphs, see Descriptions of Computepoints and Checkpoints. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

To set phases with checkpoints, follow these steps: 1. Click the Edit Phases button

Phase numbers

on the Tool toolbar to turn on the Phase toolbar:

indicating phase 0 appear beside each component.

2. In the graph, select all the components you want to place in phase 1. 3. Click the Increment Phase button

.

The phase number beside each selected component, and all downstream components, becomes 1. Phase breaks appear on the flows between phases. 4. Select all the components you want to place in phase 2 and click the Increment Phase button again. 5. Repeat steps 2, 3, and 4 as many times as needed, until all components are associated with a phase. To change phase numbers after you have set them, select the phase for which you want to change the phase number and use the Increment Phase button or the Decrement Phase button to change the phase number. To select phases in a graph one after another, repeatedly click the Show Phase button

.

NOTE: By default, all phases are checkpoints, which save graph status. If you do not want to save graph status at a phase break, you can remove the checkpoint by selecting the phase and clicking the Toggle Checkpoint button from . The phase no longer saves graph status and the number is in a white box

to

.

Deleting Phases and Checkpoints Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

0 ) Click the Edit Phases button

Phase numbers

on the Tool toolbar to turn on the Phase toolbar:

appear beside each component.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 46 of 76

What is a Phase? Setting Phases and Checkpoints Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

1

'

When you encounter a graph that fails (for example, one phase of a graph encounters an error or an operator aborts the program during execution), you do not need to rerun checkpointed phases that completed prior to failure. When you run the job again, the graph recovers from the last checkpoint. For the failed phases, all processes will be terminated, all temporary files will be deleted, and all nodes and their respective files will be rolled back to their initial state unless you select Don't roll back file if job fails on the File Properties: Access tab. If you find a hardware failure, (for example, one or more nodes loses power or any of the UNIX operating systems crash), rerun the job to recover it. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

! To delete all the phases and checkpoints in a graph, follow these steps: 1. Click the Edit Phases button

Phase numbers

on the Tool toolbar to turn on the Phase toolbar:

appear beside each component.

2. Select the highest numbered phase in the graph. To select phases in a graph one after another, repeatedly click the Show Phase button 3. Click the Decrement Phase button The phase breaks

.

as many times as necessary to reset all phases to 0, the default.

disappear.

"Setting Phases and Checkpoints" Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

( Subgraphs and Macros are both reusable graph segments constructed with Ab Initio pre-built components. They are similar to each other, but serve slightly different purposes. The creation and use of subgraphs and macros is covered by the following topics: Do I Need a Custom Component, a Subgraph, or a Macro? Creating a Subgraph Creating a Macro

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 47 of 76

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

! %/

"

"

9

9

(

Ab Initio provides three facilities beyond Ab Initio pre-built components for developing graphs. These facilities are: Subgraphs Macros Custom components All three are somewhat similar to each other — considering carefully which one best suits your purposes can prove well worth the time you spend on it.

* !

*

The following decision tree will help you decide whether a custom component, a subgraph, or a macro best suits your needs.

2#/ *.

#

*.( .( ./#/*

If the solution to the task is a single executable (as depicted by the left branch of the tree), you need to use a custom component. Typically, you have a program or script you have created in the past to perform some type of data transformation, and you now want to use it in an Ab Initio graph. Alternatively, you can write a new program or script with a specific purpose in mind. A custom component lets you integrate your program or script into an Ab Initio graph. You can use a custom component in the same way that you would use an Ab Initio pre-built component. For details about creating custom components, see Creating Custom Components.

2#/ *.

#

1 2

If you can construct the solution to the task from Ab Initio pre-built components, and you can keep the number and arrangement of components static from one run of the graph to another, you can use a subgraph. Of the three facilities, a subgraph is the easiest to use. When you use a subgraph, you can define the component parameters at runtime. You can change the value of the parameters from one run of the graph to another, but the number and arrangement of the components themselves remain static.

2#/ *.

# ( 1.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 48 of 76

You need a macro if the solution to the task requires that you change, from one run of the graph to another, any of the following: The number of components Which components you use The order in which you connect the components In a macro, the components, the flows that connect them, and their parameters become runtime parameters of the graph. You can change some or all of them from one run of the graph to another. For example, you can use a macro to perform tasks similar to the following, where the number of output files you need from one run of the graph to another varies: You need to divide data records by month into separate files. At one run of the graph you need four output files: one for each of the last three months, and one for all earlier records. At a later run of the same graph, you need six output files: one for each of the last five months, and one for all earlier records.

Creating a Subgraph Creating a Macro Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

% A subgraph is a graph fragment. Just like graphs, subgraphs contain components and flows. A subgraph groups together components that perform a subtask in a graph. The subgraph creates a reusable component that performs the subtask. If a subgraph is the proper solution to your task (see Do I Need a Custom Component, a Subgraph, or a Macro?), use one of the following procedures: Creating a New Subgraph Creating a Subgraph from an Existing Graph Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

/) To create a new subgraph: 1. From the Insert menu, choose Empty Subgraph. The New Empty Subgraph dialog appears with Graph in the New Graph Name box. 2. In the New Graph Name box, enter the name you want to use for this subgraph. 3. Click OK. The New Empty Subgraph dialog disappears, and an empty subgraph icon, labelled with the name you entered in step 2, appears in the workspace.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 49 of 76

4. Double-click the empty subgraph. The subgraph expands to become the subgraph drawing sheet. 5. Create the subgraph inside the drawing sheet by inserting components from the Component Organizer and connecting them with flows. 6. Add ports to the subgraph. Observe the following considerations that are specific to a subgraph: Connect the input port(s) of the component(s) you want to receive the subgraph's input to the left edge of the drawing sheet. An in port appears on the outside of the drawing sheet at the connection point. Connect diagnostic ports, which are on the bottom edges of the Ab Initio pre-built components, to the bottom edge of the drawing sheet. An out port appears on the outside of the drawing sheet at each connection point. Connect the output port(s) of the component(s) you want to produce the subgraph's output to the right edge of the drawing sheet. An out port appears on the outside of the drawing sheet at the connection point. 7. If you want to, create parameters for the subgraph.

Creating a Subgraph from an Existing Graph Saving a Subgraph as a Component Creating Parameters for Subgraphs Viewing Subgraphs Editing Subgraphs Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

#

To create a subgraph from components in an existing graph: 1. In an existing graph, select the components you want to be part of the subgraph. 2. On the GDE menu bar, choose Edit > Subgraph > Create from Selection The Create Subgraph from Selection dialog opens. 3. Enter a name for the subgraph in the New Graph Name box. You can use alphanumeric characters, numbers, and special characters such as underscores in the name. NOTE: Non-alphanumeric characters are eliminated when you deploy the graph. The GDE moves selected components from the active graph into a subgraph. Ports on the new subgraph replace interrupted flows. If the selected component is the outermost graph itself, this command wraps a new graph around it. Subgraph components have a double-line border for easy identification, as follows:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 50 of 76

4. If you want to, create parameters for the subgraph.

Creating a New Subgraph Saving a Subgraph as a Component Creating Parameters for Subgraphs Viewing Subgraphs Adding Ports to a Subgraph Editing Subgraphs Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

'

"

When you build a subgraph, it becomes a part of the graph in which you build it. If you want to use it in other graphs, or in other places in the original graph, you need to save it as a component. To save a subgraph as a component: 1. If you do not have a components folder in your sandbox, do the following: a. Create a components folder, and a parameter with which to reference it, in your sandbox. b. Add the components folder to the Component Organizer as a top-level folder (see Add Top-level Folder for details). 2. Select the subgraph. 3. From the File menu, choose Save Component "subgraph_name" As. 4. Navigate to the components folder in your sandbox. 5. In the Save as type text box, choose Program Components (*.mpc, *.mp). 6. Click Save. NOTE: This operation does not affect the containing graph, and does not save it. You must save the full graph separately. You may need to right-click the components folder you added to the Component Organizer and then click Refresh Folder on the shortcut menu before the subgraph will appear in the components folder. Once the subgraph appears, you can drag it from the Component Organizer into any graph in which you want to use it, just as you would a pre-built component. NOTE: You can add custom help to a subgraph that you save as a component. A subgraph you save in this way becomes a linked subgraph. If you insert an instance of such a subgraph into a graph from the Component Organizer and then double-click it, the GDE displays (linked) following the name of the subgraph. In GDE 1.5 and later, if you make changes to a linked subgraph, you can update the instances of that subgraph in existing graphs, as long as you originally inserted those instances from the Component Organizer. To update the instances of a subgraph in an existing graph:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 51 of 76

1. Save the desired changes to the subgraph in the Component Organizer that you used to create the graph. 2. Select the instances of the subgraph you want to update in the graph. 3. From the GDE Edit menu, choose Update.

Creating a New Subgraph Creating a Subgraph from an Existing Graph Creating Parameters for Subgraphs Viewing Subgraphs Adding Ports to a Subgraph Editing Subgraphs Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

" You can create parameters for a subgraph by: Using the Graph Parameters Editor Using the Create button on the Parameters tab of the Properties dialog Exporting parameters from a component within the subgraph. When you export a parameter from a component in a subgraph, the GDE creates a new parameter for the containing subgraph. The value you set for this parameter determines the value of all internal parameters exported under that name. Components in a subgraph can share exported parameters.

%/

1 (#*#1 *. 1# *#1#

-#

1 2

You can use parameters in a subgraph to make the subgraph reusable. For example, suppose: You have a subgraph that reads a file of records, performs some complex transformation on the records, and generates a log file within the subgraph to keep track of how many records it processes. You want to use this subgraph in several different graphs, and want each instance of the subgraph to produce a separate log file with a unique name. To accomplish this, create a formal parameter for the subgraph, and use a $ reference to that parameter as the URL of the log file within the subgraph. Then, when you construct a graph using the subgraph, the GDE will prompt you for a value for the formal subgraph parameter. The value you supply becomes the URL of the log file. To create the parameter and the $ reference, do the following: 1. In the Graph Parameters Editor, add a parameter named logname to the subgraph. 2. In the Scope column, specify Formal, and leave the Value column blank. This creates a parameter that requires a value from the user or developer of the graph. 3. On the Description tab of the File Properties dialog for the log file, specify the URL as $logname. For each instance of the subgraph, you can set logname to a different pathname. The $ reference in the log file will use this pathname as the URL for the file.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

1# .-0%/

Page 52 of 76

1 (#*#1 1#+#1#/ #

To resolve parameter references, the GDE uses the first value it finds as it progresses through the following: The component containing the referencing parameter The containing graph and any of its container graphs A top-level graph parameter A project parameter A parameter of any common projects included by the project A shell variable defined in a Host Profile's Setup commands A Unix environment variable NOTE: In releases 1.7 and earlier, $ identifier referred to a shell environment variable not a subgraph parameter. If the GDE detects a possible conflict, it uses Legacy 1.7 Code Generation by default. To disable this, select Run > Settings and click the Script tab. Clear the Legacy 1.7 Code Generation checkbox. For additional information on references, refer to $ substitution and ${} substitution.

Creating a New Subgraph Creating a Subgraph from an Existing Graph Saving a Subgraph as a Component Viewing Subgraphs Adding Ports to a Subgraph Editing Subgraphs Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

0 ) To view the contents of a subgraph: 1. On the Description tab of the Graph Properties dialog for the subgraph, in the Protection area, select Permit View. 2. Double-click the icon for the subgraph in the graph. A new window opens with the subgraph in it, looking as it does in the following figure:

3. To return to the graph, with the subgraph in its original state, click the Close button in the top right corner of the window.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 53 of 76

Creating a New Subgraph Creating a Subgraph from an Existing Graph Saving a Subgraph as a Component Creating Parameters for Subgraphs Adding Ports to a Subgraph Editing Subgraphs Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

To add ports to a subgraph: 1. If it is not open, open the subgraph so you can see its contents. 2. Click a port of a component in the subgraph just as you would to connect a flow. 3. Drag the mouse to the outer frame of the subgraph A port appears on the outside of the frame of the subgraph. 4. Release the mouse button.

1 2 .1* %/!%/ A port binding associates ports of components in a subgraph with ports of the subgraph itself. You create port bindings automatically when you add ports to a subgraph. In the example below, the dotted gray lines represent the port bindings. The outer port always has the same properties as the inner port.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 54 of 76

Creating a New Subgraph Creating a Subgraph from an Existing Graph Saving a Subgraph as a Component Creating Parameters for Subgraphs Viewing Subgraphs Editing Subgraphs Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

# To edit a subgraph, follow these steps: 1. On the Description tab of the Graph Properties dialog for the subgraph, in the Protection area, select Permit Edit. 2. Do one of the following: Double-click the subgraph Select the subgraph and choose Edit > Subgraph from the shortcut menu. 3. Make the changes you want to the contents of the subgraph. 4. On the GDE menu bar, choose File > Save Subgraph or File > Save Subgraph As to save this subgraph for use as a component in another graph. 5. Choose File > Close Subgraph to finish.

Creating a New Subgraph Creating a Subgraph from an Existing Graph Saving a Subgraph as a Component Creating Parameters for Subgraphs Viewing Subgraphs Adding Ports to a Subgraph Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

( % ( A macro consists of two parts: A program specification file, also known as an .mpc file A .ksh script that builds a graph fragment, using the mp commands of the Shell Development Environment. If a macro is the proper solution to your task (see Do I Need a Custom Component, a Subgraph, or a Macro?), see the

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 55 of 76

following topics: The Program Specification File The .ksh script Example macro Note that creating a macro is more complicated than creating a subgraph, although there are similarities. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

"

+

The program specification file (also called the .mpc file) for a macro provides the Co>Operating System with the information it needs to run the macro's .ksh script. Write program specification files as files with .mpc extensions. All program specification files must start with . The line is followed by a series of attribute: value lines that describe the attributes of your .ksh script. See Describing Custom Components in Program Specification Files for detailed descriptions of program specification files and how to write them. Following are some details that distinguish a program specification file for a macro: In the program specification file for a macro, you must use the mpname line. The value for this line must begin with @, which tells the Co>Operating System that this is a macro, and then it must specify the complete path to the .ksh script that the macro uses. You do not use the mpname line for a custom component. In almost all cases, you will want the port lines to define soc ports, since this is the default Ab Initio interface. The majority of the Ab Initio built-in components, from which you are building the macro, use this interface. Place the completed program specification file on both the computer running the GDE, and on the run host running the Co>Operating System. In the GDE, do the following: a. If you do not have a components folder in your sandbox, create one and a parameter with which to reference it. b. Place the program specification file in the components folder. c. Add the components folder to the Component Organizer as a top-level folder (see Add Top-level Folder for details). NOTE: You can add custom help to a macro that you save in the Component Organizer. You may need to right-click the components folder you added to the Component Organizer and then click Refresh Folder on the shortcut menu before the macro will appear in the components folder. Once the macro appears, you can drag it from the Component Organizer into any graph in which you want to use it, just as you would a pre-built component. On the run host, place the file in $AB_HOME/lib, or set the $AB_LAYERED_COMPONENTS_PATH environment variable to the directory where the file resides.

Creating a Macro The .ksh script Example macro

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 56 of 76

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

* : The .ksh script provides the code a macro uses to accomplish its task. In Example macro, there is a sample .ksh script with comments that indicate some points to consider when writing a script. When you complete the .ksh script, save it on the run host in a directory that is accessible to any Ab Initio graph. This directory must be the one specified in the mp name line of the program specification file. The simplest way to create the .ksh script for a macro is as follows: 1. In the GDE, construct a graph that performs a function similar to what you want to accomplish with the macro. 2. From the Edit menu, choose Script > Generated Script. If the "Errors were detected during compilation..." message appears, click No. The generated script for your graph appears. 3. Modify the generated script as needed. The lines in the script beginning with mp provide you with the basic mp commands you need for the macro. For more details about writing .ksh scripts, see the Shell Development Environment User's Guide.

Creating a Macro The Program Specification File Example macro Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

# "

"

This section contains an example of a macro that partitions input data records into output files based on the date field in the input record format.

*2# 1. 1 (

# %+% *%./ +%-#

Following is the program specification file that describes to the Co>Operating System the .ksh script used by the macro: label: "dynamic function partition to files" mpname: "@$RUNDIR/func-part.ksh" image: "func-part.ksh" author: "Ab Initio Software" version: "1.0" comment: "dynamically function partitions into multiple files by month/year pairing" port: soc in in 1 1 argument: literal $1 parameter: positional layout special "$layout" "layout of this node" parameter: positional metadata special "$metadata.in" "input metadata" parameter: positional data_dir literal "" "directory to land files to" required parameter: positional num_files literal "" "number of files to generate (not including overflow)" required parameter: positional key literal "" "key to function partition on" required

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 57 of 76

Note the following: In the mp name line, the @ tells the Co>Operating System that this is a macro. $RUNDIR is an environment variable set to the graph's run directory — at runtime it provides the complete path to the .ksh script. The argument: literal line tells the Co>Operating System to pass the string from the label line to the .ksh script. The layout and metadata parameters are special parameters that you must use in every macro script. They allow the layout and metadata to propagate from the graph to the macro, so that the components defined in the macro can use the same layout and metadata as the components defined outside the macro. The other three parameters — data_dir, num_files, and key — will appear as parameters on the Parameters tab of the Properties dialog for the macro in the GDE. The values you enter in these parameters control the following: data_dir — the destination directory of the data records num_files— the number of files into which the macro partitions the data records key — the field(s) according to which the macro partitions the records

*2#:7 2

1% *

Following is the .ksh script this macro uses to partition the data records into the number of output files specified in the num_files parameter, according to the key specifier in the key parameter. The program specification file above describes this script — together they create the macro. This script performs the task given as an example in When to Use a Macro . It creates one output file more than the number in the num_files parameter — the extra file holds all the records from the months earlier than the months specified in the num_files parameter. The comments in the script indicate some points to consider when writing a script for a macro. ######################################################################### #!/bin/ksh # # Show some usage information in the event that a user forgets one # of the parameters. ######################################################################### if [[ $# -ne 6 ]], then echo "usage: func-part.ksh " exit 1 fi ######################################################################### # Add the parameters that the Co>Operating System will pass to the # macro. Note that these are in the order that the parameter lines # are declared in the .mpc file. ######################################################################### macroName=$1 layout=$2 metadata=$3 datadir=$4 num_months=$5 key=$6 ######################################################################### # There are three tasks we need to acomplish in the macro: # 1- Generate the partition by key expression # 2- Generate the list of filenames we wish to create # 3- Define the Ab Initio components that make up the macro # # Here we are going to do the first two:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 58 of 76

# 1- ret_string will contain the list of filenames, ie: Mar2000, # Feb2000, etc. # 2- select_string will contain the function for partitioning, # which will wind up being a large if/else statement that # checks the key field against the year and month for each file # we wish to generate. ######################################################################### set -A month_strings nul Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec select_string="" mon=$(date +"%m") year=$(date +"%Y") ret_string="" count=0 idx=$mon while [[ $count -lt $num_months ]] do if [[ $idx -eq 0 ]], then idx=12 year=$((year - 1)) fi ret_string="$ret_string ${month_strings[$idx]}$year" select_string="if ((date_month($key) == $idx) && (date_year($key) == $year)) $((count + 1)) else $select_string" count=$((count + 1)) idx=$((idx - 1)) done select_string="$select_string 0" set -A filenames $(echo $ret_string) ######################################################################### # Now we begin the actual macro. ######################################################################### mp begin-macro $macroName ######################################################################### # There are a few static components, for which there will always # be exactly one copy of the component. We include an extra output # file here, for any records that fall outside of the range we're # separating into. ######################################################################### mp function-partition functionPartition "$select_string" -layout $layout mp ofile overflow $datadir/overflow.dat mp straight-flow flow-ov functionPartition.out overflow.write -metadata $metadata ######################################################################### # Now we iteratively declare all the specific month-associated # files, up to the number given in the macro's parameter. # Note that both the components and the flows that connect them # must be dynamically declared. ######################################################################### count=0 while [[ $count -lt $num_months ]] do mp ofile outfile$count $datadir/${filenames[$count]}.dat mp straight-flow flow-$count functionPartition.out outfile$count.write -metadata $metadata count=$((count + 1)) done ######################################################################### # The Co>Operating System must know which component in the macro # is the input to the macro.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 59 of 76

######################################################################### mp port in functionPartition.in ######################################################################### # And we're done! ######################################################################### mp end-macro $macroName

Creating a Macro The Program Specification File The .ksh script Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

You can use combinations of custom and built-in components to build Ab Initio graphs and subgraphs. This section documents the creation and use of custom components through the following topics: Overview of Custom Components Building a Custom Component Calling Custom Components Describing Custom Components in Program Specification Files Examples of Program Specification Files Creating Custom Components in the Component Organizer Creating Custom Components Based on Built-in Components Creating Subgraph Custom Components from Built-in Components Adding Help to Custom Components Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

.' ' )

"

"

The term custom component can refer to any of the following: A true custom component you build from scratch to execute your own program or shell script An Ab Initio built-in component configured in a particular way and saved for further use A subgraph constructed with built-in components and saved for further use You can integrate all of the above types of components into the Ab Initio development environments. The following topics provide details for creating all three types of custom components: Building a Custom Component Creating Custom Components Based on Built-in Components Creating Subgraph Custom Components from Built-in Components

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 60 of 76

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

This section describes building a custom component from scratch to execute your own program or shell script. If the program or shell script you want to use is a simple filter that only reads from standard input and writes to standard output, use the Run Program component, and pass the appropriate command line arguments in the commandline parameter. Otherwise, you need to build a custom component. A custom component consists of two elements: Your program or shell script A program specification file, which describes your program's command line arguments, ports, parameters, and other attributes. When the GDE (Graphical Development Environment) generates the script for a graph, a custom component appears in that script as a line beginning with mp custom. In the SDE (Shell Development Environment), you write an mp custom line to call a custom component.

Calling Custom Components Describing Custom Components in Program Specification Files Examples of Program Specification Files Creating Custom Components in the Component Organizer Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

The mp custom command calls a custom component. Whether you write the command in the SDE, or the GDE generates it, the following is the syntax for an mp custom line: mp custom label .mpc_filename [positional_argument [positional_argument ...] ] [keyword_argument [keyword_argument ...] ] -layout layout_name Following is a list of the arguments to the command, with a brief description of each: label Name you want to assign to this particular instance of this custom component. .mpc_filename

Path of the program specification file that describes the program around which this component is built.

positional_argument

Value for an option, identified by its position on the mp custom line. When the GDE generates the mp custom line, it generates this value from the value of a positional parameter.

keyword_argument

Value for an option, typically identified in the .mpc file and on the mp custom line by a keyword that precedes it.

-layout layout_name Name of a layout object. See the Shell Development Environment User's Guide for details. CAUTION: Components run in temporary directories on processing nodes. Since these directories are deleted after

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 61 of 76

execution of the graph, avoid using the default working directory of a custom component for files your program writes when it runs. In other words, make sure you specify an absolute path for any files the program writes. To see examples of mp lines generated by the GDE for Ab Initio built-in components: 1. In the GDE, place a component in the workspace. 2. From the Edit menu, choose Script > Generated Script. 3. If the Errors were detected during compilation message box appears, click No. The Edit MP script window opens. 4. Find the line in the script that begins with mp component_name.

Creating Custom Components in the Component Organizer Creating Custom Components Based on Built-in Components .mpc Files Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

! +

"

"

"

Program specification files provide the Co>Operating System with the information it needs to run your program or shell script. Program specification filenames have .mpc filename extensions. In the GDE, you can drag and drop a filename ending in .mpc onto a graph to create a custom component in the graph, or you can create and store your custom components in the the Component Organizer of the GDE. All program specification files must start with . The line is followed by a series of attribute: value lines that describe the attributes of your program. This section discusses the following topics: Syntax for Program Specification Files Writing Program Specification Files Descriptions of Lines in Program Specification Files Options Substitution in Program Specification Files Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

+

When you write a program specification file, use the following syntax: [ label: label ] [ author: author-name ] [ version: version-number ] [ comment: comment ] image: path

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 62 of 76

[ exit: code ] [ port: type direction name [location] [ordering] [fan-preference] [min-flows] [max-flows] [record-format] ] [ environment: env-variable value ] [ argument: literal value1 [value2 ...] ] [ argument: flow portname ] [ argument: partition ] [ argument: depth ] [ argument: file filename ] [ argument: expression file filename ] [ argument: expression string ] [ argument: transform file filename ] [ argument: transform string ] [ parameter: placement name type default-value description [restrictions] ] [ metadata type: dest-portname = {source-portname | value} ] Also see Examples of Program Specification Files . Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

+

When you write a program specification file: You must include the line and the image line. All other lines are optional. You can use as many port, argument, parameter, and metadata type lines as you need to describe your particular program. To see examples of program specification files, use the following procedure to display the program specification files for Ab Initio built-in components: 1. In the GDE, open the the Component Organizer. 2. In the Component Organizer, right-click the name of almost any component. A pop-up menu appears. 3. On the pop-up menu, click Edit As text. The Edit Program Component window opens with the program specification file for that component in it. Comments in a program specification file are denoted by the following: Lines that start with # From // to the end of the line From /* to */. You can write a program specification file in any text editor, or you can write the file in the Component Organizer of the GDE, which provides a template you can follow. To write a program specification file in the Component Organizer: 1. In the GDE, open the Component Organizer.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 63 of 76

2. Right-click My Components. 3. From the shortcut menu, choose New > Program. A New Component icon appears under My Components. 4. Do one of the following: Enter a name in the New Component icon. Press Enter to accept New Component as the name. 5. Right click the New Component icon. A pop-up menu appears. 6. From the pop-up menu, choose Edit As text. The Edit Program Component window opens. 7. Write your own program specification file by editing the template in the window. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

-

"

+

This section describes in detail each of the attribute: value lines you can use in writing a program specification file. For an overview of the syntax for defining a program specification file, see Syntax for Program Specification Files. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

.

"

+

Options substitution means that, at runtime, the Co>Operating System uses a value from the mp custom line or from an environment variable in place of a symbol you insert in a value in the .mpc file. The symbol you insert tells the Co>Operating System which value from the line to use. The result is the same as if you had written the value at the spot in the .mpc file where you inserted the symbol. At runtime, the Co>Operating System performs options substitution on the following lines: image, environment, and literal. A list of the symbols you can use for options substitution and the values they indicate follows : If a value in the .mpc file includes:

The Co>Operating System uses:

$n, where n is an integer

The value of the nth option on the mp custom line. The value of the first option is $1 (see Program Based on Unix grep Program ).

$n-m, where n and m are integers

The values of the nth through mth options on the mp custom line.

$n-, where n is an integer

The values nth through last options on the mp custom line.

$*

The values of all the options on the mp custom line.

$environment-variable

The value of the environment variable.

Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

# "

"

+

This section contains the following examples of program specification files: Program Based on Unix cat Program

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 64 of 76

Program Based on Unix grep Program Program Based on Unix diff Program Program Specification for Filter By Expression Component Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

The following program specification file defines an Ab Initio program component based on the UNIX cat program. It has two ports (in and out). It also specifies that cat exits with code 0 when it succeeds. image: "/bin/cat" // program image is good old /bin/cat exit: 0 // cat returns zero when successful port: std in in port: std out out Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

The following program specification file defines an Ab Initio program for a case-insensitive string searcher based on the UNIX grep program. It takes one option, the string to search for, which it passes as an argument to grep: image: "/usr/bin/grep" exit: any // ignore return status port: std in in // inputs from stdin port: std out out // outputs to stdout argument: literal "-i" // first argument to grep is "-i" argument: literal $1 // 2nd arg to grep is the search string Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

The following program specification file defines an Ab Initio program based on the UNIX diff program. It takes two input flows via two ports, in1 and in2, which can be named pipes or files. It passes the paths of those named pipes or files as arguments to diff, by using two argument: flow lines: image: "/usr/bin/diff" exit: any port: npipe in in1// input port in1 (first input) port: npipe in in2// input port in2 (second input) port: std out out // Two arguments to diff are the names of the two paths it opens. // We supply the paths of the named pipes which are connected // to the two input ports. argument: flow in1 argument: flow in2 Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

+

#

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

" 10/4/2010

Beyond the Basics

Page 65 of 76

This program specification file is from the component Filter by Expression. To view other program specification files for the Ab Initio built-in components in the GDE, see the procedure in Writing Program Specification Files . label: "Filter by Expression" mpname: "select-transform" image: "unitool" author: "Ab Initio Software" version: "Built-in" comment: "Reads records from the 'in' port ..." port: soc in in port: soc out out 1 1 port: soc out deselect right 0 1 port: soc out reject bottom 0 1 port: soc out error bottom 0 1 port: soc out log bottom 0 1 parameter: positional select_expr expression "" "Filter expression" parameter: keyword limit integer "0" "Max rejects before error" parameter: keyword ramp float "0.0" "Rate of rejects before error" parameter: keyword log string "" "Log expression" metadata type: out = in metadata type: deselect = in Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

"

.

5

To create a custom component in the Component Organizer, follow these steps: 1. If you do not have a components folder in your sandbox, do the following: a. Create a components folder, and a parameter with which to reference it, in your sandbox. b. Add the components folder to the Component Organizer as a top-level folder (see Add Top-level Folder for details). 2. In the Component Organizer, right click the components folder and choose one of the following on the shortcut menu, depending on the type of component you want to create: New > File New > Program New > Table The label Newtype_of_component of the new component or dataset appears in the components folder. 3. In the Component Organizer, click to select the Newtype_of_component label, click again and overtype to change the name, then press Enter. 4. Right-click the label and choose Edit or Edit as text from the shortcut menu to modify the .mpc file or .mdc file. NOTE: For examples of .mpc files, right-click any built-in component supplied with the GDE and choose Edit as Text. 5. On the File menu of the editor, select Save label.mpc NOTE: You can add custom help to a custom component that you create from the Component Organizer. To use the new component, drag the label from the Component Organizer onto the graph just as you would with a built-in component. A component icon with the same label appears in the graph.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 66 of 76

Creating Custom Components Based on Built-in Components Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

8

" To configure an Ab Initio built-in component and then save it for reuse as a custom component: 1. If you do not have a components folder in your sandbox, do the following: a. Create a components folder, and a parameter with which to reference it, in your sandbox. b. Add the components folder to the Component Organizer as a top-level folder (see Add Top-level Folder for details). 2. Insert the component you want to use in the graph. 3. Double-click the component to open its Properties dialog, and then do any of the following that are appropriate: Set the URL on the Description tab. Set the parameters on the Parameters tab. Set the record format on the Ports tab. Set the layout on the Layout tab. 4. In the Properties dialog, click OK. 5. Select the component. 6. On the Menu bar of the GDE, choose Edit > Save Component "Label" As. 7. Navigate to the components folder in your sandbox and click Save. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"

"

"

8

" You can quickly create a custom component by building a subgraph based on built-in components and saving it as a component. This can be helpful in a variety of situations, including testing custom help. To create a custom component by building a subgraph and saving it as a component, see the following topics: Creating a New Subgraph Creating a Subgraph from an Existing Graph Saving a Subgraph as a Component Creating Parameters for Subgraphs Viewing Subgraphs Editing Subgraphs Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 67 of 76

2

"

"

After you create a custom component, you can add help to your component. You do this by adding a HelpCmd parameter to the custom component. The value of the HelpCmd parameter is the name of compiled HTML Help file. This file must have a .chm file extension and can have an optional HTML reference to a specific topic. After you implement help, users can view it by: Right-clicking the component in the Component Organizer or in a graph and selecting Component Help Clicking Component Help in the component's Properties dialog The GDE displays the compiled HTML Help file that you specified as the value of the HelpCmd parameter. This is a standalone help file. It is not integrated with Ab Initio Help. To implement help for a custom component: 1. Add the HelpCmd parameter to your custom component. 2. Use a test help file to verify that help is working. 3. Use Microsoft HTML Help Workshop or another tool to generate the real compiled HTML Help file. For your convenience, we provide a project template that you can use to generate help that is similar in appearance to the Ab Initio component help. However, you are not required to use this template. NOTE: Currently, you cannot implement help for custom component parameters. However, the user can display component help and then scroll to the description of a particular parameter. Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

2

"

"

"

"

For your custom component to access a help file, the component requires a HelpCmd parameter. The value of the HelpCmd parameter is the location of your help file. Your help file must be a .chm file, and it must be in the same directory as your custom component. If you already created the help file, ensure that it is in the same directory as your custom component. If you have not already created the help file, we provide a test help file that you can use to set up and test help. This is the Custom_Components.chm file. This file is in the self-extracting zip file, custom_help.exe, in the GDE installation directory. Typically, this is C:|Program Files\Ab Initio\Ab Initio GDE\Program Files. Save Custom_Components.chm in the directory that contains your custom component and rename it to the name you want your help file to have. Ensure that you specify the .chm file extension when you rename the file. For example, you might save Custom_Components.chm as SalesDept_SQL_Query.chm. How you add the HelpCmd parameter to your custom component depends on whether the custom component is a subgraph.

2#/ *2#

*.( .( ./#/*% /.*

1 2

When your custom component is not a subgraph, you edit the program specification file to add the HelpCmd parameter to the component. To add the HelpCmd parameter by editing the program specification file: 1. In the GDE Component Organizer, right-click your custom component and select Edit As text to display the component's program specification file in a text editor. 2. If your custom component already has a HelpID parameter: a. Make the existing help parameter line a comment by putting a # sign at the beginning of the line. b. Put the HelpCmd parameter line (described in step 3) above the existing help parameter line. 3. As the first parameter line, add a line in the following format:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 68 of 76

parameter: {local | implicit} HelpCmd string "filename.chm[::/topic.html]" "" Be sure to include an empty string at the end of the line. For example, the following specification opens the help file and displays the custom_comp1.html file. When your help file contains help for more than one component, you would replace custom_comp1.html with the name of the HTML file that contains the help for this component. parameter: implicit HelpCmd string "SalesDept_SQL_Query.chm::/custom_comp1.html" "" The following specification opens the help file and displays the default file. You can use this form when your help file contains help for just one component. parameter: local HelpCmd string "SalesDept_SQL_Query.chm" "" 4. From the Edit Program Component menu bar, choose File > Save and close the window. 5. In the Component Organizer, right-click the top-level Components folder and click Refresh Folder. NOTE: After you add the HelpCmd parameter, you can use the GDE Properties: Parameters tab to modify the value of the HelpCmd parameter.

2#/ *2#

*.( .( ./#/*%

1 2

When your custom component is a subgraph, you use the GDE to add the HelpCmd parameter to your custom component: 1. In the GDE Component Organizer, right-click your custom component and click Edit. 2. Right-click the background of the component and click Properties. 3. On the Properties: Description tab, select Permit Edit if it is not already selected. 4. On the Parameters tab, click Create. 5. In the Parameter Properties dialog, enter the following: Name — Enter HelpCmd. Type — Select String. 6. Click OK to exit the Parameter Properties dialog. 7. On the Properties: Parameter tab, in the Embedded box, specify the path to the help file. For example: Custom_Components.chm::/custom_comp1.html opens the help file and displays the custom_comp1.html file. When your help file contains help for more than one component, follow the name of the .chm file with a two colons and a forward slash, and then the name of the HTML file that contains the help for this custom component. Custom_Components.chm opens the help file and displays its default file. If your help file contains help for just one component, use this form. 8. For Scope, select Local, and for Kind select Implicit. 9. Click OK to save the parameter values and close the Properties dialog. 10. Press Ctrl+S to save the component. 11. Exit and restart the GDE.

(

2

" 0

If you want to change the value of the HelpCmd parameter, always do so by following the first three steps in this procedure. This ensures that your changes are saved correctly.

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 69 of 76

Adding Help to Custom Components Testing Help for a Custom Component Using Ab Initio Templates to Create Custom Help Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

*

2

"

"

There are two ways to test custom component help: In the Component Organizer, right-click your custom component and click Component Help in the pop-up menu. In a graph, right-click your custom component and click Component Help. The help for your custom component should appear. If it does not, check the following: Ensure that your help file is valid by opening it from Windows Explorer. Verify that the help file and the custom component are in the same directory. Confirm that you correctly spelled the help file name when you specified the value of the HelpCmd parameter. Confirm that you correctly spelled HelpCmd when you added it as a new parameter. If the help file opens but it contains information for more than one component and help for the wrong component is appearing, check the spelling of the specification at the end of the value for the HelpCmd parameter. Make sure you are correctly specifying the name of the file that contains the information for this custom component.

Adding Help to Custom Components Adding the HelpCmd Parameter to a Custom Component Using Ab Initio Templates to Create Custom Help Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

%

*"

"2

The help file you create for a custom component must be a Microsoft compiled HTML Help file, which has a .chm extension. There are a variety of tools that you can use to create the .chm file. The most basic method is to use HTML Help Workshop to import and compile ASCII text files that contain descriptions of your components. Microsoft HTML Help Workshop is free software that you can download from: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/htmlhelp/html/hwMicrosoftHTMLHelpDownloads.asp If you are unfamiliar with HTML Help Workshop, we provide a template help project that you can use to create custom component help. You can use the template project to create help for one or more components. This section discusses the following topics: Descriptions of the Custom Help Template Files Setting Up to Create Custom Help Editing the Custom Help Project Files

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 70 of 76

Building a Custom Help File Testing Help Files Built From Ab Initio Templates Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

!

"2

*"

+

The Ab Initio template help project is in a self-extracting zip file, custom_help.exe, in the GDE installation directory. The template help project includes the following files: Filename

Description

custom_components.hhp

A Microsoft Help Workshop project. This project creates a three-pane compiled help file (.chm) containing help for one or more custom components.

custom_compTOC.hhc

The Table of Contents file for the project. This creates the Contents pane.

custom_compIX.hhk

The Index file for the project. This creates the Index pane.

Custom_Components.chm

A sample compiled help file created from the template files. You can use this file to test your GDE parameter settings before you create your own help file.

custom_comp.html

The first page of the custom help system. It contains links to one or more custom component descriptions.

custom_comp1.html

The second through n pages of the custom help system. Copy this page for each component you want to document (except for the last one).

custom_comp2.html

The last page of the custom help system. It contains the last component in the help system.

document_custom.css

A cascading style sheet that determines the appearance of the help pages on screen.

document_custom_print.css

A cascading style sheet that determines the appearance of the help pages when they are printed.

generic_component.gif

A sample graphic file you can replace with your own graphics.

Custom_Component_Help.pdf Documentation for adding help to custom components. The documentation in this PDF file and the documentation in the Ab Initio Help are identical.

Using Ab Initio Templates to Create Custom Help Setting Up to Create Custom Help Editing the Custom Help Project Files Building a Custom Help File Testing Help Files Built From Ab Initio Templates Copyright © 2004, Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.

"2 To set up the software to create a custom help file:

file://C:\Documents and Settings\kumarv1\Local Settings\Temp\~hh730C.htm

10/4/2010

Beyond the Basics

Page 71 of 76

1. If you have not already done so, install the Microsoft HTML Workshop software. 2. Run the custom_help.exe file in a work directory to unpack the template and help project files. We recommend that you save the .exe as a backup since you will be editing the files it contains. 3. Double-click Custom_Component.chm to become familiar with the organization of the three-pane help file. NOTE: If you want, you can copy and rename this sample .chm file to the name of one of your custom components and use it to test your GDE parameter setup. Remember to copy the .chm file to the same directory that contains the custom component. Notice that the help pages contain Next >> and Replace and, for each file, perform the search and global replacements specified in step 2 of Editing the Custom Help Project Files. 5. Choose File > Save Project to save your changes. 6. Click the Compile HTML file icon

and click Compile in the Create a compiled file dialog.

If the text substitutions were made correctly, there should be no errors. For help debugging in the HTML Help Workshop, refer to its online help. 7. Click the View compiled file icon

to display the compiled help file. Verify that the:

Substitutions were correctly made. Two component links are present (if applicable). Next >> and