Advanced ETL Processor extracts data from any database, transforms, validates it and loads into another database automat...
Advanced ETL Processor User manual
© DB Software Laboratory 2008-2009 www.dbsoftlab.com
Advanced ETL Processor User Manual
Contents Contents....................................................................................................................................... 2 1. Introduction ............................................................................................................................. 9 2. Key features ........................................................................................................................... 10 2.1 Extraction Process ........................................................................................................... 10 2.1.1 Multiple Data Formats.............................................................................................. 10 2.1.2 Multiple Databases and Table Processing................................................................ 10 2.1.3 Other Database Features........................................................................................... 10 2.2 Summary of the Extraction Process: ............................................................................... 10 2.3 Validation Process ........................................................................................................... 11 2.4 Summary of Validation Processes:.................................................................................. 11 2.5 Transformation Process ................................................................................................... 11 2.6 Powerful Data Transformation ........................................................................................ 11 2.7 Summary of Transformation Processes:.......................................................................... 12 2.8 Loading Process............................................................................................................... 13 2.9 Summary of the Loading Process:................................................................................... 13 3. Requirements......................................................................................................................... 14 4. Advanced ETL Processor Architecture .............................................................................. 16 5. Processing Data ..................................................................................................................... 16 5. Processing Data ..................................................................................................................... 17 5.1 Screen Overview ............................................................................................................. 17 Main tool bar ..................................................................................................................... 18 5.2 Transformation Properties ............................................................................................... 19 5.3 Template tab .................................................................................................................... 20 5.4 Execution Log Tab .......................................................................................................... 21 5.5 Rejected Records Tab...................................................................................................... 22 5.6 Creating new transformation and working with Objects................................................. 23 5.7 Working with Reader ...................................................................................................... 25 5.7.1 Universal Data Reader.............................................................................................. 25 5.7.2 Data source is a Text File. ........................................................................................ 26 5.7.3 Data source is MS Access or Excel File................................................................... 31 5.7.4 Data source is a DBF File......................................................................................... 35 5.7.5 Data source is a ODBC Connection ......................................................................... 37 5.7.6 Data source is a MS SQL Server .............................................................................. 40 5.7.7 Data source is an Oracle Database ........................................................................... 41 5.7.8 Data source is an MySql........................................................................................... 42 5.7.9 Data source is an PostgreSQL Database................................................................... 43 5.7.10 Data source is an Interbase or Firebird Database ................................................... 44 5.8 Working with Validator................................................................................................... 45 5.8.1 Debugging Validation............................................................................................... 47 5.9 Working with Transformer.............................................................................................. 50 5.9.1 Auto mapping ........................................................................................................... 52 5.10 Working with Grouper .................................................................................................. 53 5.11 Working with Sorter ...................................................................................................... 54 5.12 Working with Deduplicator........................................................................................... 56 5.13 Working with UnPivot .................................................................................................. 57 5.14 Working with Pivot ....................................................................................................... 62 5.15 Working with Writer ..................................................................................................... 67 5.15.1 Target type is a Text File........................................................................................ 67 Copyright © 2009 DB Software Laboratory
Page 2 of 194
Advanced ETL Processor User Manual 5.15.2 Target type is MS Access or Excel File.................................................................. 70 5.15.3 Target type is a DBF File........................................................................................ 72 5.15.4 Target type is ODBC connection............................................................................ 74 5.15.5 Target type is MS SQL Server................................................................................ 75 5.15.6 Target type is Oracle............................................................................................... 77 5.15.7 Target type is MySql .............................................................................................. 78 5.15.8 Target type is PostgreSQL...................................................................................... 79 5.15.9 Target type is Interbase/Firebird............................................................................. 80 5.15.10 How to Update/Delete Records ............................................................................ 82 5.15.11 Running SQL Scripts............................................................................................ 84 6. Validation Rules .................................................................................................................... 85 6.1 Strings.............................................................................................................................. 86 6.1.1 Is Null ....................................................................................................................... 86 6.1.2 Is Empty String......................................................................................................... 86 6.1.3 Is Alpha .................................................................................................................... 86 6.1.4 Is Alpha Numeric...................................................................................................... 86 6.1.4 Is Hex........................................................................................................................ 87 6.1.5 Is Equal To ............................................................................................................... 87 6.1.6 Is Starts With ............................................................................................................ 88 6.1.7 Is Ends With ............................................................................................................. 88 6.1.8 Contains.................................................................................................................... 88 6.1.9 In List........................................................................................................................ 88 6.2 Numbers .......................................................................................................................... 92 6.2.1 Is Number ................................................................................................................. 92 6.2.2 Is Integer ................................................................................................................... 92 6.2.3 Is Positive ................................................................................................................. 92 6.2.4 Is Negative................................................................................................................ 92 6.2.5 Is Number Between .................................................................................................. 93 6.2.6 Is Number Less Than................................................................................................ 94 6.2.7 Is Number More Than .............................................................................................. 95 6.3 Date.................................................................................................................................. 96 6.3.1 Is Date....................................................................................................................... 96 6.3.2 Is Date Between........................................................................................................ 97 6.3.3 Is Date Less Than ..................................................................................................... 98 6.3.4 Is Date More Than.................................................................................................... 98 6.3.5 Is Monday, Is Tuesday, Is Wednesday, Is Thursday, Is Friday, Is Saturday, Is Sunday, Is Today, Is Yesterday, Is Tomorrow, Is Weekend, Is Weekday..................... 99 6.3.6 Is January, Is February, Is March, Is April, Is May, Is June, Is July, Is August, Is September, Is October, Is November, Is December, Is Current Month, Is Last Month, Is Next Month........................................................................................................................ 99 6.3.7 Is 1st Quarter, Is 2nd Quarter, Is 3rd Quarter, Is 4th Quarter, Is Current Quarter, Is Last Quarter, Is Next Quarter .......................................................................................... 100 6.3.8 Is Leap Year, Is Current Year, Is Last Year, Is Next Year ..................................... 100 6.3.9 Is Current Week, Is Last Week, Is Next Week....................................................... 100 6.3.10 Is Within Past Minutes, Is Within Past Hours, Is Within Past Days, Is Within Past Weeks, Is Within Past Months ........................................................................................ 101 6.4 Time............................................................................................................................... 102 6.4.1 Is Time.................................................................................................................... 102 6.4.2 Is Time Between..................................................................................................... 103 6.4.3 Is Time Less Than .................................................................................................. 104 Copyright © 2009 DB Software Laboratory
Page 3 of 194
Advanced ETL Processor User Manual 6.4.4 Is Time More Than ................................................................................................. 104 6.4.5 Is Second ................................................................................................................ 105 6.4.6 Is Minute................................................................................................................. 105 6.4.7 Is Hour 24 ............................................................................................................... 105 6.4.8 Is Hour 12 ............................................................................................................... 105 6.4.9 Is PM ...................................................................................................................... 106 6.4.10 Is AM.................................................................................................................... 106 6.5 Regular Expressions ...................................................................................................... 107 6.5.1 Regular Expression................................................................................................. 108 6.5.2 Is IP Address V4..................................................................................................... 109 6.5.3 Is IP Address V6..................................................................................................... 109 6.5.4 Is Email................................................................................................................... 109 6.5.5 Is ISBN 10 .............................................................................................................. 109 6.5.6 Is ISBN 13 .............................................................................................................. 110 6.5.7 Is Credit Card ......................................................................................................... 110 6.5.8 Is URL .................................................................................................................... 110 6.5.9 Is UNC.................................................................................................................... 110 6.5.10 Is File Name.......................................................................................................... 111 6.5.11 Is UK Post Code ................................................................................................... 111 6.5.12 Is UK VAT Number ............................................................................................. 111 6.5.13 Is UK Bank Sort Code .......................................................................................... 111 6.5.14 Is UK Vehicle Registration Number .................................................................... 112 6.5.15 Is UK Driver Licence............................................................................................ 112 6.5.16 Is UK National Insurance Number ....................................................................... 112 6.5.17 Is UK Phone Number ........................................................................................... 112 6.5.18 Is UK Mobile Number.......................................................................................... 113 6.5.19 Is US Zip Code ..................................................................................................... 113 6.5.20 Is US State ............................................................................................................ 113 6.5.21 Is US Social Security Number.............................................................................. 113 6.5.22 Is US Phone Number ............................................................................................ 114 6.5.23 Is Canadian Zip Code ........................................................................................... 114 6.5.24 Is Canadian Provincial Code ................................................................................ 114 6.5.25 Is Canadian Branch-Transit/ Bank code ............................................................... 114 6.5.26 Is Canadian Social Insurance Number.................................................................. 115 6.5.27 Is Australian Post Code ........................................................................................ 115 6.5.28 Is Australian Phone Number................................................................................. 115 6.5.29 Is Australian Mobile Number ............................................................................... 116 6.5.30 Is Argentinean Post Code ..................................................................................... 116 6.5.31 Is Dutch Post Code ............................................................................................... 116 6.5.32 Is Dutch Phone Number ....................................................................................... 116 6.5.33 Is Dutch Car Number Plate................................................................................... 117 6.5.34 Is Belgian Post Code............................................................................................. 117 6.5.35 Is Belgian VAT Number....................................................................................... 117 6.5.36 Is French Post Code.............................................................................................. 117 6.5.37 Is French VAT Number........................................................................................ 118 6.5.38 Is French Phone Number ...................................................................................... 118 6.5.39 Is German Post Code ............................................................................................ 118 6.5.40 Is German VAT Number ...................................................................................... 118 6.5.41 Is German Car License Plate ................................................................................ 119 6.5.43 Is German IBAN................................................................................................... 119 Copyright © 2009 DB Software Laboratory
Page 4 of 194
Advanced ETL Processor User Manual 6.5.44 Is Italian Post Code............................................................................................... 119 6.5.45 Is Italian VAT Number......................................................................................... 119 6.5.46 Is Italian IBAN ..................................................................................................... 120 6.5.47 Is Italian Phone Number ....................................................................................... 120 6.5.48 Is Italian Mobile Number ..................................................................................... 120 6.5.49 Is Spanish Post Code ............................................................................................ 120 6.5.50 Is Spanish VAT Number ...................................................................................... 121 6.5.51 Is Spanish IBAN................................................................................................... 121 6.5.52 Is Spanish Phone Number .................................................................................... 121 6.5.53 Is Swedish Post Code ........................................................................................... 121 6.5.54 Is Swedish VAT Number ..................................................................................... 121 6.5.55 Is Swedish IBAN .................................................................................................. 122 6.5.56 Is Swedish Phone Number.................................................................................... 122 6.5.57 Is Swedish Person Number................................................................................... 122 6.5.58 Is Austrian Mobile Phone Number....................................................................... 123 6.5.59 Is Danish Post Code.............................................................................................. 123 6.5.60 Is Danish VAT Number........................................................................................ 123 6.5.61 Is Danish Phone Number...................................................................................... 123 6.5.62 Is Danish Civil Registration Number ................................................................... 124 6.5.63 Is Finnish VAT Number ....................................................................................... 124 6.5.64 Is Greek VAT Number ......................................................................................... 124 6.5.65 Is Irish VAT Number............................................................................................ 124 6.5.66 Is Luxembourg VAT Number .............................................................................. 125 6.5.67 Is Portuguese VAT Number ................................................................................. 125 6.5.68 Is Estonian Post Code........................................................................................... 125 6.5.69 Is Estonian VAT Number ..................................................................................... 125 6.5.70 Is Latvian VAT Number....................................................................................... 126 6.5.71 Is Lithuanian VAT Number.................................................................................. 126 6.5.72 Is Slovenian VAT Number ................................................................................... 126 6.5.73 Is Slovakian VAT Number ................................................................................... 126 6.5.74 Is Slovakian IBAN................................................................................................ 127 6.5.75 Is Polish VAT Number......................................................................................... 127 6.5.76 Is Polish IBAN...................................................................................................... 127 6.5.77 Is Czech Republic VAT Number.......................................................................... 127 6.5.78 Is Czech Republic IBAN ...................................................................................... 128 6.5.79 Is Netherlands VAT Number................................................................................ 128 6.5.81 Is Maltese VAT Number ...................................................................................... 128 6.5.82 Is Algerian Post Code ........................................................................................... 129 6.5.83 Is Andorran Post Code.......................................................................................... 129 6.5.84 Is Andorran IBAN ................................................................................................ 129 6.5.85 Is Azerbaijani Post Code ...................................................................................... 129 6.5.86 Is Bahraini Post Code ........................................................................................... 130 6.5.87 Is Bangladeshi Post Code ..................................................................................... 130 6.5.88 Is Belarusian Post Code........................................................................................ 130 6.5.89 Is Brunei Post Code .............................................................................................. 130 6.5.90 Is Chilean Post Code ............................................................................................ 131 6.5.91 Is Croatian Post Code ........................................................................................... 131 6.5.92 Is Cyprus Post Code ............................................................................................. 131 6.5.93 Is Cyprus VAT Number........................................................................................ 131 6.5.94 Is Egyptian Post Code........................................................................................... 132 Copyright © 2009 DB Software Laboratory
Page 5 of 194
Advanced ETL Processor User Manual 6.5.95 Is Ethiopian Post Code ......................................................................................... 132 6.5.96 Is Faroe Islands Post Code.................................................................................... 132 6.5.97 Is French Polynesia Post Code ............................................................................. 132 6.5.98 Is Guadeloupe Post Code...................................................................................... 133 6.5.99 Is Guinean Post Code ........................................................................................... 133 6.5.100 Is Guinea-Bissau Post Code ............................................................................... 133 6.5.101 Is Russian Post Code .......................................................................................... 133 6.5.102 Is Russian Phone Number .................................................................................. 134 6.5.103 Is Israeli Phone Number ..................................................................................... 134 6.5.104 Is Israeli Mobile Phone Number......................................................................... 134 6.5.105 Is Peru Phone Number........................................................................................ 134 6.5.106 Is Peru Cellular Number ..................................................................................... 135 6.5.107 Is Indian Post Code............................................................................................. 135 6.5.108 Is Indian Phone Number ..................................................................................... 135 6.5.109 Is Indian Mobile Number ................................................................................... 135 6.5.110 Is Indian Vehicle Registration Number .............................................................. 136 6.5.111 Is South African Phone Number......................................................................... 136 6.5.112 Is South African Mobile Number ....................................................................... 136 6.5.113 Is South African VAT Number .......................................................................... 136 6.5.114 Is US or CA Zip code ......................................................................................... 137 7. Transformation Functions ................................................................................................... 138 7.1 Strings............................................................................................................................ 138 7.1.1 Upper Case ............................................................................................................. 138 7.1.2 Lower Case ............................................................................................................. 138 7.1.3 Proper Case............................................................................................................. 138 7.1.4 First Up................................................................................................................... 138 7.1.5 Delete Spaces.......................................................................................................... 139 7.1.6 Trim ........................................................................................................................ 139 7.1.7 Left Trim ................................................................................................................ 139 7.1.8 Right Trim .............................................................................................................. 139 7.1.9 Sub String ............................................................................................................... 140 7.1.10 Replace ................................................................................................................. 141 7.1.11 Ensure Prefix ........................................................................................................ 142 7.1.12 Ensure Suffix ........................................................................................................ 142 7.1.13 Ensure No Prefix .................................................................................................. 142 7.1.14 Ensure No Suffix .................................................................................................. 143 7.1.15 Left Pad ................................................................................................................ 143 7.1.16 Right Pad .............................................................................................................. 143 7.1.17 Escape String ........................................................................................................ 144 7.1.18 UnEscape String ................................................................................................... 144 7.1.19 Delete.................................................................................................................... 144 7.1.20 Left ....................................................................................................................... 145 7.1.21 Right ..................................................................................................................... 145 7.1.22 Left Delete ............................................................................................................ 145 7.1.23 Right Delete.......................................................................................................... 146 7.2 Numbers ........................................................................................................................ 147 7.2.1 Round ..................................................................................................................... 147 7.2.3 Abs.......................................................................................................................... 148 7.2.4 Sign......................................................................................................................... 148 7.3 Date................................................................................................................................ 149 Copyright © 2009 DB Software Laboratory
Page 6 of 194
Advanced ETL Processor User Manual 7.3.1 Date Format ............................................................................................................ 149 7.3.2 Current Date ........................................................................................................... 149 7.4 Miscellaneous ................................................................................................................ 150 7.4.1 Length..................................................................................................................... 150 7.4.2 Literal ..................................................................................................................... 150 7.4.3 User......................................................................................................................... 151 7.4.4 Splitter .................................................................................................................... 151 7.4.5 Joiner ...................................................................................................................... 152 7.4.6 Calculation.............................................................................................................. 153 7.4.7 Lookup.................................................................................................................... 182 7.4.8 Previous Value........................................................................................................ 185 7.4.9 Reader Parameter.................................................................................................... 186 7.4.10 Sequence............................................................................................................... 187 7.4.11 Running Total....................................................................................................... 188 8. Date formats ........................................................................................................................ 189 9. Command Line .................................................................................................................... 190 10. Support Procedure ............................................................................................................. 191 11. License Agreement ............................................................................................................ 192
Copyright © 2009 DB Software Laboratory
Page 7 of 194
Advanced ETL Processor User Manual Copyright Copyright © 2009 DB Software Laboratory Limited. All rights reserved. No portion of this document may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any language, in any form or by any means, without prior written consent of DB Software Laboratory Limited.
Note to U.S. Government users: Documentation and programs related to restricted rights - use, duplication or disclosure is subject to restrictions set forth in: GSA FMSS Schedule Contract No. GS00K92AFS2505-PS05. License Information You have access to Advanced ETL Processor software and documentation pursuant to the terms of a Software License Agreement granted by DB Software Laboratory Limited. As a user of this software and documentation, you are bound by the terms of the Software License Agreement. All rights, title, and interest to this software remain with DB Software Laboratory Limited. Requests for copies of this publication and for technical information about DB Software Laboratory products should be made directly to DB Software Laboratory Limited. Disclaimer All information in this manual is subject to periodic change and revision without notice. While every effort has been made to ensure that this manual is accurate, DB Software Laboratory Limited excludes its liability for errors or inaccuracies (if any) contained herein. Registered Marks Any products or services mentioned or depicted in this document are identified by the trademarks or service marks of their respective companies or organisations. Edition Information This document refers to Advanced ETL Processor version 2.4.2.
Copyright © 2009 DB Software Laboratory
Page 8 of 194
Advanced ETL Processor User Manual
1. Introduction The Advanced ETL Processor is a Codeless ETL Tool. The Advanced ETL Processor is an end to end database extraction and importing tool. The beauty of the system is that it saves the drudgery and manual tasks normally required for tasks of this type, such as the writing of code, and all the transformations, validations and general checks normally performed. Using traditional methods, operations of this type can only be performed in stages, and not as one smooth operation! For example, the traditional method of importing data from one system to another is to write specific code to extract data from the source database, e.g. an Oracle database, by creating a CSV comma de-limited file, and then writing code in the new language or system, for instance Microsoft Access, and then performing the import. However, the operation does not end there. Any data imported has to be sorted, duplicated and loaded into the database using appropriate primary and foreign key constraints. This is only possible by creating code designed to achieve this process. You then need to manually send an e-mail to the administrator when the process is complete. In other words, each stage cannot be left to run in an automated fashion and has to be completed before proceeding to the next. As you can see, the process is not straightforward. The Advanced ETL Processor automates all these processes in a simple and transparent fashion, and all without writing any code whatsoever. As stated, the tool to handle any kind of database, including Oracle, Microsoft Access, SQL Server, DB2, MySQL, Excel spreadsheets, and a wide range of others. It is an excellent tool for those organisations who work with data warehouses, and where this involves working with a number of disparate databases. Existing users find that the tool provides several benefits over existing tools such as Oracle SQL Loader, BCP, DTS or SSIS, such as the ability to update records automatically via utilisation of the primary key.
Copyright © 2009 DB Software Laboratory
Page 9 of 194
Advanced ETL Processor User Manual
2. Key features The ability of the Advanced ETL Processor to work with a number of disparate systems means that it is provided with a rich set of tools and functionality, which can be used in isolation or combined in a powerful way with other toolsets, either within the processor or with other third party tools. It is in effect, an "engineering" environment for the movement of data to and from different sources. We will now explain and define the variety of features which are provided as part of the toolset. Let us first have a look at the data extraction process. 2.1 Extraction Process 2.1.1 Multiple Data Formats The extraction process can handle a variety of data formats, including multiple delimited or fixed width text files. The power of this system however, is in its ability to find files to load using mask. 2.1.2 Multiple Databases and Table Processing It can easily interpret and manipulate Microsoft Access data, from a number of different databases. Again the end user can use a mask to find the tables to load the data from. It does also apply to Excel and DBF/FoxPro files. 2.1.3 Other Database Features The Advanced ETL Processor also has other useful database features, such as the ability to connect to any Object Database Connectivity (ODBC) database. ODBC was intended to enable developers' access to any data through any application, regardless of the DBMS used for managing that data. ODBC boasts platform independence since it has been purposefully designed in a way that makes it distinct from database systems, programming languages and operating systems. The Advanced ETL Processor utilises this technology to great effect. So what if you don’t have an Oracle or MySQL database? What about SQL Server? No problem, the Advanced ETL Processor can handle SQL Server data as efficiently as data from any other type of database. 2.2 Summary of the Extraction Process: • • • • • • •
Multiple Delimited or Fixed width Text files Multiple Excel files + Multiple Excel Spreadsheets Multiple MS Access Databases + Multiple Tables Multiple DBF Files Any ODBC compliant database Direct connection to Oracle, SQL Server, MySql, PostgreSQL and Interbase/Firebird Multiple Tables
Copyright © 2009 DB Software Laboratory
Page 10 of 194
Advanced ETL Processor User Manual 2.3 Validation Process The Advanced ETL Process has a robust validation process built in. The types and nature of the validations taking place can be tweaked and configured by the user. A full range of validation functions are included. Validations can be performed on the basis of data type, lists of values, and regular expressions, which can be individually changed according to requirements. 2.4 Summary of Validation Processes: Validation: • • • • • • •
190 Validation Functions String Validation Number Validation Date Validation Time Validation Validation Against list of values Regular Expressions: Post codes, Phone Numbers, VAT Numbers ETC
2.5 Transformation Process The process of “transformation” involves the manipulation of data so that it can be converted from one format or data type to another. The Advanced ETL Processor provides a useful set of “built in” transformation functions. Transformations can be performed on strings, numbers, dates, coded values and whole chunks of data. The transformation functionality allows for the sorting and grouping of data according to requirements. Data can also be duplicated as required. 2.6 Powerful Data Transformation In addition to the standard data transformation abilities, the processor can also perform complex translation functions. An example would be if an integer variable = “1”, then set a text variable to “yes”. Once data is translated, it is possible to join data into a completely new format or present it in a new way. The Advanced ETL Processor provides many flexible alternatives for data manipulation, and these are not difficult to take advantage of. The Advanced ETL Processor also provides the ability to derive calculated values, join data together from multiple fields, summarise multiple rows at once, or can split or merge columns at will. The flexibility and power of the processor means that you can customize data transformation and conversion functions according to your requirements with a click of the mouse. This saves hours and hours of coding.
Copyright © 2009 DB Software Laboratory
Page 11 of 194
Advanced ETL Processor User Manual
2.7 Summary of Transformation Processes: Transformation: • • • • • • • • • • • • • • • • •
39 Transformation Functions String Transformation Number Transformation Date Transformation Sorting Grouping Deduplication Translating coded values (e.g., if the source system stores 1 for male and 2 for female, but the warehouse stores M for male and F for female) Deriving a new calculated value (e.g., sale amount = qty * unit price) Joining together data from multiple fields Summarizing multiple rows of data (e.g., total sales for each store, and for each region) Generating surrogate key values Transposing or pivoting (turning multiple columns into multiple rows or vice versa) Splitting a column into multiple columns (e.g., putting a comma-separated list specified as a string in one column as individual values in different columns) Customised Transformation Primary Key Generation Running Totals
Copyright © 2009 DB Software Laboratory
Page 12 of 194
Advanced ETL Processor User Manual
2.8 Loading Process The loading capability of the Advanced ETL Processor is superior to other basic tools such as SQL Loader, because it provides the administrator with several options and ways of providing database load capability without creating any code. Other tools allow you to load data into a single database table at a time, under a single instance. The ETL processor on the other hand allows you to specify multiple upload targets which mean you can save time loading each individual table. Another useful feature is that you can execute SQL scripts either prior to the load or after the load has completed. This is useful for tidying up the data or providing a report on the result of the load process once it completes. It does not matter if the file to be loaded is a fixed or variable length text file, the Advanced ETL Processor can handle it. It works with Access, DBF files, Oracle, SQL Server and any ODBC compliant database. It is one of the most powerful database loading tools in the marketplace. 2.9 Summary of the Loading Process: Loading: • • • • • • • • • • • •
Multiple Data Targets SQL scripts execution before and after loading Delimited or Fixed width Text files MS Access Database Microsoft Excel File DBF File Any ODBC compliant database Oracle database Microsoft SQL Server database MySQL PostgreSQL Interbase/Frebird
Oracle There are two ways of loading available: • Oracle direct path loading • Conventional path SQL server This software uses the same API as Microsoft DTS and SSIS service.
Copyright © 2009 DB Software Laboratory
Page 13 of 194
Advanced ETL Processor User Manual
3. Requirements Below is the list of Software that must be installed before installation of Advanced ETL Processor: Software Microsoft Windows MDAC
Version 98 or higher 2.6 or higher
MS Excel ODBC driver MS Excel 2007 ODBC driver MS Access ODBC driver
4.00.6305.00 or higher 12.00.4518.1014 or higher 4.00.6364.00 or higher
MS Access 2007 ODBC driver
12.00.4518.1014 or higher
Separate download
FoxPro ODBC driver SQL ODBC driver
6.1.8629.1 or higher 2000.81.9041.40
Separate download
Oracle Client
7.3.4 or higher
Interbase client
)otes Part of OS on W2K, XP, Vista. Part of OS on W2K, XP, Vista. Separate download Part of OS on W2K, XP, Vista.
Part of OS on W2K, XP, Vista. Provided by Oracle GDS32.DLL
Only to work with Excel 3.0-2003 files Only to work with Excel 2007 files Only to work with MS Access 95-2003 Databases Only to work with MS Access 2007 Databases Only to work with DBF/FoxPro Files Only to work with MS SQL Server 7/2008 Only to work with Oracle Databases Only to work with Interbase or Firebird Databases
Separate Downloads: FoxPro ODBC driver http://msdn.microsoft.com/en-us/vfoxpro/bb190233.aspx Office 2007 Data Access Components http://www.microsoft.com/downloads/details.aspx?FamilyID=7554F536-8C28-4598-9B72EF94E038C891&displaylang=en
Copyright © 2009 DB Software Laboratory
Page 14 of 194
Advanced ETL Processor User Manual Working with Oracle: Oracle client 8.1.7 to load data into/from Oracle Or Oracle client 9 to load data into/from Oracle Or Oracle client 10 to load data into/from Oracle Or Oracle client 11 to load data into/from Oracle )ote: Depending on the Requirements you may or may not need to have all components installed There is no need to install clients for MySql and PostgreSQL they are integrated into the software itself.
Copyright © 2009 DB Software Laboratory
Page 15 of 194
Advanced ETL Processor User Manual
4. Advanced ETL Processor Architecture The following graphical depiction of a typical ETL installation shows how the software interacts with other components and interfaces belonging to various databases. The processor sits in the middle of the various databases and carries out its tasks, such as converting, transforming and validating data from various sources. As you can see that the Advanced ETL Processor uses native low level API's for specific databases, such as the Oracle Call Interface (OCI), or, in order to handle the Microsoft SQL Server database the BCP API. Other API's can also be "plugged in" such as the ODBC API which allows MS Access, DBF to be processed. Note: One of major benefit of using native low level API' is a great performance boost it gives to Advanced ETL Processor.
Oracle SQL
OCI API
SQL
Advanced ETL BCP API
ODBC
ODBC
Excel MS Access DBF Files MySQL
Oracle
Reader Validator Transformer Groupper Sorter Deduplicator Pivot Unpivot Writer
ODBC API
Excel MS Access DBF Files
Integrated Clients
MySQL
PostgreSQL
PostgreSQL
Interbase
Interbase
Flat Files
Flat Files
Extraction Copyright © 2009 DB Software Laboratory
Loading Page 16 of 194
Advanced ETL Processor User Manual
5. Processing Data In order to load data from the data source into the data target you must define data mapping between target table and data source. 5.1 Screen Overview The data processing screen provides a number of settings, for the three stages of reading, transformation and writing. This is facilitated by the main tool bar, and a list of available objects. The main reader toolbar provides the user with all the necessary functionality to setup the reading process, according to the settings provided. Main Tool bar
Reader Toolbar
Available Objects
Field Width
Source file/Table Field No Reader Fields
Copyright © 2009 DB Software Laboratory
Reader data Page 17 of 194
Advanced ETL Processor User Manual Main tool bar The main tool bar provides a number of icons which allow the user to create the steps required to carry out the automatic functions required.
1 2 3 1. 2. 3. 4. 5. 6. 7. 8.
4 5
6 7
8
Map Properties New Map Load Template From the file Saves Template to the file Saves Template under the new name Prints Map Print Preview Map Process Data
Copyright © 2009 DB Software Laboratory
Page 18 of 194
Advanced ETL Processor User Manual
5.2 Transformation Properties to change Transformation properties, please note that Default Date format Click applies to all templates. Default Date format only used for new Date related validation and transformation functions for example Is Date, Is Date between ETC
Copyright © 2009 DB Software Laboratory
Page 19 of 194
Advanced ETL Processor User Manual
5.3 Template tab The template shows the actual transformation script source.
Copyright © 2009 DB Software Laboratory
Page 20 of 194
Advanced ETL Processor User Manual
5.4 Execution Log Tab The execution log provides information about the Advanced ETL Processor and the actions it took during its operations. This is useful when you wish to analyse the activities of individual processes during their execution.
Copyright © 2009 DB Software Laboratory
Page 21 of 194
Advanced ETL Processor User Manual
5.5 Rejected Records Tab Occasionally, records will be rejected by the Advanced ETL Processor. This may be due to things like corrupt records which have been read in a format not expected by the processor. The rejected records tab allows the user to see a list of all the rejected records, and other information about the nature of the rejection and where this occurred in the process.
Copyright © 2009 DB Software Laboratory
Page 22 of 194
Advanced ETL Processor User Manual
5.6 Creating new transformation and working with Objects . New transformation consists of Reader, To create new transformation click Validator, Transformer and Writer. You may add new object at any time by dragging it from the Objects panel.
Objects Panel
To join two objects together, first click on the source Output button then drag it on to the Input button Output Buttons
Drag and Drop to Join
Input Button
To remove the Join, repeat the same procedure again. To delete an object right click on the object and click delete.
Copyright © 2009 DB Software Laboratory
Page 23 of 194
Advanced ETL Processor User Manual To change object properties double click on it
Deletes All Links
Deletes Links for Selected objects
Copyright © 2009 DB Software Laboratory
Page 24 of 194
Advanced ETL Processor User Manual
5.7 Working with Reader The Reader extracts data from various data sources.
Description
Records/ Lines Read
Records Rejected
Every Transformation created must have one reader object. The Reader connects to the Data source and extracts data from it. Depending on the Data source type some or double options may not be available. To change the Reader properties click click on the Reader object. 5.7.1 Universal Data Reader One of the benefits of using Advanced ETL Processor is a concept of Universal Data Reader Consider following scenario: You have several extracts from your accounting system • Text files • Excel Files • Access Databases All files have same format such as: • Same field’s order • Same field’s names • Same date and number formats Provided that you are using Advanced ETL Processor all you need to do is to change connection type no mapping will be lost. Other tools use different connector for different databases and some of then even sell separate licences for them. That mean that user have to recreate mapping for new files/ databases.
Copyright © 2009 DB Software Laboratory
Page 25 of 194
Advanced ETL Processor User Manual
5.7.2 Data source is a Text File. The Reader is capable of extracting data from delimited or fixed width files. All parameters are user definable. It can also skip a number of Header and Footer lines
Copyright © 2009 DB Software Laboratory
Page 26 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 27 of 194
Advanced ETL Processor User Manual The Rejected records file can have a pre-defined format. The following dialogue allows the user to set this up.
Copyright © 2009 DB Software Laboratory
Page 28 of 194
Advanced ETL Processor User Manual Data View One of the useful features of the Advanced ETL Processor is the ability to view the resultant data prior, or subsequently processed. The data view looks like a spreadsheet view as follows:
Source file
Number of records to show
Field Width
Field Number
Field Name
Data View Toolbar The data view toolbar allows you to change various aspects of the data view, such as refreshing data as it changes and setting properties for how the data will look. You can also switch between viewing of the data and checking to see how the data is defined i.e. the data dictionary, via the “Switch to Data Definition View”.
1
2 1. 2. 3. 4. 5. 6. 7.
3
4 5
6 7
Reader Properties Refresh Data Edit file in external editor Add new column Delete last column Switch to Data View Switch to Data Definition View
Copyright © 2009 DB Software Laboratory
Page 29 of 194
Advanced ETL Processor User Manual Data Definition View )ote: You may rename fields and change field’s width here. (Works only for text files)
Data Definition View Toolbar Within the Data Definition view you can perform a number of actions. These allow you to change how you want data to be represented in the data view screens. The navigation also allows switching between views.
1
2
3
4
5 6
7
8
9 10
1. Reader Properties 2. Refresh Data 3. Print Data Definition 4. Print Preview Data Definition 5. Find 6. Edit file in external editor 7. Add new column 8. Delete last column 9. Switch to Data View 10. Switch to Data Definition View
Copyright © 2009 DB Software Laboratory
Page 30 of 194
Advanced ETL Processor User Manual
5.7.3 Data source is MS Access or Excel File
Copyright © 2009 DB Software Laboratory
Page 31 of 194
Advanced ETL Processor User Manual
Query Builder
)ote: It is also possible to use Query builder to design queries
Copyright © 2009 DB Software Laboratory
Page 32 of 194
Advanced ETL Processor User Manual
Data View
Copyright © 2009 DB Software Laboratory
Page 33 of 194
Advanced ETL Processor User Manual Data View Toolbar
1
2 1. 2. 3. 4. 5.
3
4 5
Reader Properties Refresh Data Edit file in external editor Switch to Data View Switch to Data Definition View
Data Definition View
Data Definition View Toolbar
1
2 1. 2. 3. 4. 5. 6. 7. 8.
3 4
5 6
7 8
Reader Properties Refresh Data Print Data Definition Print Preview Data Definition Find Edit file in external editor Switch to Data View Switch to Data Definition View
Copyright © 2009 DB Software Laboratory
Page 34 of 194
Advanced ETL Processor User Manual
5.7.4 Data source is a DBF File
Source Directory
Copyright © 2009 DB Software Laboratory
Page 35 of 194
Advanced ETL Processor User Manual Data View Source Directory
Data View Toolbar
1
2 1. 2. 3. 4.
3 4
Reader Properties Refresh Data Switch to Data View Switch to Data Definition View
Data Definition View
Data Definition View Toolbar
1
2 1. 2. 3. 4. 5. 6. 7.
3 4
5
6 7
Reader Properties Refresh Data Print Data Definition Print Preview Data Definition Find Switch to Data View Switch to Data Definition View
Copyright © 2009 DB Software Laboratory
Page 36 of 194
Advanced ETL Processor User Manual 5.7.5 Data source is a ODBC Connection
Copyright © 2009 DB Software Laboratory
Page 37 of 194
Advanced ETL Processor User Manual
ODBC Connection Strings It also possible to use ODBC connection strings for both Reader and Writer connections. For example for MS SQL Server connection string is: Driver={SQL Native Client}; Server=myServerAddress;Database=myDataBase;Uid=myUsername;Pwd=myPassword; One of the major benefits of using connection strings that it makes it no longer necessary to create ODBC Dsn’s manually on every single computer where Advanced ETL Processor is installed. It also gives a greater control over the connection parameters. )ote: Leave user name and password blank and provide it within connection string More information about connection strings can be found at: http://www.connectionstrings.com The simplest way to create ODBC connection string is to use ODBC Connection builder dialog. Double click on ODBC driver name to create a connection string
Copyright © 2009 DB Software Laboratory
Page 38 of 194
Advanced ETL Processor User Manual Data View
Source Table
Data View Toolbar
1
2 1. 2. 3. 4.
3 4
Reader Properties Refresh Data Switch to Data View Switch to Data Definition View
Data Definition View
Data Definition View Toolbar
1
2 1. 2. 3. 4. 5. 6. 7.
3 4
5
6 7
Reader Properties Refresh Data Print Data Definition Print Preview Data Definition Find Switch to Data View Switch to Data Definition View
Copyright © 2009 DB Software Laboratory
Page 39 of 194
Advanced ETL Processor User Manual
5.7.6 Data source is a MS SQL Server
)ote: All properties are very similar to ODBC connection
Copyright © 2009 DB Software Laboratory
Page 40 of 194
Advanced ETL Processor User Manual
5.7.7 Data source is an Oracle Database
)ote: All properties are very similar to ODBC connection
Copyright © 2009 DB Software Laboratory
Page 41 of 194
Advanced ETL Processor User Manual
5.7.8 Data source is an MySql
)ote: All properties are very similar to MS SQL Server connection
Copyright © 2009 DB Software Laboratory
Page 42 of 194
Advanced ETL Processor User Manual
5.7.9 Data source is an PostgreSQL Database
)ote: All properties are very similar to MS SQL Server connection
Copyright © 2009 DB Software Laboratory
Page 43 of 194
Advanced ETL Processor User Manual
5.7.10 Data source is an Interbase or Firebird Database
)ote: All properties are very similar to ODBC/MS Access connection
Copyright © 2009 DB Software Laboratory
Page 44 of 194
Advanced ETL Processor User Manual
5.8 Working with Validator Advanced ETL Processor Data Validator guarantees to your application database that every data value is correct and accurate. There are several types of data validation. • • • • •
Data type validation Range checking Code checking Complex validation Pattern checking
One of the simplest forms of data validation is verifying the data type. Data type validation answers such simple questions as "Is the string alphabetic?" and "Is the number valid?" As an extension of simple type validation, range checking ensures that the provided value is within allowable minimums and maximums. For example, a character data type service code may only allow the alphabetic letters A through Z. All other characters would not be valid. Code checking is a bit more complicated, typically requiring a lookup table. For example, maybe your application calculates sales tax for only certain state codes. You would need to create a lookup object to hold the authorized, taxable state codes. Pattern checking when you checking structure of the data field for example social security number format or car registration number. Regular expressions used quite often for pattern checks. It does not matter which business you are in sooner or later you will discover that there is something wrong with the data ant it has to be validated. Here when Advanced ETL Processor Validation can help. Records Rejected by validation Rules
Processed Records Discarded Records )ote: • Records can also be rejected by the Server. • If you have several validation rules and one of them rejects record and another discards it, record will be discarded
Copyright © 2009 DB Software Laboratory
Page 45 of 194
Advanced ETL Processor User Manual To change Validator properties double click on it
Validator Toolbar
Inputs Validation Rules
Objects Panel
Data Sample
Copyright © 2009 DB Software Laboratory
Page 46 of 194
Advanced ETL Processor User Manual Validator Toolbar
1 2
3 4
5
6 7 8 9
10 11
1. Print 2. Print Preview 3. Delete All objects 4. Delete All Links 5. Process Data 6. First Record 7. Previous Record 8. Next Record 9. Last Record 10. Show Data 11. Show Objects Panel 5.8.1 Debugging Validation To start debugging validation press Process Data button the Data grid. Pass
Warning
. To test data edit it in
Failed
To test different value edit data in the Data grid
Copyright © 2009 DB Software Laboratory
Page 47 of 194
Advanced ETL Processor User Manual To Change Validation Rule properties double click on it
)ote: Use to include actual value into default value To add new Validation rule drag and drop it from the Validation rules panel
Copyright © 2009 DB Software Laboratory
Page 48 of 194
Advanced ETL Processor User Manual There more than 190 Validation Functions at the moment. They are grouped in five different categories 1. 2. 3. 4. 5.
String Number Date Time Regular Expressions
To choose appropriate category click on Category toolbar
It is also possible to apply several validation rules to Input field by joining them
Data is considered as validated when all validation rules are succeeded.
Validated Data
Failed Data )ote: If you have several validation rules and one of them rejects record and another discards it, record will be discarded Copyright © 2009 DB Software Laboratory
Page 49 of 194
Advanced ETL Processor User Manual 5.9 Working with Transformer
Records Transformed )ote: To change Transformer properties double click on it Transformer Toolbar
Inputs
Data Sample
Copyright © 2009 DB Software Laboratory
Outputs
Objects Panel
Transformations
Page 50 of 194
Advanced ETL Processor User Manual Transformer Toolbar
1 2
3
4 5
6
7 8 9 10 11 12 13
1. Print 2. Print Preview 3. Auto Map 4. Delete All objects 5. Delete All Links 6. Process Data 7. First Record 8. Previous Record 9. Next Record 10. Last Record 11. Copies Inputs to Outputs (Only visible if transformer is connected to any object other than writer) 12. Show Data 13. Show Objects Panel )ote: If transformer is connected to any object other than writer it is possible to modify list of Outputs. When transformer is connected to writer list of Outputs is taken from Writer. Examples: Example below splits date field into Day, Month and Year using ‘/’ as a delimiter
Another Example Converts Customer Name into More readable format:
Copyright © 2009 DB Software Laboratory
Page 51 of 194
Advanced ETL Processor User Manual 5.9.1 Auto mapping If the Inputs and Outputs have got the same names you may use Auto map feature. Click
, Fill in all necessary data and click map.
Copyright © 2009 DB Software Laboratory
Page 52 of 194
Advanced ETL Processor User Manual
5.10 Working with Grouper
Records In
Records Out
To change grouper properties double click on the object. Choose appropriate grouping order and grouping type. Fields with grouping type Ignore are not passed to the next object. Grouper does not perform sorting.
Copyright © 2009 DB Software Laboratory
Page 53 of 194
Advanced ETL Processor User Manual
5.11 Working with Sorter
Records In
Records Out
About Sorting In computer science and mathematics, a sorting algorithm is an algorithm that puts elements of a list in a certain order. All sorting algorithms are very well documented and it is hard to invent something new or much faster that existing algorithm. To achieve the best performance is important to load all the data into the memory which is not possible for large amount of data. Advanced ETL Processor uses combination of Quicksort and Merge sort. Portion of Data is loaded in the memory and sorted using Quicksort algorithm than data is saved into temporary file. Once all portions are sorted temporary files merged together using Merge sort algorithm. Quicksort Quicksort is a divide and conquer algorithm which relies on a partition operation: to partition an array, we choose an element, called a pivot, move all smaller elements before the pivot, and move all greater elements after it. This can be done efficiently in linear time and in-place. We then recursively sort the lesser and greater sublists. Efficient implementations of quicksort (with in-place partitioning) are typically unstable sorts and somewhat complex, but are among the fastest sorting algorithms in practice. Together with its modest O(log n) space usage, this makes quicksort one of the most popular sorting algorithms, available in many standard libraries. The most complex issue in quicksort is choosing a good pivot element; consistently poor choices of pivots can result in drastically slower (O(n²)) performance, but if at each step we choose the median as the pivot then it works in O(n log n). Merge sort Merge sort takes advantage of the ease of merging already sorted lists into a new sorted list. It starts by comparing every two elements (i.e., 1 with 2, then 3 with 4...) and swapping them if the first should come after the second. It then merges each of the resulting lists of two into lists of four, then merges those lists of four, and so on; until at last two lists are merged into the final sorted list. Of the algorithms described here, this is the first that scales well to very large lists, because its worst-case running time is O(n log n).
Copyright © 2009 DB Software Laboratory
Page 54 of 194
Advanced ETL Processor User Manual To change Sorter properties double click on the object. Tick sort, select field order and sort order. Field order must be unique for sort field. Data is loaded in the memory first than sorted and passed to the next object.
Copyright © 2009 DB Software Laboratory
Page 55 of 194
Advanced ETL Processor User Manual
5.12 Working with Deduplicator
Records In
Records Out
To change Deduplicator properties double click on the object. Tick Deduplicate for the fields you wish to depuplicate, only ticked fields are passed to the next object. )ote: There is no need to sort data before deduplication
Copyright © 2009 DB Software Laboratory
Page 56 of 194
Advanced ETL Processor User Manual
5.13 Working with UnPivot
Records In
Records In
Example of Pivoted Data
Group 1
Group 2
Group 3
Group 4
To change UnPivot properties double click on the object. Fill in Description, Group field name, create all necessary Groups and Outputs. Once it is done Map input fields to outputs and Groups
UnPivoted Data Group field
Copyright © 2009 DB Software Laboratory
Page 57 of 194
Advanced ETL Processor User Manual UnPivot Properties
Copyright © 2009 DB Software Laboratory
Page 58 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 59 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 60 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 61 of 194
Advanced ETL Processor User Manual 5.14 Working with Pivot
Records In
UnPivoted Data
Records Out
Fields to Pivot
Pivot Key
Set Key
Pivoted Data
Copyright © 2009 DB Software Laboratory
Page 62 of 194
Advanced ETL Processor User Manual Since there are more than one field to pivot let’s join Profit and Sales fields together first using tab a delimiter than pivot the data and split it again
Joining Data
Copyright © 2009 DB Software Laboratory
Page 63 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 64 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 65 of 194
Advanced ETL Processor User Manual Data after Pivoting
Splitting Pivoted Data
Copyright © 2009 DB Software Laboratory
Page 66 of 194
Advanced ETL Processor User Manual
5.15 Working with Writer
Inserted Records
Deleted Records
Updated Records
Errors
Every Transformation created must have at least one writer object. Writer connects to Target database and loads data into it. Depending of the target type some options may not be available. To change the Writer properties click or double click on the Writer object. 5.15.1 Target type is a Text File Writer is capable of saving data into delimited or fixed width files. All parameters are user definable.
Copyright © 2009 DB Software Laboratory
Page 67 of 194
Advanced ETL Processor User Manual
Data View
Number of records to show
Copyright © 2009 DB Software Laboratory
Target File
Page 68 of 194
Advanced ETL Processor User Manual
Data View Toolbar
1
2 1. 2. 3. 4. 5. 6. 7.
3
4 5
6 7
Reader Properties Refresh Data Edit file in external editor Add new column Delete last column Switch to Data View Switch to Data Definition View
Data Definition View )ote: You may rename fields and change field’s width here. (Works only for text files)
Data Definition View Toolbar
1
2
3
4
5 6
7
8
9 10
1. Reader Properties 2. Refresh Data 3. Print Data Definition 4. Print Preview Data Definition 5. Find 6. Edit file in external editor 7. Add new column 8. Delete last column 9. Switch to Data View 10. Switch to Data Definition View
Copyright © 2009 DB Software Laboratory
Page 69 of 194
Advanced ETL Processor User Manual 5.15.2 Target type is MS Access or Excel File
Data View
Copyright © 2009 DB Software Laboratory
Page 70 of 194
Advanced ETL Processor User Manual Data View Toolbar
1
2 1. 2. 3. 4. 5. 6. 7. 8.
3 4
5 6 7 8
Reader Properties Refresh Data Print Print Preview Find Edit file in external editor Switch to Data View Switch to Data Definition View
Data Definition View
Data Definition View Toolbar
1
2 1. 2. 3. 4. 5. 6. 7. 8.
3
4
5 6
7 8
Reader Properties Refresh Data Print Data Definition Print Preview Data Definition Find Edit file in external editor Switch to Data View Switch to Data Definition View
Copyright © 2009 DB Software Laboratory
Page 71 of 194
Advanced ETL Processor User Manual 5.15.3 Target type is a DBF File
Data View
Copyright © 2009 DB Software Laboratory
Page 72 of 194
Advanced ETL Processor User Manual
Data View Toolbar
1
2 1. 2. 3. 4. 5. 6. 7.
3 4
5 6 7
Reader Properties Refresh Data Print Print Preview Find Switch to Data View Switch to Data Definition View
Data Definition View
Copyright © 2009 DB Software Laboratory
Page 73 of 194
Advanced ETL Processor User Manual 5.15.4 Target type is ODBC connection
Copyright © 2009 DB Software Laboratory
Page 74 of 194
Advanced ETL Processor User Manual
5.15.5 Target type is MS SQL Server
Check constraints Ensure that any constraints on the destination table are checked during the bulk copy operation. By default, constraints are ignored. Keep identity Specify that there are values in the data file for an identity column. Keep )ULLS Specify that any columns containing a null value should be retained as null values, even if a default value was specified for that column in the destination table. Batch size Specify the number of rows in a batch. The default is the entire data file. The following values for the Batch size property have these effects: If you set Batch size to zero, the data is loaded in a single batch. The first row that fails will cause the entire load to be cancelled, and the step fails.
Copyright © 2009 DB Software Laboratory
Page 75 of 194
Advanced ETL Processor User Manual If you set Batch size to one, the data is loaded a row at a time. Each row that fails is counted as one row failure. Previously loaded rows are committed. If you set Batch size to a value greater than one, the data is loaded one batch at a time. Any row that fails in a batch fails that entire batch; loading stops and the step fails. Rows in previously loaded batches are either committed or, if the step has joined the package transaction, provisionally retained in the transaction, subject to later commitment or rollback.
Copyright © 2009 DB Software Laboratory
Page 76 of 194
Advanced ETL Processor User Manual
5.15.6 Target type is Oracle
)ote: Option ‘Commit every Array’ works only for Oracle conventional path loading
Copyright © 2009 DB Software Laboratory
Page 77 of 194
Advanced ETL Processor User Manual
5.15.7 Target type is MySql
Copyright © 2009 DB Software Laboratory
Page 78 of 194
Advanced ETL Processor User Manual
5.15.8 Target type is PostgreSQL
Copyright © 2009 DB Software Laboratory
Page 79 of 194
Advanced ETL Processor User Manual
5.15.9 Target type is Interbase/Firebird
Copyright © 2009 DB Software Laboratory
Page 80 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 81 of 194
Advanced ETL Processor User Manual
5.15.10 How to Update/Delete Records In order to Update/Delete records you must specify update key.
Update Key For the example provided below, Advanced ETL Processor will execute the following SQL (Update key is CustomerId, OrderNo).
Add )ew And Update Old Records Select count(*) from [DEMO].[dbo].[orders] where CustomerId=? And OrderNo=? If any records found Advanced ETL will update them by executing Copyright © 2009 DB Software Laboratory
Page 82 of 194
Advanced ETL Processor User Manual Update [DEMO].[dbo].[orders] set orderdate=?, amount=? where customerid=? And OrderNo=? If no records found Advanced ETL will add new records Update Records Update [DEMO].[dbo].[orders] set OrderDate=?, Amount=? where CustomerId=? And OrderNo=? Delete Records Delete from [DEMO].[dbo].[orders] Where CustomerId=? And OrderNo=? )ote: “Add New And Update Old Records” is not supported for SQL Server Connection use ODBC connection instead.
Copyright © 2009 DB Software Laboratory
Page 83 of 194
Advanced ETL Processor User Manual 5.15.11 Running SQL Scripts Advanced ETL Processor is capable of running SQL Scripts before and after the transformation.
SQL Editor Toolbar
1 2
3 4
5
6 7 8 9 10 11 12 13 14 15 16
1. Loads Script from a file 2. Saves Script to a file 3. Print 4. Print Preview 5. Clear Script 6. Cut 7. Copy 8. Paste 9. Undo 10. Find 11. Find Next 12. Find Previous 13. Replace 14. Run 15. Stop on error 16. Show/Hide Log Copyright © 2009 DB Software Laboratory
Page 84 of 194
Advanced ETL Processor User Manual
6. Validation Rules This chapter represents list of validation rules together with short description grouped by category. Every Validation Rule/function could have some specific parameters and common parameters to control data flow of Advanced ETL. Most if it is self explanatory see the example below:
)ote: To change validation rule properties double click on it
Copyright © 2009 DB Software Laboratory
Page 85 of 194
Advanced ETL Processor User Manual
6.1 Strings
6.1.1 Is Null
Category: Validation Function Description: Checks if the data is equal to null. Properties: Default 6.1.2 Is Empty String
Category: Validation Function Description: Checks if the data is Empty String. (Length of the string is zero). Properties: Default 6.1.3 Is Alpha
Category: Validation Function Description: Checks if the data contains only Alpha characters. Properties: Default Pass: FCD|||mmm|||ABC Fail: adc1|||ad b||12 D 6.1.4 Is Alpha Numeric
Category: Validation Function Description: Checks if the data contains only Alpha Numeric characters. Properties: Default Pass: FCD12|||mm11m|||1ABC Fail: adc-1|||ad_b||12 D
Copyright © 2009 DB Software Laboratory
Page 86 of 194
Advanced ETL Processor User Manual 6.1.4 Is Hex
Category: Validation Function Description: Checks if the data contains valid HEX string Properties: Default Pass: 3D|||FF|||CE Fail: DK|||KK||12D 6.1.5 Is Equal To
Category: Validation Function Description: Checks if the data is equal to supplied value Properties:
Copyright © 2009 DB Software Laboratory
Page 87 of 194
Advanced ETL Processor User Manual 6.1.6 Is Starts With
Category: Validation Function Description: Checks if the data starts with supplied value Properties: Same as ‘Is Equal To’ 6.1.7 Is Ends With
Category: Validation Function Description: Checks if the data ends with supplied value Properties: Same as ‘Is Equal To’ 6.1.8 Contains
Category: Validation Function Description: Checks if the data contains supplied value Properties: Same as ‘Is Equal To’ 6.1.9 In List
Category: Validation Function Description: Checks if the data exists within supplied list values Properties: List of values can be entered by the user or extracted for the file(s) or database. When source data for the “List of values” consists of more than 1 field only first one is used.
Copyright © 2009 DB Software Laboratory
Page 88 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 89 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 90 of 194
Advanced ETL Processor User Manual
Copyright © 2009 DB Software Laboratory
Page 91 of 194
Advanced ETL Processor User Manual
6.2 Numbers
6.2.1 Is Number
Category: Validation Function Description: Checks if the data is a Number Properties: Default Pass: 12|||123.6|||1 Fail: adc1|||ad b||11,112.D 6.2.2 Is Integer
Category: Validation Function Description: Checks if the data is an Integer Properties: Default Pass: 12|||123|||1 Fail: 12.5|||ad b||11,112.8 6.2.3 Is Positive
Category: Validation Function Description: Checks if the data is a Positive Number Properties: Default 6.2.4 Is Negative
Category: Validation Function Description: Checks if the data is a Negative Number Properties: Default
Copyright © 2009 DB Software Laboratory
Page 92 of 194
Advanced ETL Processor User Manual
6.2.5 Is Number Between
Category: Validation Function Description: Checks if the data is a number between two values Properties:
Copyright © 2009 DB Software Laboratory
Page 93 of 194
Advanced ETL Processor User Manual
6.2.6 Is Number Less Than
Category: Validation Function Description: Checks if the data is a number less than value Properties:
Copyright © 2009 DB Software Laboratory
Page 94 of 194
Advanced ETL Processor User Manual
6.2.7 Is Number More Than
Category: Validation Function Description: Checks if the data is a number more than value Properties:
Copyright © 2009 DB Software Laboratory
Page 95 of 194
Advanced ETL Processor User Manual 6.3 Date 6.3.1 Is Date
Category: Validation Function Description: Checks if the data is a Date Properties:
Copyright © 2009 DB Software Laboratory
Page 96 of 194
Advanced ETL Processor User Manual
6.3.2 Is Date Between
Category: Validation Function Description: Checks if the data is a Date between two values Properties:
Copyright © 2009 DB Software Laboratory
Page 97 of 194
Advanced ETL Processor User Manual
6.3.3 Is Date Less Than
Category: Validation Function Description: Checks if the data is a Date less than value Properties:
6.3.4 Is Date More Than
Category: Validation Function Description: Checks if the data is a Date more than value Properties: Same as ‘Is Date Less Than’
Copyright © 2009 DB Software Laboratory
Page 98 of 194
Advanced ETL Processor User Manual 6.3.5 Is Monday, Is Tuesday, Is Wednesday, Is Thursday, Is Friday, Is Saturday, Is Sunday, Is Today, Is Yesterday, Is Tomorrow, Is Weekend, Is Weekday
Category: Validation Function Description: Checks if the data is a Date value which belongs to specific day of week Properties: Same as ‘Is Date’ 6.3.6 Is January, Is February, Is March, Is April, Is May, Is June, Is July, Is August, Is September, Is October, Is November, Is December, Is Current Month, Is Last Month, Is Next Month
Category: Validation Function Description: Checks if the data is a Date value which belongs to specific month of a year Properties: Same as ‘Is Date’
Copyright © 2009 DB Software Laboratory
Page 99 of 194
Advanced ETL Processor User Manual 6.3.7 Is 1st Quarter, Is 2nd Quarter, Is 3rd Quarter, Is 4th Quarter, Is Current Quarter, Is Last Quarter, Is Next Quarter
Category: Validation Function Description: Checks if the data is a Date value which belongs to specific quarter of a year Properties: Same as ‘Is Date’ 6.3.8 Is Leap Year, Is Current Year, Is Last Year, Is Next Year
Category: Validation Function Description: Checks if the data is a Date value which belongs to specific year Properties: Same as ‘Is Date’ 6.3.9 Is Current Week, Is Last Week, Is Next Week
Category: Validation Function Description: Checks if the data is a Date value which belongs to specific week Properties: Same as ‘Is Date’
Copyright © 2009 DB Software Laboratory
Page 100 of 194
Advanced ETL Processor User Manual 6.3.10 Is Within Past Minutes, Is Within Past Hours, Is Within Past Days, Is Within Past Weeks, Is Within Past Months
Category: Validation Function Description: Checks if the data is a Date value within in past minutes etc. Properties:
Copyright © 2009 DB Software Laboratory
Page 101 of 194
Advanced ETL Processor User Manual
6.4 Time
6.4.1 Is Time
Category: Validation Function Description: Checks if the data is a time value Properties:
Copyright © 2009 DB Software Laboratory
Page 102 of 194
Advanced ETL Processor User Manual 6.4.2 Is Time Between
Category: Validation Function Description: Checks if the data is a time between two values Properties:
Copyright © 2009 DB Software Laboratory
Page 103 of 194
Advanced ETL Processor User Manual
6.4.3 Is Time Less Than
Category: Validation Function Description: Checks if the data is a time less than value Properties:
6.4.4 Is Time More Than
Category: Validation Function Description: Checks if the data is a time more than value Properties: Same as ‘Is Time Less Than’
Copyright © 2009 DB Software Laboratory
Page 104 of 194
Advanced ETL Processor User Manual 6.4.5 Is Second
Category: Validation Function Description: Checks if data is a number between 0 and 59 Properties: Default 6.4.6 Is Minute
Category: Validation Function Description: Checks if data is a number between 0 and 59 Properties: Default 6.4.7 Is Hour 24
Category: Validation Function Description: Checks if data is a number between 0 and 23 Properties: Default 6.4.8 Is Hour 12
Category: Validation Function Description: Checks if data is a number between 0 and 12 Properties: Default
Copyright © 2009 DB Software Laboratory
Page 105 of 194
Advanced ETL Processor User Manual
6.4.9 Is PM
Category: Validation Function Description: Checks if the data is a PM Date Properties: Same as ‘Is Date’ 6.4.10 Is AM
Category: Validation Function Description: Checks if the data is a AM Date Properties: Same as ‘Is Date’
Copyright © 2009 DB Software Laboratory
Page 106 of 194
Advanced ETL Processor User Manual
6.5 Regular Expressions Regular expression is a string that is used to describe or match a set of strings, according to certain syntax rules. Regular expressions are used by many text editors, utilities, and programming languages to search and manipulate text based on patterns. For example, Perl and Tcl have a powerful regular expression engine built directly into their syntax. Several utilities provided by Unix distributions—including the editor ed and the filter grep— were the first to popularize the concept of regular expressions. "Regular expression" is often shortened to regex or regexp (singular), or regexes, regexps, or regexen (plural). Some authors distinguish between regular expression and abbreviated forms such as regex, restricting the former to true regular expressions, which describe regular languages, while using the latter for any regular expression-like pattern, including those that describe languages that are not regular. As only some authors observe this distinction, it is not safe to rely upon it. As an example of the syntax, the regular expression \bex can be used to search for all instances of the string "ex" that occur at word boundaries (signified by the \b). Thus in the string, "Texts for experts," \bex matches the "ex" in "experts," but not in "Texts" (because the "ex" occurs inside the word there and not immediately after a word boundary). Source: Wikipedia. For more information on regular expressions please visit: www.regularexpression.info Most of regular expressions were taken from www.regexlib.com. We are not able to include full list of contributors because it is too big. We did our best to test and modify if necessary pattern strings however as time goes by standards changes and some of them could became no longer valid. Therefore Regular Expression Validation Rules should be used with caution. If you have any useful Regular expressions please let us know we would be more than happy to include it into the next release of Advanced ETL Processor.
Copyright © 2009 DB Software Laboratory
Page 107 of 194
Advanced ETL Processor User Manual
6.5.1 Regular Expression
Category: Validation Function Description: Validates data against regular expression Properties:
Copyright © 2009 DB Software Laboratory
Page 108 of 194
Advanced ETL Processor User Manual
6.5.2 Is IP Address V4
Category: Validation Function Description: Validates IP addresses in the dotted quad form Properties: Default Pass: 192.168.1.1|||10.2.234.1|||66.129.71.122 Fail: 192.168.1|||10.2.1234.1|||66 129 71 122 6.5.3 Is IP Address V6
Category: Validation Function Description: Validates all IPv6 text representations as defined within RFC 2373 Properties: Default Pass: ::0:0:0:FFFF:129.144.52.38|||FEDC:BA98::3210:FEDC:BA98:7654:3210|||::13.1.68.3 Fail: FEDC:BA98:7654:3210:FEDC:BA98:7654:3210:1234|||3210:FEDC:BA98:7654:3210:1234|||: FEDC:BA98:7654:3210: 6.5.4 Is Email
Category: Validation Function Description: Email Address validation that allows both IP addresses and regular domains. In the case of an IP address it makes sure that it is no more than 255 for each part. Properties: Default Pass:
[email protected]|||
[email protected]|||
[email protected] Fail: john@doe|||
[email protected]|||@john.doe.com 6.5.5 Is ISBN 10
Category: Validation Function Description: Validation of 10 digits ISBN. The ISBN number must be preceded by the text "ISBN:" or "ISBN-10:", the colon is optional. Properties: Default Pass: ISBN-10: 0-93028-923-4 Fail: ISBN-13: 978-0-5960-0289-3
Copyright © 2009 DB Software Laboratory
Page 109 of 194
Advanced ETL Processor User Manual 6.5.6 Is ISBN 13
Category: Validation Function Description: Validation of new 13 digits ISBN. The ISBN number must be preceded by the text "ISBN:" or "ISBN-13:", the colon is optional. Properties: Default Pass: ISBN-13: 978-0-5960-0289-3 Fail: ISBN-10: 0596002890 6.5.7 Is Credit Card
Category: Validation Function Description: Validation of all major credit cards formats Properties: Default Pass: 341-1111-1111-1111|||5431-1111-1111-1111|||30569309025904 Fail: 30-5693-0902-5904|||5631-1111-1111-1111|||31169309025904 6.5.8 Is URL
Category: Validation Function Description: Validation of URL Properties: Default Pass: http://www.download.com Fail: http://999.999.999.999/dfd 6.5.9 Is UNC
Category: Validation Function Description: Validation of UNC Properties: Default Pass: \\server\c$ Fail: //server
Copyright © 2009 DB Software Laboratory
Page 110 of 194
Advanced ETL Processor User Manual
6.5.10 Is File Name
Category: Validation Function Description: Validation of file name format, file name must have a drive letter and an extension Properties: Default Pass: c:\1.txt Fail: ftc 6.5.11 Is UK Post Code
Category: Validation Function Description: Validates United Kingdom Post Code Properties: Default Pass: SW1W 8EU Fail: SW1WK8EU 6.5.12 Is UK VAT Number
Category: Validation Function Description: United Kingdom VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: GB123123412|||GBGD123|||123123412123 Fail: GB1231234124|||GBGD1234|||023 1234 12 123 6.5.13 Is UK Bank Sort Code
Category: Validation Function Description: Validation of United Kingdom Bank Sort code Properties: Default Pass: 09-01-29|||05-06-25 Fail: 90125
Copyright © 2009 DB Software Laboratory
Page 111 of 194
Advanced ETL Processor User Manual
6.5.14 Is UK Vehicle Registration Number
Category: Validation Function Description: Validation of United Kingdom Vehicle registration Number Properties: Default Pass: ABC 123 A|||A 123 ABC|||AB 12 ABC Fail: AB 123 C|||A 123 AB|||AB 12 AB 6.5.15 Is UK Driver Licence
Category: Validation Function Description: Validates the United Kingdom Drivers License format as described by the DVLA. See http://www.govtalk.gov.uk/gdsc/html/frames/default.htm. Matches: 1. Must be 16 characters 2. First 5 characters are alphanumeric. 3. Next 6 characters must be numeric 4. Next 3 characters are alphanumeric 5. Last 2 characters are alpha 6. Second character of numeric section can only be 0, 1, 5 or 6. 7. Fourth and fifth characters of numeric section must be in the range 01 to 31 Properties: Default Pass: JOHNS711215GG9SY Fail: JOHNS731215GG9SY 6.5.16 Is UK National Insurance Number
Category: Validation Function Description: Validation of United Kingdom National Insurance Number Properties: Default Pass: SP939393H|||PX123456D|||SW355667G Fail: 12SP9393H|||S3P93930D|||11223344SP00ddSS 6.5.17 Is UK Phone Number
Category: Validation Function Description: Validation of United Kingdom phone number, also allows +44 national code in addition to extension numbers of 3 or 4 digits Properties: Default Pass: +447222555555|||+44 7222 555 555|||(0722) 5555555 #2222 Copyright © 2009 DB Software Laboratory
Page 112 of 194
Advanced ETL Processor User Manual Fail: (+447222)555555|||+44(7222)555555|||(0722) 5555555 #22 6.5.18 Is UK Mobile Number
Category: Validation Function Description: United Kingdom mobile phone number validation with optional +44 national code, also allows optional brackets and spaces at appropriate positions Properties: Default Pass: 07222 555555|||(07222) 555555|||+44 7222 555 555 Fail: 7222 555555|||+44 07222 555555|||(+447222) 555555 6.5.19 Is US Zip Code
Category: Validation Function Description: Validates US Zip Code Properties: Default Pass: 48222|||48222-1746 Fail: 4632|||Blake|||37333-32 6.5.20 Is US State
Category: Validation Function Description: Validates US State 2 letter abbreviations Properties: Default Pass: CO|||GA|||TX Fail: A|||ZZ|||Florida 6.5.21 Is US Social Security Number
Category: Validation Function Description: Validates US Social security number Properties: Default Pass: 123-45-6789|||123 45 6789|||123456789 Fail: 12345-67-890123|||1234-56-7890|||123-45-78901
Copyright © 2009 DB Software Laboratory
Page 113 of 194
Advanced ETL Processor User Manual 6.5.22 Is US Phone Number
Category: Validation Function Description: US Phone Number Validation Properties: Default Pass: 5305551212|||(530) 555-1212|||530-555-1212 Fail: 0010011212|||1991991212|||123) not-good 6.5.23 Is Canadian Zip Code
Category: Validation Function Description: Canadian postal code format verification. The format of a Canadian postal code is LDL DLD where L are alpha characters and D are numeric digits. But there are some exceptions. The letters D, F, I, O, Q and U never appear in a postal code because of their Advanced similarity to 0, E, 1, 0, 0, and V respectively. In addition to avoiding the six "forbidden" letters W and Z also do not appear as the first letter of a postal code (at least not at present). Properties: Default Pass: M1R 4B0|||L0R 1B1|||L0R1B9 Fail: W1R 4B0|||L0R 1D1|||LOR1B9 6.5.24 Is Canadian Provincial Code
Category: Validation Function Description: Validates Canadian provincial codes Properties: Default Pass: ON|||PE|||NB Fail: OB|||NM 6.5.25 Is Canadian Branch-Transit/ Bank code
Category: Validation Function Description: Validates Canadian Branch-Transit number. The branch number must be 3 or 4 digits then '-' then five digits Properties: Default Pass: 654-45654|||5544-65486 Fail: 455645564|||44-55545|||55455-54565 Copyright © 2009 DB Software Laboratory
Page 114 of 194
Advanced ETL Processor User Manual 6.5.26 Is Canadian Social Insurance Number
Category: Validation Function Description: Validates Canadian Social Insurance Number Properties: Default Pass: 046 454 286|||123-456-789 Fail: 046FF 454 286|||123GG-456-789 6.5.27 Is Australian Post Code
Category: Validation Function Description: Australian postal code verification. Australia has 4-digit numeric postal codes with the following state based specific ranges. ACT: 0200-0299 and 2600-2639. NSW: 10001999, 2000-2599 and 2640-2914. NT: 0900-0999 and 0800-0899. QLD: 9000-9999 and 40004999. SA: 5000-5999. TAS: 7800-7999 and 7000-7499. VIC: 8000-8999 and 3000-3999. WA: 6800-6999 and 6000-6799 Properties: Default Pass: 0200|||7312|||2415 Fail: 0300|||7612|||2915 6.5.28 Is Australian Phone Number
Category: Validation Function Description: Validation of Australian phone number. Accepts all forms of Australian phone numbers in different formats (area code in brackets, no area code, spaces between 2-3 and 67th digits, +61 international dialling code). Checks that area codes are valid (when entered). Properties: Pass: 0403111111|||(03) 1111 1111|||+61403111111|||02 9111 1111|||0403 111 111|||91111111 Fail: 9111 11111|||99 1111 1111
Copyright © 2009 DB Software Laboratory
Page 115 of 194
Advanced ETL Processor User Manual
6.5.29 Is Australian Mobile Number
Category: Validation Function Description: Validates Australian Mobile Number Properties: Default Pass: 0412345678 Fail: 0299999999 6.5.30 Is Argentinean Post Code
Category: Validation Function Description: Validates Argentinean Post Code Properties: Default Pass: C1406HHA|||A4126AAB|||c1406hha Fail: c1406HHA|||4126|||C1406hha 6.5.31 Is Dutch Post Code
Category: Validation Function Description: Validates Dutch Postal Code Properties: Default Pass: 1234AB|||1234 AB|||1001 AB Fail: 0123AB|||1234A B|||0123 AB 6.5.32 Is Dutch Phone Number
Category: Validation Function Description: Validates Dutch Phone Number Properties: Default Pass: +31235256677|||+31(0)235256677|||023-5256677 Fail: +3123525667788999|||3123525667788|||232-2566778
Copyright © 2009 DB Software Laboratory
Page 116 of 194
Advanced ETL Processor User Manual
6.5.33 Is Dutch Car Number Plate
Category: Validation Function Description: Validates Dutch Car Number Plate Properties: Default Pass: ABC123|||12-AB-AB|||12-ABC-1 Fail: 12ABAB|||12ABC1|||1ABC12 6.5.34 Is Belgian Post Code
Category: Validation Function Description: Validates Belgian Post Code Properties: Default Pass: 1234 Fail: 123|||123A 6.5.35 Is Belgian VAT Number
Category: Validation Function Description: Belgian VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: BE123456789|||0123456789 Fail: BE12345678|||O123456789 6.5.36 Is French Post Code
Category: Validation Function Description: French Post Code Validation Properties: Default Pass: 12345|||F-12345|||F-2B100 Fail: F12345|||F-123456|||123456
Copyright © 2009 DB Software Laboratory
Page 117 of 194
Advanced ETL Processor User Manual
6.5.37 Is French VAT Number
Category: Validation Function Description: French VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: FRAB 123456789|||L7 123456789 Fail: FRAB123456789|||L7 L23456789 6.5.38 Is French Phone Number
Category: Validation Function Description: French Phone Number Validation Properties: Default Pass: 01 46 70 89 12|||01-46-70-89-12|||0146708912 Fail: 01-46708912|||01 46708912|||+33235256677 6.5.39 Is German Post Code
Category: Validation Function Description: Validates German Post Code format Properties: Default Pass: 12556|||01550|||80796 Fail: 05234|||8973|||62980 6.5.40 Is German VAT Number
Category: Validation Function Description: German VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: DE123456789|||224466880 Fail: DE12345678|||22446688B
Copyright © 2009 DB Software Laboratory
Page 118 of 194
Advanced ETL Processor User Manual 6.5.41 Is German Car License Plate
Category: Validation Function Description: German Car License Plate Validation Properties: Default Pass: F-RM373|||WÜ-Z1|||K- 07333|||HN-WK11H Fail: MOS-AS 6.5.43 Is German IBAN
Category: Validation Function Description: German International Bank Account Number Validation Properties: Default Pass: DE89 3704 0044 0532 0130 00|||DE89370400440532013000 Fail: DE89-3704-0044-0532-0130-00 6.5.44 Is Italian Post Code
Category: Validation Function Description: Postcode check for Italy (including possible Vatican/Italy indications) Properties: Default Pass: 1234|||V-1234 Fail: 12345 6.5.45 Is Italian VAT Number
Category: Validation Function Description: Italian VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: IT12345678901|||12345678901 Fail: IT1234567890|||1234567890I
Copyright © 2009 DB Software Laboratory
Page 119 of 194
Advanced ETL Processor User Manual
6.5.46 Is Italian IBAN
Category: Validation Function Description: Italian International Bank Account Number Validation Properties: Default Pass: IT28 W800 0000 2921 0064 5211 151|||IT28W8000000292100645211151 Fail: IT28-W800-0000-2921-0064-5211-151 6.5.47 Is Italian Phone Number
Category: Validation Function Description: Validates Italian phone number Properties: Default Pass: 02-343536|||02/343536|||02 343536 Fail: 02a343536|||02+343536 6.5.48 Is Italian Mobile Number
Category: Validation Function Description: Validates Italian mobile phone number with (or without) the International code Properties: Default Pass: +393471234567|||3381234567 Fail: +39 3401234567|||347 1234567|||338-1234567 6.5.49 Is Spanish Post Code
Category: Validation Function Description: Spanish Post Code Validation Properties: Default Pass: 12345|||10234|||01234 Fail: 00123
Copyright © 2009 DB Software Laboratory
Page 120 of 194
Advanced ETL Processor User Manual
6.5.50 Is Spanish VAT Number
Category: Validation Function Description: Spanish VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: ES01234567A|||A12345678 Fail: ES012345678|||AB2345678 6.5.51 Is Spanish IBAN
Category: Validation Function Description: Spanish IBAN Validation Properties: Default Pass: ES91 2100 0418 4502 0005 1332|||ES9121000418450200051332 Fail: ES91-2100-0418-4502-0005-1332 6.5.52 Is Spanish Phone Number
Category: Validation Function Description: Spanish Phone Number Validation Properties: Default Pass: 972367087|||97-3770866|||972-377086 Fail: 9988-989898|||989898988989|||9 9898989898 6.5.53 Is Swedish Post Code
Category: Validation Function Description: Validates Swedish Post Code Format Properties: Default Pass: 12345|||932 68|||S-621 46 Fail: 5367|||425611|||31 545 6.5.54 Is Swedish VAT Number
Copyright © 2009 DB Software Laboratory
Page 121 of 194
Advanced ETL Processor User Manual
Category: Validation Function Description: Swedish VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: SE123456789012|||123456789012 Fail: SE12345678901|||123456789O12 6.5.55 Is Swedish IBAN
Category: Validation Function Description: Swedish International Bank Account Number IBAN Properties: Default Pass: SE35 5000 0000 0549 1000 0003|||SE3550000000054910000003 Fail: SE35-5000-0000-0549-1000-0003 6.5.56 Is Swedish Phone Number
Category: Validation Function Description: Validates Swedish Phone Number Properties: Default Pass: +46 8 123 456 78|||08-123 456 78|||0123-456 78 Fail: +46 08-123 456 78|||08 123 456 78|||0123 456 78 6.5.57 Is Swedish Person Number
Category: Validation Function Description: Validates Swedish Person Number Properties: Default Pass: 19740609-7845|||19811116-7845|||20010913-7598 Fail: 21003612-9999|||18790505-4545|||19740641-5559
Copyright © 2009 DB Software Laboratory
Page 122 of 194
Advanced ETL Processor User Manual
6.5.58 Is Austrian Mobile Phone Number
Category: Validation Function Description: Validates Austrian Mobile Phone Number Properties: Default Pass: 0664-1234567|||06641234567|||0699-11234567 Fail: 06991-1234567|||+43650-1234567|||07661234567 6.5.59 Is Danish Post Code
Category: Validation Function Description: Validates Danish Post Code Properties: Default Pass: DK-1234|||dk 1234|||Dk-1234 Fail: DK 12345 6.5.60 Is Danish VAT Number
Category: Validation Function Description: Validates Danish VAT Numbers format with support for optional member state definition. Properties: Default Pass: DK11 22 33 44 Fail: DK11223344 6.5.61 Is Danish Phone Number
Category: Validation Function Description: Validates Danish Phone Number Properties: Default Pass: (+45) 35 35 35 35 ||| +45 35 35 35 35 ||| 35 35 35 35 ||| 35353535 Fail: (45)35353535 ||| 4535353535
Copyright © 2009 DB Software Laboratory
Page 123 of 194
Advanced ETL Processor User Manual 6.5.62 Is Danish Civil Registration Number
Category: Validation Function Description: Validates Danish civil registration number, which consists of 10 numeric characters and one special character '-'. The format is DDMMYY-XXXX Properties: Default Pass: 010380-2343 || 120454-5467 || 390178-3434 Fail: 1223876776 || 4023100923 6.5.63 Is Finnish VAT Number
Category: Validation Function Description: Finnish VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: FI12345678|||22446688 Fail: FI1234567|||224466BB 6.5.64 Is Greek VAT Number
Category: Validation Function Description: Greek VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: EL123456789|||224466880 Fail: EL12345678|||22446688B 6.5.65 Is Irish VAT Number
Category: Validation Function Description: Irish VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: IE4*12345Z|||0012345K Fail: IE4-12345Z|||00123456
Copyright © 2009 DB Software Laboratory
Page 124 of 194
Advanced ETL Processor User Manual 6.5.66 Is Luxembourg VAT Number
Category: Validation Function Description: Luxemburg VAT Numbers format verification with support for optional member state definition Properties: Default Pass: LU12345678|||22446688 Fail: LU1234567|||224466BB 6.5.67 Is Portuguese VAT Number
Category: Validation Function Description: Portuguese VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: PT123456789|||224466880 Fail: PT12345678|||22446688B 6.5.68 Is Estonian Post Code
Category: Validation Function Description: Validates Estonian Post Codes Properties: Default Pass: 00000|||99999 Fail: 00 000 6.5.69 Is Estonian VAT Number
Category: Validation Function Description: Estonian VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: EE123456789|||224466880 Fail: EE12345678|||22446688B
Copyright © 2009 DB Software Laboratory
Page 125 of 194
Advanced ETL Processor User Manual 6.5.70 Is Latvian VAT Number
Category: Validation Function Description: Validates Latvian VAT Number Properties: Default Pass: LV12345678901|||12345678901 Fail: IT1234567890|||1234567890I 6.5.71 Is Lithuanian VAT Number
Category: Validation Function Description: Lithuanian VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: LT123456789|||123456789012 Fail: LX123456789|||123466789O12 6.5.72 Is Slovenian VAT Number
Category: Validation Function Description: Slovenian VAT Numbers format verification with support for optional member state definition Properties: Default Pass: SI12345678|||22446688 Fail: SI1234567|||224466BB 6.5.73 Is Slovakian VAT Number
Category: Validation Function Description: Slovakian VAT Numbers format verification with support for optional member state definition Properties: Default Pass: SK1234567890|||1234567890 Fail: SK1234567890|||1234567890
Copyright © 2009 DB Software Laboratory
Page 126 of 194
Advanced ETL Processor User Manual 6.5.74 Is Slovakian IBAN
Category: Validation Function Description: Validates Slovakian International Bank Account Number Properties: Default Pass: SK31 1200 0000 1987 4263 7541|||SK3112000000198742637541 Fail: SK31-1200-0000-1987-4263-7541 6.5.75 Is Polish VAT Number
Category: Validation Function Description: Polish VAT Numbers format verification with support for optional member state definition Properties: Default Pass: PL1234567890|||1234567890 Fail: PL123456789|||123456789O 6.5.76 Is Polish IBAN
Category: Validation Function Description: Validates Polish International Bank Account Number Properties: Default Pass: PL27 1140 2004 0000 3002 0135 5387|||PL27114020040000300201355387 Fail: PL27-1140-2004-0000-3002-0135-5387 6.5.77 Is Czech Republic VAT Number
Category: Validation Function Description: Czech Republic VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: CZ12345678|||1234567890 Fail: CZ1234567|||12345678901
Copyright © 2009 DB Software Laboratory
Page 127 of 194
Advanced ETL Processor User Manual 6.5.78 Is Czech Republic IBAN
Category: Validation Function Description: Validates Czech International Bank Account Number Properties: Default Pass: CZ65 0800 0000 1920 0014 5399|||CZ6508000000192000145399 Fail: CZ65-0800-0000-1920-0014-5399 6.5.79 Is Netherlands VAT Number
Category: Validation Function Description: The Netherlands VAT Numbers format verification with support for optional member state definition. Properties: Default Pass: NL123456789B12 Fail: NL123456789012 6.5.80 Is Hungarian VAT Number
Category: Validation Function Description: Hungarian VAT Numbers format verification with support for optional member state definition Properties: Default Pass: HU12345678|||22446688 Fail: HU1234567|||224466BB 6.5.81 Is Maltese VAT Number
Category: Validation Function Description: Maltese VAT Numbers format verification with support for optional member state definition Properties: Default Pass: MT12345678|||22446688 Fail: MT1234567|||224466BB
Copyright © 2009 DB Software Laboratory
Page 128 of 194
Advanced ETL Processor User Manual 6.5.82 Is Algerian Post Code
Category: Validation Function Description: Validates Algerian Post Code Format Properties: Default Pass: 12345|||99999|||00000 Fail: 1|||1234|||999999 6.5.83 Is Andorran Post Code
Category: Validation Function Description: Validates Andorran Post Code format Properties: Default Pass: AB 123|||AB123 Fail: 12 abc|||12345 6.5.84 Is Andorran IBAN
Category: Validation Function Description: Validates Andorran International Bank Account Number Properties: Default Pass: AD12 0001 2030 2003 5910 0100|||AD1200012030200359100100 Fail: AD12-0001-2030-2003-5910-0100 6.5.85 Is Azerbaijani Post Code
Category: Validation Function Description: Validates Azerbaijani Post Code format Properties: Default Pass: AZ 1234|||AZ1234 Fail: 12 abcd|||AB 1234
Copyright © 2009 DB Software Laboratory
Page 129 of 194
Advanced ETL Processor User Manual
6.5.86 Is Bahraini Post Code
Category: Validation Function Description: Validates Bahraini Post Code format Properties: Default Pass: 123|||1299|||199 Fail: 1300|||000|||99 6.5.87 Is Bangladeshi Post Code
Category: Validation Function Description: Validates Bangladeshi Post Code format Properties: Default Pass: 1000|||9999|||1234 Fail: 123456|||0123|||99 6.5.88 Is Belarusian Post Code
Category: Validation Function Description: Validates Belarusian Post Code format Properties: Default Pass: 123456|||000000|||999999 Fail: 0|||9|||1234 6.5.89 Is Brunei Post Code
Category: Validation Function Description: Validates Brunei Post Code format Properties: Default Pass: BT2328|||KA1234 Fail: AB1234|||AB 1234
Copyright © 2009 DB Software Laboratory
Page 130 of 194
Advanced ETL Processor User Manual
6.5.90 Is Chilean Post Code
Category: Validation Function Description: Validates Chilean Post Code format Properties: Default Pass: 872-0019|||000-0000|||999-9999 Fail: 000 0000|||000000 6.5.91 Is Croatian Post Code
Category: Validation Function Description: Validates Croatian Post Code format Properties: Default Pass: HR-00000|||HR-99999 Fail: HR 00000|||00000 6.5.92 Is Cyprus Post Code
Category: Validation Function Description: Validates Cyprus Post Code format Properties: Default Pass: 1000|||9999|||0000 Fail: 1|||99999|||99 000 6.5.93 Is Cyprus VAT Number
Category: Validation Function Description: Cyprus VAT Numbers format verification with support for optional member state definition Properties: Default Pass: CY12345678A|||12345678A Fail: CY1234567A|||123456789
Copyright © 2009 DB Software Laboratory
Page 131 of 194
Advanced ETL Processor User Manual 6.5.94 Is Egyptian Post Code
Category: Validation Function Description: Validates Egyptian Post Code format Properties: Default Pass: 00000|||99999 Fail: 0 0 0 00|||00 6.5.95 Is Ethiopian Post Code
Category: Validation Function Description: Validates Ethiopian Post Code format Properties: Default Pass: 1000|||9999 Fail: 1 999 6.5.96 Is Faroe Islands Post Code
Category: Validation Function Description: Validates Faroe Islands Post Code format Properties: Default Pass: FO100|||FO000|||FO 000 Fail: AB 123|||AB123 6.5.97 Is French Polynesia Post Code
Category: Validation Function Description: Validates French Polynesia Post Code format Properties: Default Pass: 00000|||99999 Fail: 00 000|||99 9 9 9
Copyright © 2009 DB Software Laboratory
Page 132 of 194
Advanced ETL Processor User Manual
6.5.98 Is Guadeloupe Post Code
Category: Validation Function Description: Validates Guadeloupe Post Code format Properties: Default Pass: 97100|||98099 Fail: 12345|||97 100 6.5.99 Is Guinean Post Code
Category: Validation Function Description: Validates Guinean Post Code format Properties: Default Pass: 001 BP 123|||499 BP 999 Fail: 001BP999|||999 BP 000 6.5.100 Is Guinea-Bissau Post Code
Category: Validation Function Description: Validates Guinea-Bissau Post Code format Properties: Default Pass: 1000|||1999|||9999 Fail: 0000|||0123 6.5.101 Is Russian Post Code
Category: Validation Function Description: Validates Russian Post Code Properties: Default Pass: 150016 Fail: wert12
Copyright © 2009 DB Software Laboratory
Page 133 of 194
Advanced ETL Processor User Manual
6.5.102 Is Russian Phone Number
Category: Validation Function Description: Validates Russian Phone Number Properties: Default Pass: +7(916)9985670|||8-912-268-5440|||8905148-3339|||8(913)448-51-90|||903-345-34-34||| 903-34-334-34|||903-34-33434 Fail: +33903-34-33434 6.5.103 Is Israeli Phone Number
Category: Validation Function Description: Validates Israeli Phone Number Format Properties: Default Pass: +972-4-8253055|||972-4-8615466|||972-04-8564521 Fail: 04-5487665|||05-455665545|||456884624 6.5.104 Is Israeli Mobile Phone Number
Category: Validation Function Description: Validates Israeli Mobile Phone Number Properties: Default Pass: 050-1234567, 0501234567, 501234567 Fail: 0601234567, 5412345678 6.5.105 Is Peru Phone Number
Category: Validation Function Description: Validates Peru Phone Number format Properties: Default Pass: 263-8854|||5660578 1235|||622-4588 21 Fail: 1227585|||4224586 50124|||0554499
Copyright © 2009 DB Software Laboratory
Page 134 of 194
Advanced ETL Processor User Manual
6.5.106 Is Peru Cellular Number
Category: Validation Function Description: Validates Peru Cellular Number format Properties: Default Pass: 8874-2544|||99106800|||84509955 Fail: 6540-9985|||77329390|||725-2763 6.5.107 Is Indian Post Code
Category: Validation Function Description: Validates Indian Post Code Properties: Default Pass: 400 099|||400099|||400050 Fail: 2345678|||12345|||asdf 6.5.108 Is Indian Phone Number
Category: Validation Function Description: Validates Indian Phone Number Properties: Default Pass: 0493 - 3227341|||0493 3227341|||493 3227341 Fail: 93 0227341|||493 322734111|||493 -- 3227341 6.5.109 Is Indian Mobile Number
Category: Validation Function Description: Validates Indian Mobile Phone Number Properties: Default Pass: +919847444225|||+91-98-44111112|||98 44111116 Fail: +919800444225|||+947444225|||44111116
Copyright © 2009 DB Software Laboratory
Page 135 of 194
Advanced ETL Processor User Manual
6.5.110 Is Indian Vehicle Registration Number
Category: Validation Function Description: Validates Indian Vehicle Registration Number Properties: Default Pass: mh 12 bj 1780|||mmx 1234 Fail: mmm 123 1234|||mm 12 bj 345|||mm 12345 6.5.111 Is South African Phone Number
Category: Validation Function Description: Validates South African Phone Number Properties: Default Pass: 0333872119|||033 386162|||074 101 2850 Fail: 033386 2612 6.5.112 Is South African Mobile Number
Category: Validation Function Description: only allows valid cell phones from South African service providers Properties: Default Pass: 0848298477|||084 829 8477|||084-829 8477 Fail: 084829 8477 6.5.113 Is South African VAT Number
Category: Validation Function Description: Validates South African VAT Number Properties: Default Pass: 2001/067995/23|||1989/064445/73 Fail: LX123456789|||123466789O12
Copyright © 2009 DB Software Laboratory
Page 136 of 194
Advanced ETL Processor User Manual
6.5.114 Is US or CA Zip code
Category: Validation Function Description: Validates US or CA Zip code format Properties: Default Pass: 00501|||84118-3423|||n3a 3B7 Fail: 501-342|||123324|||Q4B 5C5
Copyright © 2009 DB Software Laboratory
Page 137 of 194
Advanced ETL Processor User Manual
7. Transformation Functions )ote: To change Transformation function properties double click on it 7.1 Strings 7.1.1 Upper Case
Category: Transformation Function Description: Converts all characters of a string to Upper Case My String => MYSTRING Properties: None 7.1.2 Lower Case
Category: Transformation Function Description: Converts all characters of a string to Lower Case My String => my string Properties: None 7.1.3 Proper Case
Category: Transformation Function Description: Converts first character of all words within a string to Upper Case and the rest to the lower case my strinG => My String Properties: None 7.1.4 First Up
Category: Transformation Function Description: Converts first character a string to Upper Case and the rest to the lower case my strinG => My string Properties: None
Copyright © 2009 DB Software Laboratory
Page 138 of 194
Advanced ETL Processor User Manual 7.1.5 Delete Spaces
Category: Transformation Function Description: Makes sure that there is only one space between words My string => My string Properties: None 7.1.6 Trim
Category: Transformation Function Description: Removes Leading and Trailing Spaces from a string Properties: None 7.1.7 Left Trim
Category: Transformation Function Description: Removes Leading Spaces from a string Properties: None 7.1.8 Right Trim
Category: Transformation Function Description: Removes Trailing Spaces from a string Properties: None
Copyright © 2009 DB Software Laboratory
Page 139 of 194
Advanced ETL Processor User Manual
7.1.9 Sub String
Category: Transformation Function Description: Returns part of a string. Properties:
Copyright © 2009 DB Software Laboratory
Page 140 of 194
Advanced ETL Processor User Manual
7.1.10 Replace
Category: Transformation Function Description: Replaces a one or more characters from the data with zero or more new characters. Properties:
Copyright © 2009 DB Software Laboratory
Page 141 of 194
Advanced ETL Processor User Manual
7.1.11 Ensure Prefix
Category: Transformation Function Description: Check string for prefix and if it is missing adds it Properties:
7.1.12 Ensure Suffix
Category: Transformation Function Description: Check string for suffix and if it is missing adds it Properties: Same as ‘Ensure Prefix’ 7.1.13 Ensure No Prefix
Category: Transformation Function Description: Check string for prefix and if it is present removes it Properties: Same as ‘Ensure Prefix’
Copyright © 2009 DB Software Laboratory
Page 142 of 194
Advanced ETL Processor User Manual 7.1.14 Ensure No Suffix
Category: Transformation Function Description: Check string for suffix and if it is present removes it Properties: Same as ‘Ensure Prefix’ 7.1.15 Left Pad
Category: Transformation Function Description: Returns the string left-padded to the length with spaces or specified character. Properties:
7.1.16 Right Pad
Category: Transformation Function Description: Returns the string right-padded to the length with spaces or specified character. Properties: Same as ‘Right Pad’
Copyright © 2009 DB Software Laboratory
Page 143 of 194
Advanced ETL Processor User Manual 7.1.17 Escape String
Category: Transformation Function Description: Replaces special characters such as TAB, CR, LF,NULL,BEL,BS,ESC,HT,FF,VT with there C-Style escape equivalents. Properties: None 7.1.18 UnEscape String
Category: Transformation Function Description: Opposite to Escape String, Replaces C-Style escape characters with special characters such as TAB, CR, LF,NULL,BEL,BS,ESC,HT,FF,VT. Properties: None 7.1.19 Delete
Category: Transformation Function Description: Deletes part of a string. Properties:
Copyright © 2009 DB Software Laboratory
Page 144 of 194
Advanced ETL Processor User Manual 7.1.20 Left
Category: Transformation Function Description: Returns left part of a string. Properties:
7.1.21 Right
Category: Transformation Function Description: Returns right part of a string Properties: Same as ‘Left’ 7.1.22 Left Delete
Category: Transformation Function Description: Deletes left part of a string Properties: Same as ‘Left’
Copyright © 2009 DB Software Laboratory
Page 145 of 194
Advanced ETL Processor User Manual 7.1.23 Right Delete
Category: Transformation Function Description: Deletes right part of a string Properties: Same as ‘Left’
Copyright © 2009 DB Software Laboratory
Page 146 of 194
Advanced ETL Processor User Manual
7.2 Numbers
7.2.1 Round
Category: Transformation Function Description: Rounding is the process of reducing the number of significant digits in a number. The result of rounding is a "shorter" number having fewer non-zero digits yet similar in magnitude. The result is less precise but easier to use. For example: 73 rounded to the nearest ten is 70, because 73 is closer to 70 than to 80. Properties:
Copyright © 2009 DB Software Laboratory
Page 147 of 194
Advanced ETL Processor User Manual
7.2.3 Abs
Category: Transformation Function Description: Returns absolute value of a number. Absolute value is always positive. For example -2 will give 2 Properties:
7.2.4 Sign
Category: Transformation Function Description: Returns Sign of numeric value -1 for negative values, 1 for positive values and 0 for 0 Properties: Same as ‘Abs’
Copyright © 2009 DB Software Laboratory
Page 148 of 194
Advanced ETL Processor User Manual
7.3 Date 7.3.1 Date Format
Category: Transformation Function Description: Returns Date in ‘yyyy-mm-dd hh:nn:ss.zzz' format. In order to load Date fields into target database they should be converted into ‘yyyy-mm-dd hh:nn:ss.zzz' format Properties:
7.3.2 Current Date
Category: Transformation Function Description: Returns Current Date in ‘yyyy-mm-dd hh:nn:ss.zzz' format Properties: None
Copyright © 2009 DB Software Laboratory
Page 149 of 194
Advanced ETL Processor User Manual
7.4 Miscellaneous
7.4.1 Length
Category: Transformation Function Description: Returns Length of String Properties: None 7.4.2 Literal
Category: Transformation Function Description: Populates field with a constant value Properties:
Copyright © 2009 DB Software Laboratory
Page 150 of 194
Advanced ETL Processor User Manual
7.4.3 User
Category: Transformation Function Description: Returns Current Windows User Name Properties: None 7.4.4 Splitter
Category: Transformation Function Description: Splits data into number of values Properties:
Copyright © 2009 DB Software Laboratory
Page 151 of 194
Advanced ETL Processor User Manual
7.4.5 Joiner
Category: Transformation Function Description: Joins several values into one value delimited Delimiter and surrounded by Qualifier if specified Properties:
Copyright © 2009 DB Software Laboratory
Page 152 of 194
Advanced ETL Processor User Manual
7.4.6 Calculation
Category: Transformation Function Description: Allows end user to write scripts to transform data. Input values represented as [F00N] Where N is values number. For example calculation you can see above has got two input values [F001] and [F002]. Programming language used is very close to Pascal. The Structure of a Calculation Below is the basic structure that every Calculation transformation must follow: Var VariableName : VariableType; VariableName : VariableType; ... // Some single line Comment if necessary { Multi line Comment } Procedure ProcedureName; variables here if necessary Begin Some Code; End; Function FunctionName(variableList): VariableType; variables here if necessary Begin Some Code if necessary; Result := some expression More Code if necessary; End; ... some more functions and procedures if necessary ... Begin the main program block. Result:= some calculation End. )ote: Copyright © 2009 DB Software Laboratory
Page 153 of 194
Advanced ETL Processor User Manual The functions and procedures can appear in any order. The only requirement is that if one procedure or function uses another one, that latter one must have been defined already. Declaring Variables var // This starts a section of variables LineTotal : Integer; // This defines an Integer variable called LineTotal First,Second : String; // This defines two variables to hold strings of text Assigning Values to Variables Variables are simply a name for a block of memory cells in main memory. If a value is assigned to a variable, that value must be of the same type as the variable, and will be stored in the memory address designated by the variable name. The assignment statement is the semicolon-equal :=. • • •
Variables must be declared at the beginning of the program, a procedure, or a function Variables must be initialized before they can be used. Variables can be reused as often as necessary. Their old value is simply overwritten by a new assignment.
Example: Var i : Integer: Begin i := 10; End.
{ variable name is i, type is integer) { valid integer number assigned to variable i }
Copyright © 2009 DB Software Laboratory
Page 154 of 194
Advanced ETL Processor User Manual
Variable Types )umeric Data Types Type Byte ShortInt Word SmallInt LongWord Cardinal LongInt Integer Int64 Single Currency Double Extended
Storage size 1 1 2 2 4 4* 4 4* 8 4 8 8 10
Range 0 to 255 -127 to 127 0 to 65,535 -32,768 to 32,767 0 to 4,294,967,295 0 to 4,294,967,295 -2,147,483,648 to 2,147,483,647 -2,147,483,648 to 2,147,483,647 -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 7 significant digits, exponent -38 to +38 50+ significant digits, fixed 4 decimal places 15 significant digits, exponent -308 to +308 19 significant digits, exponent -4932 to +4932
Assigning to and from number variables Number variables can be assigned from other numeric variables, and expressions: var Age : Byte; // Smallest positive integer type Books : SmallInt; // Bigger signed integer Salary : Currency; // Decimal used to hold financial amounts Expenses : Currency; TakeHome : Currency; begin Expenses := 12345.67; // Assign from a literal constant TakeHome := Salary; // Assign from another variable TakeHome := TakeHome - Expenses; // Assign from an expression end; )umerical operators Number calculations, or expressions, have a number of primitive operators available: + * / div mod
Add one number to another Subtract one number from another Multiply two numbers Divide one decimal number by another Divide one integer number by another Remainder from dividing one integer by another
Copyright © 2009 DB Software Laboratory
Page 155 of 194
Advanced ETL Processor User Manual When using these multiple operators in one expression, you should use round brackets to wrap around sub-expressions to ensure that the result is obtained. This is illustrated in the examples below: var myInt : Integer; // Define integer and decimal variables myDec : Single; begin myInt := 20; // myInt is now 20 myInt := myInt + 10; // myInt is now 30 myInt := myInt - 5; // myInt is now 25 myInt := myInt * 4; // myInt is now 100 myInt := 14 div 3; // myInt is now 4 (14 / 3 = 4 remainder 2) myInt := 14 mod 3; // myInt is now 2 (14 / 3 = 4 remainder 2) myInt := 12 * 3 - 4; // myInt is now 32 (* comes before -) myInt := 12 * (3 - 4); // myInt is now -12 (brackets come before *) myDec := 2.222 / 2.0; // myDec is now 1.111 end; Character Types var Str1 : Char; Str2 : WideChar; Str3 : AnsiChar; Str4 : ShortString; Str5 : String; Str6 : AnsiString; Str7 : WideString;
// Holds a single character, small alphabet // Holds a single character, International alphabet // Holds a single character, small alphabet // Holds a string of up to 255 Char's // Holds strings of Char's of any size desired // Holds strings of AnsiChar's any size desired // Holds strings of WideChar's of any size desired
Some simple text variable usage examples are given below: + = < >=
Concatenates two strings together Compares for string equality Is one string lower in sequence than another Is one string lower or equal in sequence with another Is one string greater in sequence than another Is one string greater or equal in sequence with another Compares for string inequality
Copyright © 2009 DB Software Laboratory
Page 156 of 194
Advanced ETL Processor User Manual Variants The Variant data type provides a flexible general purpose data type. It can hold anything but structured data and pointers. Variants are useful in very specific circumstances, where data types and their content are determined at run time rather than at compile time. Example: var myVar : Variant; Date Variables TDateTime Description The TDateTime type holds a date and time value. It is stored as a Double variable, with the date as the integral part, and time as fractional part. The date is stored as the number of days since 30 Dec 1899. Quite why it is not 31 Dec is not clear. 01 Jan 1900 has a days value of 2. Because TDateTime is actually a double, you can perform calculations on it as if it were a number. This is useful for calculations such as the difference between two dates. )ote: No local time information is held with TDateTime - just the day and time values. Example: Finding the difference between two dates var day1, day2 : TDateTime; diff : Double; begin day1 := StrToDate('12/06/2002'); day2 := StrToDate('12/07/2002'); diff := day2 - day1; Result:='day2 - day1 = '+FloatToStr(diff)+' days'; end; day2 - day1 = 30 days
Logical data types Copyright © 2009 DB Software Laboratory
Page 157 of 194
Advanced ETL Processor User Manual These are used in conjunction with programming logic. They are very simple: Var Log1 : Boolean;
// Can be 'True' or 'False'
Boolean variables are a form of enumerated type. This means that they can hold one of a fixed number of values, designated by name. Here, the values can be True or False. Logical Operation Simple if then else Here is an example of how the if statement works: var number : Integer; text : String; begin number := Sqr(17); // Calculate the square of 17 if number > 400 then text := '17 squared > 400' // Action when if condition is true else text := '17 squared 400'; Note that here, the then condition is not executed (because 17 squared is not > 400), but there is no else clause. This means that the if statement simply finishes without doing anything. Note also that the then clause now has a terminating ; to signify the end of the if statement. Compound if conditions, and multiple statements We can have multiple conditions for the if condition. And we can have more than one statement for the then and else clauses. Here are some examples: If (condition1) And (condition2) // Both conditions must be satisfied Copyright © 2009 DB Software Laboratory
Page 158 of 194
Advanced ETL Processor User Manual then begin statement1; statement2; ... end // Notice no terminating ';' - still part of 'if' else begin statement3; statement4; ... end; We used And to join the if conditions together - both must be satisfied for the then clause to execute. Otherwise, the else clause will execute. We could have used a number of different logical primitives, of which And is one, covered under logical primitives below.
)ested if statements: There is nothing to stop you using if statements as the statement of an if statement. Nesting can be useful, and is often used like this: if condition1 then statement1 else if condition2 then statement2 else statement3;
However, too many nested if statements can make the code confusing. The Case statement, discussed below, can be used to overcome a lot of these problems. Logical primitives Before we introduce these, it is appropriate to introduce the Boolean data type. It is an enumerated type that can have one of only two values: True or False. We will use it in place of a condition in the if clauses below to clarify how they work: begin if false And false then Result:='false and false = true'; if true And false then Result:= 'true and false = true'; if false And true then Result:= 'false and true = true'; if true And true Copyright © 2009 DB Software Laboratory
Page 159 of 194
Advanced ETL Processor User Manual then Result:= 'true and true = true'; if false Or false then Result:= 'false or false = true'; if true Or false then Result:= 'true or false = true'; if false Or true then Result:= 'false or true = true'; if true Or true then Result:= 'true or true = true'; if false Xor false then Result:= 'false xor false = true'; if true Xor false then Result:= 'true xor false = true'; if false Xor true then Result:= 'false xor true = true'; if true Xor true then Result:= 'true xor true = true'; if Not false then Result:= 'not false = true'; if Not true then Result:= 'not true = true'; end; true and true = true false or true = true true or false = true true or true = true false xor true = true true xor false = true not false = true Note that the Xor primitive returns true when one, but not both of the conditions are true.
Case statements The If statement is useful when you have a simple two way decision. Ether you go one way or another way. Case statements are used when you have a set of 3 or more alternatives. A simple numerical case statement: Copyright © 2009 DB Software Laboratory
Page 160 of 194
Advanced ETL Processor User Manual
var i : Integer; begin i := [F1]; Case i of 15 : Resut := ('Number was fifteen'); 16 : Resut := ('Number was sixteen'); 17 : Resut := ('Number was seventeen'); 18 : Resut := ('Number was eighteen'); 19 : Resut := ('Number was nineteen'); 20 : Resut := ('Number was twenty'); end; end; Number was fifteen The case statement above routes the processing to just one of the statements. OK, the code is a bit silly, but it is used to illustrate the point.
Using the otherwise clause Supposing we were not entirely sure what value our case statement was processing? Or we wanted to cover a known set of values in one fell swoop? The Else clause allows us to do that: var i : Integer; begin i := [F1]; Case i of 15 : Resut := ‘Number was fifteen'; 16 : Resut := 'Number was sixteen'; 17 : Resut := 'Number was seventeen'; 18 : Resut := 'Number was eighteen'; 19 : Resut := 'Number was nineteen'; 20 : Resut := 'Nuumber was twenty'; else Resut := 'Unexpected number‘; end; end; Unexpected number : 10 Repeating sets of commands Why loops are used in programming
Copyright © 2009 DB Software Laboratory
Page 161 of 194
Advanced ETL Processor User Manual One of the main reasons for using computers is to save the tedium of many repetitive tasks. One of the main uses of loops in programs is to carrry out such repetitive tasks. A loop will execute one or more lines of code (statements) as many times as you want. Your choice of loop type depends on how you want to control and terminate the looping. The For loop This is the most common loop type. For loops are executed a fixed number of times, determined by a count. They terminate when the count is exhausted. The count (loop) is held in a variable that can be used in the loop. The count can proceed upwards or downwards, but always does so by a value of 1 unit. This count variable can be a number or even an enumeration. Counting up Here is a simple example counting up using numeric values: var count : Integer; begin For count := 1 to 5 do Result:= 'Count is now '+IntToStr(count); end; Counting down Here is a simple example counting up using numeric values: var count : Integer; begin For count := 5 downto 1 do Result:= 'Count is now '+IntToStr(count); end; The For statements in the examples above have all executed one statement. If you want to execute more than one, you must enclose these in a Begin and End pair.
Copyright © 2009 DB Software Laboratory
Page 162 of 194
Advanced ETL Processor User Manual The Repeat loop The Repeat loop type is used for loops where we do not know in advance how many times we will execute. For example, when we keep asking a user for a value until one is provided, or the user aborts. Here, we are more concerned with the loop termination condition. Repeat loops always execute at least once. At the end, the Until condition is checked, and the loop aborts of condition works out as true. A simple example var stop : Boolean; // Our exit condition flag i : Integer; begin i := 1; exit := False; // do not exit until we are ready repeat i := i+1; // Increment a count if Sqr(i) > 99 then stop:= true; // Exit if the square of our number exceeds 99 until stop; // Shorthand for 'until exit := true' result:=I; end; Upon exit, i will be 10 (since Sqr(10) > 99) Here we exit the repeat loop when a Boolean variable is true. Notice that we use a shorthand - just specifying the variable as the condition is sufficient since the variable value is either true or false. Using a compound condition var i : Integer; begin i := 1; repeat i := i+1; // Increment a count until (Sqr(i) > 99) or (Sqrt(i) > 2.5); result:=i; end; Upon exit, i will be 7 (since Sqrt(7) > 2.5) Notice that compound statements require separating brackets. Notice also that Repeat statements can accomodate multiple statements without the need for a begin/end pair. The repeat and until clauses form a natural pairing. While loops Copyright © 2009 DB Software Laboratory
Page 163 of 194
Advanced ETL Processor User Manual While loops are very similar to Repeat loops except that they have the exit condition at the start. This means that we use them when we wish to avoid loop execution altogether if the condition for exit is satisfied at the start. Var i : Integer; begin i := 1; while (Sqr(i) S2
>0
S1 < S2
S2, and returns 0 if S1 = S2. AnsiStrLIComp (S1,S2,MaxLen) AnsiStrLIComp compares S1 to S2, without case sensitivity. If S1 or S2 is longer than MaxLen characters, AnsiStrLIComp only compares up to the first MaxLen characters. The return value is: Condition
Return Value
S1 > S2
>0
S1 < S2
