Actually this book is targeted for DB2 but the reader can have good reference for other db technologies like Oracle, MyS...
DB2 LUW V9.7 SQL Cookbook Graeme Birchall 14-Jan-2011
Graeme Birchall ©
2
DB2 V9.7 Cookbook ©
Preface Important!
If you didn't get this document directly from my personal website, you may have got an older edition. The book is changed very frequently, so if you want the latest, go to the source. Also, the latest edition is usually the best book to have, as the examples are often much better. This is true even if you are using an older version of DB2. This Cookbook is written for DB2 for LUW (i.e. Linux, Unix, Windows). It is not suitable for DB2 for z/OS unless you are running DB2 8 in new-function-mode, or (even better) DB2 9. Acknowledgements
I did not come up with all of the ideas presented in this book. Many of the best examples were provided by readers, friends, and/or coworkers too numerous to list. Thanks also to the many people at IBM for their (strictly unofficial) assistance. Disclaimer & Copyright
DISCLAIMER: This document is a best effort on my part. However, I screw up all the time, so it would be extremely unwise to trust the contents in its entirety. I certainly don't. And if you do something silly based on what I say, life is tough. COPYRIGHT: You can make as many copies of this book as you wish. And I encourage you to give it to others. But you cannot charge for it (other than to recover reproduction costs), nor claim the material as your own, nor replace my name with another. You are also encouraged to use the related class notes for teaching. In this case, you can charge for your time and materials - and your expertise. But you cannot charge any licensing fee, nor claim an exclusive right of use. In other words, you can pretty well do anything you want. And if you find the above too restrictive, just let me know. TRADEMARKS: Lots of words in this document, like "DB2", are registered trademarks of the IBM Corporation. Lots of other words, like "Windows", are registered trademarks of the Microsoft Corporation. Acrobat is a registered trademark of the Adobe Corporation. Tools Used
This book was written on a Dell PC that came with oodles of RAM. All testing was done in DB2 V9.7 Express-C for Windows. Word for Windows was used to write the document. Adobe Acrobat was used to make the PDF file. Book Binding
This book looks best when printed on a doubled sided laser printer and then suitably bound. To this end, I did some experiments a few years ago to figure out how to bind books cheaply using commonly available materials. I came up with what I consider to be a very satisfactory solution that is fully documented on page 461. Author / Book
Author: Email: Web: Title: Date:
Preface
Graeme Birchall ©
[email protected] http://mysite.verizon.net/Graeme_Birchall/ DB2 9.7 SQL Cookbook © 14-Jan-2011
3
Graeme Birchall ©
Author Notes Book History
This book originally began a series of notes for my own use. After a while, friends began to ask for copies, and enemies started to steal it, so I decided to tidy everything up and give it away. Over the years, new chapters have been added as DB2 has evolved, and as I have found new ways to solve problems. Hopefully, this process will continue for the foreseeable future. Why Free
This book is free because I want people to use it. The more people that use it, and the more that it helps them, the more inclined I am to keep it up to date. For these reasons, if you find this book to be useful, please share it with others. This book is free, rather than formally published, because I want to deliver the best product that I can. If I had a publisher, I would have the services of an editor and a graphic designer, but I would not be able to get to market so quickly, and when a product changes as quickly as DB2 does, timeliness is important. Also, giving it away means that I am under no pressure to make the book marketable. I simply include whatever I think might be useful. Other Free Documents
The following documents are also available for free from my web site:
SAMPLE SQL: The complete text of the SQL statements in this Cookbook is available in an HTML file. Only the first and last few lines of the file have HTML tags, the rest is raw text, so it can easily be cut and paste into other files.
CLASS OVERHEADS: Selected SQL examples from this book have been rewritten as class overheads. This enables one to use this material to teach DB2 SQL to others. Use this cookbook as the student notes.
OLDER EDITIONS: This book is rewritten, and usually much improved, with each new version of DB2. Some of the older editions are available from my website. The others can be emailed upon request. However, the latest edition is the best, so you should probably use it, regardless of the version of DB2 that you have.
Answering Questions
As a rule, I do not answer technical questions because I need to have a life. But I'm interested in hearing about interesting SQL problems, and also about any bugs in this book. However you may not get a prompt response, or any response. And if you are obviously an idiot, don't be surprised if I point out (for free, remember) that you are an idiot. Software Whines
This book is written using Microsoft Word for Windows. I've been using this software for many years, and it has generally been a bunch of bug-ridden junk. I do confess that it has been mildly more reliable in recent years. However, I could have written more than twice as much that was twice as good in half the time - if it weren't for all of the bugs in Word. Graeme
4
DB2 V9.7 Cookbook ©
Book Editions Upload Dates
1996-05-08: First edition of the DB2 V2.1.1 SQL Cookbook was posted to my web site. This version was in Postscript Print File format. 1998-02-26: The DB2 V2.1.1 SQL Cookbook was converted to an Adobe Acrobat file and posted to my web site. Some minor cosmetic changes were made. 1998-08-19: First edition of DB2 UDB V5 SQL Cookbook posted. Every SQL statement was checked for V5, and there were new chapters on OUTER JOIN and GROUP BY. 1998-08-26: About 20 minor cosmetic defects were corrected in the V5 Cookbook. 1998-09-03: Another 30 or so minor defects were corrected in the V5 Cookbook. 1998-10-24: The Cookbook was updated for DB2 UDB V5.2. 1998-10-25: About twenty minor typos and sundry cosmetic defects were fixed. 1998-12-03: This book was based on the second edition of the V5.2 upgrade. 1999-01-25: A chapter on Summary Tables (new in the Dec/98 fixpack) was added and all the SQL was checked for changes. 1999-01-28: Some more SQL was added to the new chapter on Summary Tables. 1999-02-15: The section of stopping recursive SQL statements was completely rewritten, and a new section was added on denormalizing hierarchical data structures. 1999-02-16: Minor editorial changes were made. 1999-03-16: Some bright spark at IBM pointed out that my new and improved section on stopping recursive SQL was all wrong. Damn. I undid everything. 1999-05-12: Minor editorial changes were made, and one new example (on getting multiple counts from one value) was added. 1999-09-16: DB2 V6.1 edition. All SQL was rechecked, and there were some minor additions - especially to summary tables, plus a chapter on "DB2 Dislikes". 1999-09-23: Some minor layout changes were made. 1999-10-06: Some bugs fixed, plus new section on index usage in summary tables. 2000-04-12: Some typos fixed, and a couple of new SQL tricks were added. 2000-09-19: DB2 V7.1 edition. All SQL was rechecked. The new areas covered are: OLAP functions (whole chapter), ISO functions, and identity columns. 2000-09-25: Some minor layout changes were made. 2000-10-26: More minor layout changes. 2001-01-03: Minor layout changes (to match class notes). 2001-02-06: Minor changes, mostly involving the RAND function. 2001-04-11: Document new features in latest fixpack. Also add a new chapter on Identity Columns and completely rewrite sub-query chapter. 2001-10-24: DB2 V7.2 fixpack 4 edition. Tested all SQL and added more examples, plus a new section on the aggregation function. 2002-03-11: Minor changes, mostly to section on precedence rules. 2002-08-20: DB2 V8.1 (beta) edition. A few new functions are added. New section on temporary tables. Identity Column and Join chapters rewritten. Whine chapter removed. 2003-01-02: DB2 V8.1 (post-Beta) edition. SQL rechecked. More examples added. 2003-07-11: New sections added on DML, temporary tables, compound SQL, and user defined functions. Halting recursion section changed to use user-defined function. 2003-09-04: New sections on complex joins and history tables. 2003-10-02: Minor changes. Some more user-defined functions. 2003-11-20: Added "quick find" chapter.
Book Editions
5
Graeme Birchall ©
6
2003-12-31: Tidied up the SQL in the Recursion chapter, and added a section on the merge statement. Completely rewrote the chapter on materialized query tables. 2004-02-04: Added select-from-DML section, and tidied up some code. Also managed to waste three whole days due to bugs in Microsoft Word. 2004-07-23: Rewrote chapter of identity column and sequences. Made DML separate chapter. Added chapters on protecting data and XML functions. Other minor changes. 2004-11-03: Upgraded to V8.2. Retested all SQL. Documented new SQL features. Some major hacking done on the GROUP BY chapter. 2005-04-15: Added short section on cursors, and a chapter on using SQL to make SQL. 2005-06-01: Added a chapter on triggers. 2005-11-11: Updated MQT table chapter and added bibliography. Other minor changes. 2005-12-01: Applied fixpack 10. Changed my website name. 2005-12-16: Added notes on isolation levels, data-type functions, transforming data. 2006-01-26: Fixed dumb bugs generated by WORD. What stupid software. Also wrote an awesome new section on joining meta-data to real data. 2006-02-17: Touched up the section on joining meta-data to real data. Other minor fixes. 2006-02-27: Added precedence rules for SQL statement processing, and a description of a simplified nested table expression. 2006-03-23: Added better solution to avoid fetching the same row twice. 2006-04-26: Added trigger that can convert HEX value to number. 2006-09-08: Upgraded to V9.1. Retested SQL. Removed the XML chapter as it is now obsolete. I'm still cogitating about XQuery. Looks hard. Added some awesome java code. 2006-09-13: Fixed some minor problems in the initial V9.1 book. 2006-10-17: Fixed a few cosmetic problems that were bugging me. 2006-11-06: Found out that IBM had removed the "UDB" from the DB2 product name, so I did the same. It is now just plain "DB2 V9". 2006-11-29: I goofed. Turns out DB2 is now called "DB2 9". I relabeled accordingly. 2006-12-15: Improved code to update or delete first "n" rows. 2007-02-22: Get unique timestamp values during multi-row insert. Other minor changes. 2007-11-20: Finished the DB2 V9.5 edition. Lots of changes! 2008-09-20: Fixed some minor problems. 2008-11-28: Fixed some minor problems. 2009-01-18: Fixed some minor problems, plus lots of bugs in Microsoft WORD! 2009-03-12: Converted to a new version of Adobe Acrobat, plus minor fixes. 2010-10-12: Finished initial V9.7 edition. Only minor changes. More to come. 2010-11-05: First batch of cute/deranged V9.7 SQL examples added. 2010-11-14: Fixed some minor typos. 2011-01-11: Added LIKE_COLUMN function. Removed bibliography. 2011-01-14: Added HASH function. Other minor edits.
DB2 V9.7 Cookbook ©
Table of Contents
PREFACE .............................................................................................................................. 3 AUTHOR NOTES .................................................................................................................... 4 BOOK EDITIONS .................................................................................................................... 5 TABLE OF CONTENTS ............................................................................................................ 7 QUICK FIND ........................................................................................................................ 17 Index of Concepts........................................................................................................................................ 17
INTRODUCTION TO SQL....................................................................................................... 21 Syntax Diagram Conventions.............................................................................................................................................. 21
SQL Components ........................................................................................................................................ 22 DB2 Objects........................................................................................................................................................................ 22 DB2 Data Types ................................................................................................................................................................. 24 DECFLOAT Arithmetic........................................................................................................................................................ 25 Date/Time Arithmetic .......................................................................................................................................................... 27 DB2 Special Registers ........................................................................................................................................................ 29 Distinct Types ..................................................................................................................................................................... 31 Fullselect, Subselect, & Common Table Expression ........................................................................................................... 32 SELECT Statement............................................................................................................................................................. 33 FETCH FIRST Clause ........................................................................................................................................................ 35 Correlation Name................................................................................................................................................................ 36 Renaming Fields ................................................................................................................................................................. 36 Working with Nulls .............................................................................................................................................................. 37 Quotes and Double-quotes ................................................................................................................................................. 38
SQL Predicates ............................................................................................................................................ 38 Basic Predicate................................................................................................................................................................... 39 Quantified Predicate ........................................................................................................................................................... 39 BETWEEN Predicate .......................................................................................................................................................... 40 EXISTS Predicate ............................................................................................................................................................... 40 IN Predicate ........................................................................................................................................................................ 41 LIKE Predicate.................................................................................................................................................................... 41 LIKE_COLUMN Function .................................................................................................................................................... 43 NULL Predicate .................................................................................................................................................................. 44 Special Character Usage .................................................................................................................................................... 44 Precedence Rules............................................................................................................................................................... 44 Processing Sequence ......................................................................................................................................................... 45
CAST Expression......................................................................................................................................... 46 VALUES Statement ...................................................................................................................................... 47 CASE Expression ........................................................................................................................................ 50 CASE Syntax Styles ........................................................................................................................................................... 50 Sample SQL ....................................................................................................................................................................... 51
Miscellaneous SQL Statements ................................................................................................................. 54 Cursor................................................................................................................................................................................. 54 Select Into........................................................................................................................................................................... 56 Prepare............................................................................................................................................................................... 56 Describe ............................................................................................................................................................................. 56 Execute............................................................................................................................................................................... 57 Execute Immediate ............................................................................................................................................................. 57 Set Variable ........................................................................................................................................................................ 57 Set DB2 Control Structures................................................................................................................................................. 58
Table of Contents
7
Graeme Birchall ©
Unit-of-Work Processing............................................................................................................................. 58 Commit................................................................................................................................................................................58 Savepoint ............................................................................................................................................................................59 Release Savepoint ..............................................................................................................................................................60 Rollback ..............................................................................................................................................................................60
DATA MANIPULATION LANGUAGE.........................................................................................61 Insert ............................................................................................................................................................. 61 Update........................................................................................................................................................... 65 Delete ............................................................................................................................................................ 68 Select DML Changes ................................................................................................................................... 70 Merge ............................................................................................................................................................ 73
COMPOUND SQL.................................................................................................................79 Introduction.................................................................................................................................................. 79 Statement Delimiter .............................................................................................................................................................79
SQL Statement Usage ................................................................................................................................. 80 DECLARE Variables............................................................................................................................................................80 FOR Statement ...................................................................................................................................................................81 GET DIAGNOSTICS Statement ..........................................................................................................................................81 IF Statement........................................................................................................................................................................82 ITERATE Statement ............................................................................................................................................................82 LEAVE Statement ...............................................................................................................................................................83 SIGNAL Statement ..............................................................................................................................................................83 WHILE Statement................................................................................................................................................................84
Other Usage ................................................................................................................................................. 84 Trigger.................................................................................................................................................................................85 Scalar Function ...................................................................................................................................................................86 Table Function.....................................................................................................................................................................87
COLUMN FUNCTIONS ...........................................................................................................89 Introduction .........................................................................................................................................................................89
Column Functions, Definitions................................................................................................................... 89 ARRAY_AGG ......................................................................................................................................................................89 AVG.....................................................................................................................................................................................89 CORRELATION ..................................................................................................................................................................91 COUNT ...............................................................................................................................................................................91 COUNT_BIG .......................................................................................................................................................................92 COVARIANCE.....................................................................................................................................................................92 GROUPING.........................................................................................................................................................................93 MAX ....................................................................................................................................................................................93 MIN .....................................................................................................................................................................................94 Regression Functions..........................................................................................................................................................95 STDDEV..............................................................................................................................................................................95 SUM ....................................................................................................................................................................................96 VAR or VARIANCE..............................................................................................................................................................96
OLAP FUNCTIONS ..............................................................................................................97 Introduction.................................................................................................................................................. 97 The Bad Old Days ...............................................................................................................................................................97
Concepts ...................................................................................................................................................... 98 PARTITION Expression..................................................................................................................................................... 100 Window Definition.............................................................................................................................................................. 101 ROWS vs. RANGE ............................................................................................................................................................103 ORDER BY Expression ..................................................................................................................................................... 104 Table Designator ............................................................................................................................................................... 105 Nulls Processing................................................................................................................................................................ 105
OLAP Functions......................................................................................................................................... 106 RANK and DENSE_RANK ................................................................................................................................................ 106 ROW_NUMBER ................................................................................................................................................................ 111 FIRST_VALUE and LAST_VALUE .................................................................................................................................... 117
8
DB2 V9.7 Cookbook ©
LAG and LEAD ................................................................................................................................................................. 119 Aggregation ...................................................................................................................................................................... 120
SCALAR FUNCTIONS ......................................................................................................... 127 Introduction ....................................................................................................................................................................... 127 Sample Data ..................................................................................................................................................................... 127
Scalar Functions, Definitions ................................................................................................................... 127 ABS or ABSVAL ............................................................................................................................................................... 127 ACOS ............................................................................................................................................................................... 128 ASCII ................................................................................................................................................................................ 128 ASIN ................................................................................................................................................................................. 128 ATAN ................................................................................................................................................................................ 128 ATAN2 .............................................................................................................................................................................. 128 ATANH ............................................................................................................................................................................. 128 BIGINT.............................................................................................................................................................................. 128 BIT Functions.................................................................................................................................................................... 129 BLOB ................................................................................................................................................................................ 132 CARDINALITY .................................................................................................................................................................. 132 CEIL or CEILING .............................................................................................................................................................. 132 CHAR ............................................................................................................................................................................... 133 CHARACTER_LENGTH ................................................................................................................................................... 135 CHR.................................................................................................................................................................................. 136 CLOB................................................................................................................................................................................ 136 COALESCE ...................................................................................................................................................................... 136 COLLATION_KEY_BIT..................................................................................................................................................... 137 COMPARE_DECFLOAT................................................................................................................................................... 138 CONCAT .......................................................................................................................................................................... 138 COS.................................................................................................................................................................................. 139 COSH ............................................................................................................................................................................... 139 COT .................................................................................................................................................................................. 139 DATAPARTITIONNUM ..................................................................................................................................................... 139 DATE ................................................................................................................................................................................ 140 DAY .................................................................................................................................................................................. 140 DAYNAME ........................................................................................................................................................................ 141 DAYOFWEEK................................................................................................................................................................... 141 DAYOFWEEK_ISO........................................................................................................................................................... 141 DAYOFYEAR.................................................................................................................................................................... 142 DAYS................................................................................................................................................................................ 142 DBCLOB ........................................................................................................................................................................... 142 DBPARTITIONNUM.......................................................................................................................................................... 143 DECFLOAT....................................................................................................................................................................... 143 DEC or DECIMAL ............................................................................................................................................................. 143 DECODE .......................................................................................................................................................................... 144 DECRYPT_BIN and DECRYPT_CHAR ............................................................................................................................ 144 DEGREES ........................................................................................................................................................................ 145 DEREF ............................................................................................................................................................................. 145 DIFFERENCE................................................................................................................................................................... 145 DIGITS.............................................................................................................................................................................. 145 DOUBLE or DOUBLE_PRECISION.................................................................................................................................. 146 ENCRYPT......................................................................................................................................................................... 146 EVENT_MON_STATE ...................................................................................................................................................... 147 EXP .................................................................................................................................................................................. 147 FLOAT .............................................................................................................................................................................. 147 FLOOR ............................................................................................................................................................................. 147 GENERATE_UNIQUE ...................................................................................................................................................... 147 GETHINT .......................................................................................................................................................................... 149 GRAPHIC ......................................................................................................................................................................... 150 GREATEST ...................................................................................................................................................................... 150 HASHEDVALUE ............................................................................................................................................................... 150 HEX .................................................................................................................................................................................. 150 HOUR ............................................................................................................................................................................... 151 IDENTITY_VAL_LOCAL ................................................................................................................................................... 151 INSERT ............................................................................................................................................................................ 151 INT or INTEGER............................................................................................................................................................... 152 JULIAN_DAY .................................................................................................................................................................... 152 LCASE or LOWER............................................................................................................................................................ 154 LEAST .............................................................................................................................................................................. 154 LEFT................................................................................................................................................................................. 154 LENGTH ........................................................................................................................................................................... 155 LN or LOG ........................................................................................................................................................................ 155 LOCATE ........................................................................................................................................................................... 155 LOG or LN ........................................................................................................................................................................ 156 LOG10 .............................................................................................................................................................................. 156 LONG_VARCHAR ............................................................................................................................................................ 156
Table of Contents
9
Graeme Birchall ©
LONG_VARGRAPHIC....................................................................................................................................................... 156 LOWER ............................................................................................................................................................................. 156 LTRIM ............................................................................................................................................................................... 156 MAX ..................................................................................................................................................................................156 MAX_CARDINALITY .........................................................................................................................................................157 MICROSECOND ............................................................................................................................................................... 157 MIDNIGHT_SECONDS ..................................................................................................................................................... 157 MIN ...................................................................................................................................................................................158 MINUTE ............................................................................................................................................................................158 MOD..................................................................................................................................................................................158 MONTH .............................................................................................................................................................................158 MONTHNAME...................................................................................................................................................................158 MULTIPLY_ALT ................................................................................................................................................................ 159 NORMALIZE_DECFLOAT................................................................................................................................................. 159 NULLIF.............................................................................................................................................................................. 160 NVL ...................................................................................................................................................................................160 OCTET_LENGTH.............................................................................................................................................................. 160 OVERLAY .........................................................................................................................................................................160 PARTITION ....................................................................................................................................................................... 161 POSITION ......................................................................................................................................................................... 161 POSSTR ...........................................................................................................................................................................162 POWER............................................................................................................................................................................. 162 QUANTIZE ........................................................................................................................................................................ 163 QUARTER......................................................................................................................................................................... 163 RADIANS .......................................................................................................................................................................... 163 RAISE_ERROR.................................................................................................................................................................163 RAND ................................................................................................................................................................................164 REAL................................................................................................................................................................................. 167 REPEAT............................................................................................................................................................................168 REPLACE .........................................................................................................................................................................168 RID....................................................................................................................................................................................168 RID_BIT ............................................................................................................................................................................169 RIGHT ............................................................................................................................................................................... 170 ROUND .............................................................................................................................................................................170 RTRIM...............................................................................................................................................................................171 SECLABEL Functions ....................................................................................................................................................... 171 SECOND...........................................................................................................................................................................171 SIGN ................................................................................................................................................................................. 171 SIN ....................................................................................................................................................................................171 SINH..................................................................................................................................................................................171 SMALLINT.........................................................................................................................................................................172 SNAPSHOT Functions ...................................................................................................................................................... 172 SOUNDEX ........................................................................................................................................................................172 SPACE .............................................................................................................................................................................. 173 SQRT ................................................................................................................................................................................ 173 STRIP................................................................................................................................................................................173 SUBSTR............................................................................................................................................................................174 TABLE............................................................................................................................................................................... 175 TABLE_NAME...................................................................................................................................................................175 TABLE_SCHEMA.............................................................................................................................................................. 175 TAN ...................................................................................................................................................................................176 TANH ................................................................................................................................................................................176 TIME..................................................................................................................................................................................176 TIMESTAMP ..................................................................................................................................................................... 176 TIMESTAMP_FORMAT .................................................................................................................................................... 177 TIMESTAMP_ISO ............................................................................................................................................................. 177 TIMESTAMPDIFF ............................................................................................................................................................. 177 TO_CHAR .........................................................................................................................................................................178 TO_DATE.......................................................................................................................................................................... 178 TOTALORDER.................................................................................................................................................................. 178 TRANSLATE ..................................................................................................................................................................... 179 TRIM .................................................................................................................................................................................180 TRUNC or TRUNCATE ..................................................................................................................................................... 180 TYPE_ID ........................................................................................................................................................................... 180 TYPE_NAME..................................................................................................................................................................... 180 TYPE_SCHEMA................................................................................................................................................................ 180 UCASE or UPPER.............................................................................................................................................................180 VALUE .............................................................................................................................................................................. 181 VARCHAR.........................................................................................................................................................................181 VARCHAR_BIT_FORMAT ................................................................................................................................................ 181 VARCHAR_FORMAT........................................................................................................................................................ 181 VARCHAR_FORMAT_BIT ................................................................................................................................................ 181 VARGRAPHIC...................................................................................................................................................................181 WEEK................................................................................................................................................................................ 182 WEEK_ISO........................................................................................................................................................................ 182 YEAR ................................................................................................................................................................................ 182
10
DB2 V9.7 Cookbook ©
"+" PLUS........................................................................................................................................................................... 183 "-" MINUS ......................................................................................................................................................................... 183 "*" MULTIPLY ................................................................................................................................................................... 183 "/" DIVIDE ......................................................................................................................................................................... 184 "||" CONCAT ..................................................................................................................................................................... 184
USER DEFINED FUNCTIONS ............................................................................................... 185 Sourced Functions .................................................................................................................................... 185 Scalar Functions........................................................................................................................................ 187 Description........................................................................................................................................................................ 187 Examples .......................................................................................................................................................................... 188
Table Functions ......................................................................................................................................... 192 Description........................................................................................................................................................................ 192 Examples .......................................................................................................................................................................... 193
Useful User-Defined Functions ................................................................................................................ 194 Julian Date Functions ....................................................................................................................................................... 194 Get Prior Date................................................................................................................................................................... 194 Generating Numbers......................................................................................................................................................... 196 Check Data Value Type .................................................................................................................................................... 197 Hash Function................................................................................................................................................................... 199
ORDER BY, GROUP BY, AND HAVING ................................................................................. 201 Order By ..................................................................................................................................................... 201 Notes ................................................................................................................................................................................ 201 Sample Data ..................................................................................................................................................................... 201 Order by Examples ........................................................................................................................................................... 202
Group By and Having ................................................................................................................................ 204 Rules and Restrictions ...................................................................................................................................................... 204 GROUP BY Flavors .......................................................................................................................................................... 205 GROUP BY Sample Data ................................................................................................................................................. 206 Simple GROUP BY Statements ........................................................................................................................................ 206 GROUPING SETS Statement........................................................................................................................................... 207 ROLLUP Statement .......................................................................................................................................................... 211 CUBE Statement............................................................................................................................................................... 215 Complex Grouping Sets - Done Easy ............................................................................................................................... 218 Group By and Order By .................................................................................................................................................... 220 Group By in Join ............................................................................................................................................................... 220 COUNT and No Rows....................................................................................................................................................... 221
JOINS ............................................................................................................................... 223 Why Joins Matter .............................................................................................................................................................. 223 Sample Views ................................................................................................................................................................... 223
Join Syntax................................................................................................................................................. 223 Query Processing Sequence ............................................................................................................................................ 225 ON vs. WHERE ................................................................................................................................................................ 225
Join Types .................................................................................................................................................. 226 Inner Join .......................................................................................................................................................................... 226 Left Outer Join .................................................................................................................................................................. 227 Right Outer Join................................................................................................................................................................ 229 Full Outer Joins................................................................................................................................................................. 230 Cartesian Product ............................................................................................................................................................. 234
Join Notes .................................................................................................................................................. 236 Using the COALESCE Function........................................................................................................................................ 236 Listing non-matching rows only......................................................................................................................................... 236 Join in SELECT Phrase .................................................................................................................................................... 238 Predicates and Joins, a Lesson ........................................................................................................................................ 240 Joins - Things to Remember ............................................................................................................................................. 241 Complex Joins .................................................................................................................................................................. 242
SUB-QUERY ..................................................................................................................... 245 Sample Tables.................................................................................................................................................................. 245
Sub-query Flavors ..................................................................................................................................... 245 Sub-query Syntax ............................................................................................................................................................. 245
Table of Contents
11
Graeme Birchall ©
Correlated vs. Uncorrelated Sub-Queries .......................................................................................................................... 252 Multi-Field Sub-Queries..................................................................................................................................................... 253 Nested Sub-Queries .......................................................................................................................................................... 253
Usage Examples ........................................................................................................................................ 254 True if NONE Match .......................................................................................................................................................... 254 True if ANY Match ............................................................................................................................................................. 255 True if TEN Match .............................................................................................................................................................256 True if ALL match ..............................................................................................................................................................257
UNION, INTERSECT, AND EXCEPT .......................................................................................259 Syntax Diagram................................................................................................................................................................. 259 Sample Views....................................................................................................................................................................259
Usage Notes ............................................................................................................................................... 260 Union & Union All ..............................................................................................................................................................260 Intersect & Intersect All...................................................................................................................................................... 260 Except, Except All, & Minus............................................................................................................................................... 260 Precedence Rules ............................................................................................................................................................. 261 Unions and Views..............................................................................................................................................................262
MATERIALIZED QUERY TABLES ..........................................................................................263 Introduction ....................................................................................................................................................................... 263
Usage Notes ............................................................................................................................................... 263 Syntax Options .................................................................................................................................................................. 264 Select Statement ...............................................................................................................................................................265 Optimizer Options.............................................................................................................................................................. 266 Refresh Deferred Tables ...................................................................................................................................................268 Refresh Immediate Tables................................................................................................................................................. 269 Usage Notes and Restrictions ........................................................................................................................................... 271 Multi-table Materialized Query Tables ............................................................................................................................... 272 Indexes on Materialized Query Tables .............................................................................................................................. 274 Organizing by Dimensions................................................................................................................................................. 275 Using Staging Tables ........................................................................................................................................................ 275
IDENTITY COLUMNS AND SEQUENCES .................................................................................277 Identity Columns........................................................................................................................................ 277 Rules and Restrictions....................................................................................................................................................... 278 Altering Identity Column Options ....................................................................................................................................... 281 Gaps in Identity Column Values ........................................................................................................................................ 282 Find Gaps in Values .......................................................................................................................................................... 283 IDENTITY_VAL_LOCAL Function ..................................................................................................................................... 284
Sequences.................................................................................................................................................. 286 Getting the Sequence Value.............................................................................................................................................. 287 Multi-table Usage .............................................................................................................................................................. 289 Counting Deletes............................................................................................................................................................... 290 Identity Columns vs. Sequences - a Comparison .............................................................................................................. 291
Roll Your Own ............................................................................................................................................ 292 Support Multi-row Inserts................................................................................................................................................... 293
TEMPORARY TABLES .........................................................................................................297 Introduction................................................................................................................................................ 297 Temporary Tables - in Statement ............................................................................................................. 299 Common Table Expression ............................................................................................................................................... 300 Full-Select ......................................................................................................................................................................... 302
Declared Global Temporary Tables ......................................................................................................... 306
RECURSIVE SQL ...............................................................................................................309 Use Recursion To.............................................................................................................................................................. 309 When (Not) to Use Recursion............................................................................................................................................ 309
How Recursion Works............................................................................................................................... 309 List Dependents of AAA .................................................................................................................................................... 310 Notes & Restrictions .......................................................................................................................................................... 311 Sample Table DDL & DML ................................................................................................................................................ 311
12
DB2 V9.7 Cookbook ©
Introductory Recursion ............................................................................................................................. 312 List all Children #1 ............................................................................................................................................................ 312 List all Children #2 ............................................................................................................................................................ 312 List Distinct Children ......................................................................................................................................................... 313 Show Item Level ............................................................................................................................................................... 313 Select Certain Levels ........................................................................................................................................................ 314 Select Explicit Level .......................................................................................................................................................... 315 Trace a Path - Use Multiple Recursions ............................................................................................................................ 315 Extraneous Warning Message .......................................................................................................................................... 316
Logical Hierarchy Flavours....................................................................................................................... 317 Divergent Hierarchy .......................................................................................................................................................... 317 Convergent Hierarchy ....................................................................................................................................................... 318 Recursive Hierarchy.......................................................................................................................................................... 318 Balanced & Unbalanced Hierarchies................................................................................................................................. 319 Data & Pointer Hierarchies................................................................................................................................................ 319
Halting Recursive Processing .................................................................................................................. 320 Sample Table DDL & DML................................................................................................................................................ 320 Stop After "n" Levels ......................................................................................................................................................... 321 Stop When Loop Found .................................................................................................................................................... 322 Keeping the Hierarchy Clean ............................................................................................................................................ 325
Clean Hierarchies and Efficient Joins ..................................................................................................... 327 Introduction ....................................................................................................................................................................... 327 Limited Update Solution.................................................................................................................................................... 327 Full Update Solution.......................................................................................................................................................... 329
TRIGGERS......................................................................................................................... 333 Trigger Syntax............................................................................................................................................ 333 Usage Notes ..................................................................................................................................................................... 333 Trigger Usage ................................................................................................................................................................... 334
Trigger Examples....................................................................................................................................... 335 Sample Tables.................................................................................................................................................................. 335 Before Row Triggers - Set Values..................................................................................................................................... 335 Before Row Trigger - Signal Error ..................................................................................................................................... 336 After Row Triggers - Record Data States .......................................................................................................................... 336 After Statement Triggers - Record Changes ..................................................................................................................... 337 Examples of Usage........................................................................................................................................................... 338
PROTECTING YOUR DATA.................................................................................................. 341 Sample Application ................................................................................................................................... 341 Enforcement Tools............................................................................................................................................................ 342 Distinct Data Types........................................................................................................................................................... 343 Customer-Balance Table .................................................................................................................................................. 343 US-Sales Table................................................................................................................................................................. 344 Triggers ............................................................................................................................................................................ 345 Conclusion ........................................................................................................................................................................ 348
RETAINING A RECORD ....................................................................................................... 351 Schema Design .......................................................................................................................................... 351 Recording Changes .......................................................................................................................................................... 351 Multiple Versions of the World .......................................................................................................................................... 354
USING SQL TO MAKE SQL ............................................................................................... 361 Export Command ....................................................................................................................................... 361 SQL to Make SQL............................................................................................................................................................. 362
RUNNING SQL WITHIN SQL.............................................................................................. 365 Introduction ................................................................................................................................................ 365 Generate SQL within SQL................................................................................................................................................. 365 Make Query Column-Independent .................................................................................................................................... 366 Business Uses .................................................................................................................................................................. 367 Meta Data Dictionaries...................................................................................................................................................... 368
DB2 SQL Functions ................................................................................................................................... 368
Table of Contents
13
Graeme Birchall ©
Function and Stored Procedure Used................................................................................................................................ 368 Different Data Types..........................................................................................................................................................369 Usage Examples ...............................................................................................................................................................370
Java Functions........................................................................................................................................... 372 Scalar Functions................................................................................................................................................................ 372 Tabular Functions..............................................................................................................................................................373 Transpose Function...........................................................................................................................................................376
Update Real Data using Meta-Data .......................................................................................................... 383 Usage Examples ...............................................................................................................................................................384
FUN WITH SQL..................................................................................................................389 Creating Sample Data................................................................................................................................ 389 Data Generation ................................................................................................................................................................ 389 Make Reproducible Random Data..................................................................................................................................... 389 Make Random Data - Different Ranges............................................................................................................................. 390 Make Random Data - Varying Distribution......................................................................................................................... 390 Make Random Data - Different Flavours ........................................................................................................................... 391 Make Test Table & Data.................................................................................................................................................... 391
Time-Series Processing ............................................................................................................................ 393 Find Overlapping Rows ..................................................................................................................................................... 394 Find Gaps in Time-Series.................................................................................................................................................. 395 Show Each Day in Gap ..................................................................................................................................................... 396
Other Fun Things....................................................................................................................................... 396 Randomly Sample Data..................................................................................................................................................... 396 Convert Character to Numeric ........................................................................................................................................... 398 Convert Number to Character............................................................................................................................................ 400 Convert Timestamp to Numeric ......................................................................................................................................... 403 Selective Column Output................................................................................................................................................... 404 Making Charts Using SQL ................................................................................................................................................. 404 Multiple Counts in One Pass ............................................................................................................................................. 405 Find Missing Rows in Series / Count all Values................................................................................................................. 406 Multiple Counts from the Same Row ................................................................................................................................. 408 Normalize Denormalized Data........................................................................................................................................... 409 Denormalize Normalized Data........................................................................................................................................... 410 Transpose Numeric Data................................................................................................................................................... 412 Reversing Field Contents .................................................................................................................................................. 415 Fibonacci Series................................................................................................................................................................ 416 Business Day Calculation.................................................................................................................................................. 418 Query Runs for "n" Seconds.............................................................................................................................................. 418 Sort Character Field Contents ........................................................................................................................................... 419 Calculating the Median ......................................................................................................................................................421 Converting HEX Data to Number....................................................................................................................................... 424
QUIRKS IN SQL.................................................................................................................427 Trouble with Timestamps .................................................................................................................................................. 427 No Rows Match .................................................................................................................................................................428 Dumb Date Usage............................................................................................................................................................. 429 RAND in Predicate ............................................................................................................................................................ 430 Date/Time Manipulation..................................................................................................................................................... 432 Use of LIKE on VARCHAR ................................................................................................................................................ 433 Comparing Weeks.............................................................................................................................................................434 DB2 Truncates, not Rounds .............................................................................................................................................. 434 CASE Checks in Wrong Sequence.................................................................................................................................... 435 Division and Average......................................................................................................................................................... 435 Date Output Order............................................................................................................................................................. 435 Ambiguous Cursors ........................................................................................................................................................... 436 Multiple User Interactions .................................................................................................................................................. 437 What Time is It .................................................................................................................................................................. 441 Floating Point Numbers ..................................................................................................................................................... 442
APPENDIX .........................................................................................................................447 DB2 Sample Tables ................................................................................................................................... 447 ACT ................................................................................................................................................................................... 447 CATALOG ......................................................................................................................................................................... 447 CL_SCHED ....................................................................................................................................................................... 448 CUSTOMER......................................................................................................................................................................448 DATA_FILE_NAMES......................................................................................................................................................... 448
14
DB2 V9.7 Cookbook ©
DEPARTMENT ................................................................................................................................................................. 449 EMPLOYEE ...................................................................................................................................................................... 450 EMPMDC.......................................................................................................................................................................... 451 EMPPROJACT ................................................................................................................................................................. 451 EMP_PHOTO ................................................................................................................................................................... 453 EMP_RESUME................................................................................................................................................................. 453 IN_TRAY .......................................................................................................................................................................... 454 INVENTORY..................................................................................................................................................................... 454 ORG ................................................................................................................................................................................. 454 PRODUCT ........................................................................................................................................................................ 455 PRODUCTSUPPLIER ...................................................................................................................................................... 455 PROJACT ......................................................................................................................................................................... 455 PROJECT ......................................................................................................................................................................... 457 PURCHASEORDER ......................................................................................................................................................... 457 SALES .............................................................................................................................................................................. 458 STAFF .............................................................................................................................................................................. 459 SUPPLIERS...................................................................................................................................................................... 459
BOOK BINDING ................................................................................................................. 461 INDEX ............................................................................................................................... 463
Table of Contents
15
Graeme Birchall ©
16
DB2 V9.7 Cookbook ©
Quick Find This brief chapter is for those who want to find how to do something, but are not sure what the task is called. Hopefully, this list will identify the concept.
Index of Concepts Join Rows
To combine matching rows in multiple tables, use a join (see page 223). EMP_NM +----------+ |ID|NAME | |--|-------| |10|Sanders| |20|Pernal | |50|Hanes | +----------+
EMP_JB +--------+ |ID|JOB | |--|-----| |10|Sales| |20|Clerk| +--------+
SELECT
nm.id ,nm.name ,jb.job FROM emp_nm nm ,emp_jb jb WHERE nm.id = jb.id ORDER BY 1;
ANSWER ================ ID NAME JOB -- ------- ----10 Sanders Sales 20 Pernal Clerk
Figure 1, Join example Outer Join
To get all of the rows from one table, plus the matching rows from another table (if there are any), use an outer join (see page 226). EMP_NM +----------+ |ID|NAME | |--|-------| |10|Sanders| |20|Pernal | |50|Hanes | +----------+
EMP_JB +--------+ |ID|JOB | |--|-----| |10|Sales| |20|Clerk| +--------+
SELECT
nm.id ,nm.name ,jb.job FROM emp_nm nm LEFT OUTER JOIN emp_jb jb ON nm.id = jb.id ORDER BY nm.id;
ANSWER ================ ID NAME JOB -- ------- ----10 Sanders Sales 20 Pernal Clerk 50 Hanes -
Figure 2, Left-outer-join example To get rows from either side of the join, regardless of whether they match (the join) or not, use a full outer join (see page 230). Null Values - Replace
Use the COALESCE function (see page 136) to replace a null value (e.g. generated in an outer join) with a non-null value. Select Where No Match
To get the set of the matching rows from one table where something is true or false in another table (e.g. no corresponding row), use a sub-query (see page 245). EMP_NM +----------+ |ID|NAME | |--|-------| |10|Sanders| |20|Pernal | |50|Hanes | +----------+
EMP_JB +--------+ |ID|JOB | |--|-----| |10|Sales| |20|Clerk| +--------+
SELECT * FROM emp_nm nm WHERE NOT EXISTS (SELECT * FROM emp_jb jb WHERE nm.id = jb.id) ORDER BY id;
ANSWER ======== ID NAME == ===== 50 Hanes
Figure 3, Sub-query example
Quick Find
17
Graeme Birchall ©
Append Rows
To add (append) one set of rows to another set of rows, use a union (see page 259). EMP_NM +----------+ |ID|NAME | |--|-------| |10|Sanders| |20|Pernal | |50|Hanes | +----------+
EMP_JB +--------+ |ID|JOB | |--|-----| |10|Sales| |20|Clerk| +--------+
SELECT FROM WHERE UNION SELECT FROM ORDER BY
* emp_nm name < 'S'
ANSWER ========= ID 2 -- -----10 Sales 20 Clerk 20 Pernal 50 Hanes
* emp_jb 1,2;
Figure 4, Union example Assign Output Numbers
To assign line numbers to SQL output, use the ROW_NUMBER function (see page 111). EMP_JB +--------+ |ID|JOB | |--|-----| |10|Sales| |20|Clerk| +--------+
SELECT
id ,job ,ROW_NUMBER() OVER(ORDER BY job) AS R FROM emp_jb ORDER BY job;
ANSWER ========== ID JOB R -- ----- 20 Clerk 1 10 Sales 2
Figure 5, Assign row-numbers example Assign Unique Key Numbers
The make each row inserted into a table automatically get a unique key value, use an identity column, or a sequence, when creating the table (see page 277). If-Then-Else Logic
To include if-then-else logical constructs in SQL stmts, use the CASE phrase (see page 50). EMP_JB +--------+ |ID|JOB | |--|-----| |10|Sales| |20|Clerk| +--------+
SELECT
FROM
id ,job ,CASE WHEN job = 'Sales' THEN 'Fire' ELSE 'Demote' END AS STATUS emp_jb;
ANSWER =============== ID JOB STATUS -- ----- -----10 Sales Fire 20 Clerk Demote
Figure 6, Case stmt example Get Dependents
To get all of the dependents of some object, regardless of the degree of separation from the parent to the child, use recursion (see page 309). FAMILY +-----------+ |PARNT|CHILD| |-----|-----| |GrDad|Dad | |Dad |Dghtr| |Dghtr|GrSon| |Dghtr|GrDtr| +-----------+
WITH temp (persn, lvl) AS (SELECT parnt, 1 FROM family WHERE parnt = 'Dad' UNION ALL SELECT child, Lvl + 1 FROM temp, family WHERE persn = parnt) SELECT * FROM temp;
ANSWER ========= PERSN LVL ----- --Dad 1 Dghtr 2 GrSon 3 GrDtr 3
Figure 7, Recursion example Convert String to Rows
To convert a (potentially large) set of values in a string (character field) into separate rows (e.g. one row per word), use recursion (see page 409).
18
Index of Concepts
DB2 V9.7 Cookbook ©
INPUT DATA ================= "Some silly text"
Recursive SQL ============>
ANSWER =========== TEXT LINE# ----- ----Some 1 silly 2 text 3
Figure 8, Convert string to rows Be warned - in many cases, the code is not pretty. Convert Rows to String
To convert a (potentially large) set of values that are in multiple rows into a single combined field, use recursion (see page 410). INPUT DATA =========== TEXT LINE# ----- ----Some 1 silly 2 text 3
Recursive SQL ============>
ANSWER ================= "Some silly text"
Figure 9, Convert rows to string Fetch First "n" Rows
To fetch the first "n" matching rows, use the FETCH FIRST notation (see page 35). EMP_NM +----------+ |ID|NAME | |--|-------| |10|Sanders| |20|Pernal | |50|Hanes | +----------+
SELECT * FROM emp_nm ORDER BY id DESC FETCH FIRST 2 ROWS ONLY;
ANSWER ========= ID NAME -- -----50 Hanes 20 Pernal
Figure 10, Fetch first "n" rows example Another way to do the same thing is to assign row numbers to the output, and then fetch those rows where the row-number is less than "n" (see page 112). Fetch Subsequent "n" Rows
To the fetch the "n" through "n + m" rows, first use the ROW_NUMBER function to assign output numbers, then put the result in a nested-table-expression, and then fetch the rows with desired numbers (see page 112). Fetch Uncommitted Data
To retrieve data that may have been changed by another user, but which they have yet to commit, use the WITH UR (Uncommitted Read) notation. EMP_NM +----------+ |ID|NAME | |--|-------| |10|Sanders| |20|Pernal | |50|Hanes | +----------+
SELECT * FROM emp_nm WHERE name like 'S%' WITH UR;
ANSWER ========== ID NAME -- ------10 Sanders
Figure 11, Fetch WITH UR example Using this option can result in one fetching data that is subsequently rolled back, and so was never valid. Use with extreme care.
Quick Find
19
Graeme Birchall ©
Summarize Column Contents
Use a column function (see page 89) to summarize the contents of a column. EMP_NM +----------+ |ID|NAME | |--|-------| |10|Sanders| |20|Pernal | |50|Hanes | +----------+
SELECT FROM
AVG(id) AS avg ,MAX(name) AS maxn ,COUNT(*) AS #rows emp_nm;
ANSWER ================= AVG MAXN #ROWS --- ------- ----26 Sanders 3
Figure 12, Column Functions example Subtotals and Grand Totals
To obtain subtotals and grand-totals, use the ROLLUP or CUBE statements (see page 211). SELECT
FROM WHERE AND AND GROUP ORDER
job ,dept ,SUM(salary) AS sum_sal ,COUNT(*) AS #emps staff dept < 30 salary < 90000 job < 'S' BY ROLLUP(job, dept) BY job ,dept;
ANSWER ========================== JOB DEPT SUM_SAL #EMPS ----- ---- --------- ----Clerk 15 84766.70 2 Clerk 20 77757.35 2 Clerk - 162524.05 4 Mgr 10 243453.45 3 Mgr 15 80659.80 1 Mgr - 324113.25 4 - 486637.30 8
Figure 13, Subtotal and Grand-total example Enforcing Data Integrity
When a table is created, various DB2 features can be used to ensure that the data entered in the table is always correct:
Uniqueness (of values) can be enforced by creating unique indexes.
Check constraints can be defined to limit the values that a column can have.
Default values (for a column) can be defined - to be used when no value is provided.
Identity columns (see page 277), can be defined to automatically generate unique numeric values (e.g. invoice numbers) for all of the rows in a table. Sequences can do the same thing over multiple tables.
Referential integrity rules can be created to enforce key relationships between tables.
Triggers can be defined to enforce more complex integrity rules, and also to do things (e.g. populate an audit trail) whenever data is changed.
See the DB2 manuals for documentation or page 341 for more information about the above. Hide Complex SQL
One can create a view (see page 22) to hide complex SQL that is run repetitively. Be warned however that doing so can make it significantly harder to tune the SQL - because some of the logic will be in the user code, and some in the view definition. Summary Table
Some queries that use a GROUP BY can be made to run much faster by defining a summary table (see page 263) that DB2 automatically maintains. Subsequently, when the user writes the original GROUP BY against the source-data table, the optimizer substitutes with a much simpler (and faster) query against the summary table.
20
Index of Concepts
DB2 V9.7 Cookbook ©
Introduction to SQL This chapter contains a basic introduction to DB2 SQL. It also has numerous examples illustrating how to use this language to answer particular business problems. However, it is not meant to be a definitive guide to the language. Please refer to the relevant IBM manuals for a more detailed description. Syntax Diagram Conventions
This book uses railroad diagrams to describe the DB2 SQL statements. The following diagram shows the conventions used. Start
Continue
, Default
ALL
SELECT
an item
DISTINCT * Resume
Repeat
End
, FROM
table name view name Mandatory
WHERE
Optional
expression and / or
Figure 14, Syntax Diagram Conventions Rules
Upper Case text is a SQL keyword.
Italic text is either a placeholder, or explained elsewhere.
Backward arrows enable one to repeat parts of the text.
A branch line going above the main line is the default.
A branch line going below the main line is an optional item.
SQL Comments
A comment in a SQL statement starts with two dashes and goes to the end of the line: SELECT name FROM staff ORDER BY id;
-- this is a comment. -- this is another comment.
Figure 15, SQL Comment example Some DB2 command processors (e.g. DB2BATCH on the PC, or SPUFI on the mainframe) can process intelligent comments. These begin the line with a "--#SET" phrase, and then identify the value to be set. In the following example, the statement delimiter is changed using an intelligent comment: --#SET SELECT --#SET SELECT
DELIMITER name FROM DELIMITER name FROM
! staff WHERE id = 10! ; staff WHERE id = 20;
Figure 16, Set Delimiter example
Introduction to SQL
21
Graeme Birchall ©
When using the DB2 Command Processor (batch) script, the default statement terminator can be set using the "-tdx" option, where "x" is the value have chosen. NOTE: See the section titled Special Character Usage on page 44 for notes on how to refer to the statement delimiter in the SQL text. Statement Delimiter
DB2 SQL does not come with a designated statement delimiter (terminator), though a semicolon is often used. A semi-colon cannot be used when writing a compound SQL statement (see page 79) because that character is used to terminate the various sub-components of the statement.
SQL Components DB2 Objects
DB2 is a relational database that supports a variety of object types. In this section we shall overview those items which one can obtain data from using SQL. Table
A table is an organized set of columns and rows. The number, type, and relative position, of the various columns in the table is recorded in the DB2 catalogue. The number of rows in the table will fluctuate as data is inserted and deleted. The CREATE TABLE statement is used to define a table. The following example will define the EMPLOYEE table, which is found in the DB2 sample database. CREATE TABLE employee (empno CHARACTER (00006) ,firstnme VARCHAR (00012) ,midinit CHARACTER (00001) ,lastname VARCHAR (00015) ,workdept CHARACTER (00003) ,phoneno CHARACTER (00004) ,hiredate DATE ,job CHARACTER (00008) ,edlevel SMALLINT ,SEX CHARACTER (00001) ,birthdate DATE ,salary DECIMAL (00009,02) ,bonus DECIMAL (00009,02) ,comm DECIMAL (00009,02) ) DATA CAPTURE NONE;
NOT NOT NOT NOT
NULL NULL NULL NULL
NOT NULL
Figure 17, DB2 sample table - EMPLOYEE View
A view is another way to look at the data in one or more tables (or other views). For example, a user of the following view will only see those rows (and certain columns) in the EMPLOYEE table where the salary of a particular employee is greater than or equal to the average salary for their particular department.
22
SQL Components
DB2 V9.7 Cookbook ©
CREATE VIEW employee_view AS SELECT a.empno, a.firstnme, a.salary, a.workdept FROM employee a WHERE a.salary >= (SELECT AVG(b.salary) FROM employee b WHERE a.workdept = b.workdept);
Figure 18, DB2 sample view - EMPLOYEE_VIEW A view need not always refer to an actual table. It may instead contain a list of values: CREATE VIEW silly (c1, c2, c3) AS VALUES (11, 'AAA', SMALLINT(22)) ,(12, 'BBB', SMALLINT(33)) ,(13, 'CCC', NULL);
Figure 19, Define a view using a VALUES clause Selecting from the above view works the same as selecting from a table: SELECT c1, c2, c3 FROM silly ORDER BY c1 aSC;
ANSWER =========== C1 C2 C3 -- --- -11 AAA 22 12 BBB 33 13 CCC -
Figure 20, SELECT from a view that has its own data We can go one step further and define a view that begins with a single value that is then manipulated using SQL to make many other values. For example, the following view, when selected from, will return 10,000 rows. Note however that these rows are not stored anywhere in the database - they are instead created on the fly when the view is queried. CREATE VIEW test_data AS WITH temp1 (num1) AS (VALUES (1) UNION ALL SELECT num1 + 1 FROM temp1 WHERE num1 < 10000) SELECT * FROM temp1;
Figure 21, Define a view that creates data on the fly Alias
An alias is an alternate name for a table or a view. Unlike a view, an alias can not contain any processing logic. No authorization is required to use an alias other than that needed to access to the underlying table or view. CREATE ALIAS COMMIT;
employee_al1 FOR employee;
CREATE ALIAS COMMIT;
employee_al2 fOR employee_al1;
CREATE ALIAS COMMIT;
employee_al3 FOR employee_al2;
Figure 22, Define three aliases, the latter on the earlier Neither a view, nor an alias, can be linked in a recursive manner (e.g. V1 points to V2, which points back to V1). Also, both views and aliases still exist after a source object (e.g. a table) has been dropped. In such cases, a view, but not an alias, is marked invalid.
Introduction to SQL
23
Graeme Birchall ©
Nickname
A nickname is the name that one provides to DB2 for either a remote table, or a non-relational object that one wants to query as if it were a table. CREATE NICKNAME emp FOR unixserver.production.employee;
Figure 23, Define a nickname Tablesample
Use of the optional TABLESAMPLE reference enables one to randomly select (sample) some fraction of the rows in the underlying base table: SELECT FROM
* staff TABLESAMPLE BERNOULLI(10);
Figure 24, TABLESAMPLE example See page 396 for information on using the TABLESAMPLE feature. DB2 Data Types
DB2 comes with the following standard data types:
SMALLINT, INT, and BIGINT (i.e. integer numbers).
FLOAT, REAL, and DOUBLE (i.e. floating point numbers).
DECIMAL and NUMERIC (i.e. decimal numbers).
DECFLOAT (i.e. decimal floating-point numbers).
CHAR, VARCHAR, and LONG VARCHAR (i.e. character values).
GRAPHIC, VARGRAPHIC, and LONG VARGRAPHIC (i.e. graphical values).
BLOB, CLOB, and DBCLOB (i.e. binary and character long object values).
DATE, TIME, and TIMESTAMP (i.e. date/time values – see page: 25).
DATALINK (i.e. link to external object).
XML (i.e. contains well formed XML data).
Below is a simple table definition that uses some of the above data types: CREATE TABLE sales_record (sales# INTEGER NOT NULL GENERATED ALWAYS AS IDENTITY (START WITH 1 ,INCREMENT BY 1 ,NO MAXVALUE ,NO CYCLE) ,sale_ts TIMESTAMP NOT NULL ,num_items SMALLINT NOT NULL ,payment_type CHAR(2) NOT NULL ,sale_value DECIMAL(12,2) NOT NULL ,sales_tax DECIMAL(12,2) ,employee# INTEGER NOT NULL ,CONSTRAINT sales1 CHECK(payment_type IN ('CS','CR')) ,CONSTRAINT sales2 CHECK(sale_value > 0) ,CONSTRAINT sales3 CHECK(num_items > 0) ,CONSTRAINT sales4 FOREIGN KEY(employee#) REFERENCES staff(id) ON DELETE RESTRICT ,PRIMARY KEY(sales#));
Figure 25, Sample table definition
24
SQL Components
DB2 V9.7 Cookbook ©
In the above table, we have listed the relevant columns, and added various checks to ensure that the data is always correct. In particular, we have included the following:
The sales# is automatically generated (see page 277 for details). It is also the primary key of the table, and so must always be unique.
The payment-type must be one of two possible values.
Both the sales-value and the num-items must be greater than zero.
The employee# must already exist in the staff table. Furthermore, once a row has been inserted into this table, any attempt to delete the related row from the staff table will fail.
Default Lengths
The following table has two columns: CREATE TABLE default_values (c1 CHAR NOT NULL ,d1 DECIMAL NOT NULL);
Figure 26, Table with default column lengths The length has not been provided for either of the above columns. In this case, DB2 defaults to CHAR(1) for the first column and DECIMAL(5,0) for the second column. Data Type Usage
In general, use the standard DB2 data types as follows:
Always store monetary data in a decimal field.
Store non-fractional numbers in one of the integer field types.
Use floating-point when absolute precision is not necessary.
A DB2 data type is not just a place to hold data. It also defines what rules are applied when the data in manipulated. For example, storing monetary data in a DB2 floating-point field is a no-no, in part because the data-type is not precise, but also because a floating-point number is not manipulated (e.g. during division) according to internationally accepted accounting rules. DECFLOAT Arithmetic
DECFLOAT numbers have quite different processing characteristics from the other number types. For a start, they support more values:
Zero.
Negative and positive numbers (e.g. -1234.56).
Negative and positive infinity.
Negative and positive NaN (i.e. Not a Number).
Negative and positive sNaN (i.e. signaling Not a Number).
NaN Usage
The value NaN represents the result of an arithmetic operation that does not return a number (e.g. the square root of a negative number), but is also not infinity. For example, the expression 0/0 returns NaN, while 1/0 returns infinity.
Introduction to SQL
25
Graeme Birchall ©
The value NaN propagates through any arithmetic expression. Thus the final result is always either positive or negative NaN, as the following query illustrates: SELECT
FROM
DECFLOAT(+1.23) ,DECFLOAT(-1.23) ,DECFLOAT(-1.23) ,DECFLOAT(+infinity) ,DECFLOAT(+sNaN) ,DECFLOAT(-sNaN) ,DECFLOAT(+NaN) ,DECFLOAT(-NaN) sysibm.sysdummy1;
+ NaN + NaN + -NaN + NaN + NaN + NaN + NaN + NaN
AS AS AS AS AS AS AS AS
" " " " " " " "
NaN" NaN" -NaN" NaN" NaN" -NaN" NaN" -NaN"
Figure 27, NaN arithmetic usage NOTE: Any reference to a signaling NaN value in a statement (as above) will result in a warning message being generated. Infinity Usage
The value infinity works similar to NaN. Its reference in an arithmetic expression almost always returns either positive or negative infinity (assuming NaN is not also present). The one exception is division by infinity, which returns a really small, but still finite, number: SELECT
FROM
DECFLOAT(1) / +infinity ,DECFLOAT(1) * +infinity ,DECFLOAT(1) + +infinity ,DECFLOAT(1) - +infinity ,DECFLOAT(1) / -infinity ,DECFLOAT(1) * -infinity ,DECFLOAT(1) + -infinity ,DECFLOAT(1) - -infinity sysibm.sysdummy1;
AS AS AS AS AS AS AS AS
" 0E-6176" " Infinity" " Infinity" "-Infinity" " -0E-6176" "-Infinity" "-Infinity" " Infinity"
Figure 28, Infinity arithmetic usage The next query shows some situations where either infinity or NaN is returned: SELECT
FROM
DECFLOAT(+1.23) ,DECFLOAT(-1.23) ,DECFLOAT(+1.23) ,DECFLOAT(0) ,DECFLOAT(infinity) ,LOG(DECFLOAT(0)) ,LOG(DECFLOAT(-123)) ,SQRT(DECFLOAT(-123)) sysibm.sysdummy1;
/ / + / +
0 0 infinity 0 -infinity
AS AS AS AS AS AS AS AS
" Infinity" "-Infinity" " Infinity" " NaN" " NaN" "-Infinity" " NaN" " NaN"
Figure 29, DECFLOAT arithmetic results DECFLOAT Value Order
The DECFLOAT values have the following order, from low to high: -NaN
-sNan
-infinity
-1.2
-1.20
0
1.20
1.2
infinity
sNaN
NaN
Figure 30, DECFLOAT value order Please note that the numbers 1.2 and 1.200 are "equal", but they will be stored as different values, and will have a different value order. The TOTALORDER function can be used to illustrate this. It returns one of three values:
Zero if the two values have the same order.
+1 if the first value has a higher order (even if it is equal).
-1 if the first value has a lower order (even if it is equal).
26
SQL Components
DB2 V9.7 Cookbook ©
WITH temp1 (d1, d2) AS (VALUES (DECFLOAT(+1.0), ,(DECFLOAT(-1.0), ,(DECFLOAT(+0.0), ,(DECFLOAT(-0.0), ,(DECFLOAT(+0), ) SELECT TOTALORDER(d1,d2) FROM temp1;
DECFLOAT(+1.00)) DECFLOAT(-1.00)) DECFLOAT(+0.00)) DECFLOAT(-0.00)) DECFLOAT(-0))
ANSWER ====== 1 -1 1 1 0
Figure 31, Equal values that may have different orders The NORMALIZE_DECFLOAT scalar function can be used to strip trailing zeros from a DECFLOAT value: WITH temp1 (d1) AS (VALUES (DECFLOAT(+0 ,16)) ,(DECFLOAT(+0.0 ,16)) ,(DECFLOAT(+0.00 ,16)) ,(DECFLOAT(+0.000 ,16)) ) SELECT d1 ,HEX(d1) ,NORMALIZE_DECFLOAT(d1) ,HEX(NORMALIZE_DECFLOAT(d1)) FROM temp1;
AS hex_d1 AS d2 AS hex_d2
ANSWER ========================================== D1 HEX_D1 D2 HEX_D2 ----- ---------------- -- ---------------0 0000000000003822 0 0000000000003822 0.0 0000000000003422 0 0000000000003822 0.00 0000000000003022 0 0000000000003822 0.000 0000000000002C22 0 0000000000003822
Figure 32, Remove trailing zeros DECFLOAT Scalar Functions
The following scalar functions support the DECFLOAT data type:
COMPARE_DECFLOAT: Compares order of two DECFLOAT values.
DECFLOAT: Converts input value to DECFLOAT.
NORMALIZE_DECFLOAT: Removes trailing blanks from DECFLOAT value.
QUANTIZE: Converts number to DECFLOAT, using mask to define precision.
TOTALORDER: Compares order of two DECFLOAT values.
Date/Time Arithmetic
Manipulating date/time values can sometimes give unexpected results. What follows is a brief introduction to the subject. The basic rules are:
Multiplication and division is not allowed.
Subtraction is allowed using date/time values, date/time durations, or labeled durations.
Addition is allowed using date/time durations, or labeled durations.
The valid labeled durations are listed below:
Introduction to SQL
27
Graeme Birchall ©
LABELED DURATIONS SINGULAR PLURAL =========== ============ YEAR YEARS MONTH MONTHS DAY DAYS HOUR HOURS MINUTE MINUTES SECOND SECONDS MICROSECOND MICROSECONDS
ITEM FIXED SIZE ===== N N Y Y Y Y Y
WORKS WITH DATE/TIME DATE TIME TIMESTAMP ==== ==== ========= Y Y Y Y Y Y Y Y Y Y Y Y Y Y
Figure 33, Labeled Durations and Date/Time Types Usage Notes
It doesn't matter if one uses singular or plural. One can add "4 day" to a date.
Some months and years are longer than others. So when one adds "2 months" to a date the result is determined, in part, by the date that you began with. More on this below.
One cannot add "minutes" to a date, or "days" to a time, etc.
One cannot combine labeled durations in parenthesis: "date - (1 day + 2 months)" will fail. One should instead say: "date - 1 day - 2 months".
Adding too many hours, minutes or seconds to a time will cause it to wrap around. The overflow will be lost.
Adding 24 hours to the time '00.00.00' will get '24.00.00'. Adding 24 hours to any other time will return the original value.
When a decimal value is used (e.g. 4.5 days) the fractional part is discarded. So to add (to a timestamp value) 4.5 days, add 4 days and 12 hours.
Now for some examples: SELECT
FROM WHERE AND
sales_date ,sales_date ,sales_date ,sales_date ,sales_date
- 10 DAY AS + -1 MONTH AS + 99 YEARS AS + 55 DAYS - 22 MONTHS AS ,sales_date + (4+6) DAYS AS sales sales_person = 'GOUNOT' sales_date = '1995-12-31'
d1 d2 d3
n" "c>n>c" "nl"
ANSWER ================= C C>N C>N>C NL - --- ----- -A 65 A ÿ
Figure 355, CHR function examples NOTE: At present, the CHR function has a bug that results in it not returning a null value when the input value is greater than 255.
CLOB
Converts the input (1st argument) to a CLOB. The output length (2nd argument) is optional. If the input is truncated during conversion, a warning message is issued. For example, in the following example the second CLOB statement will induce a warning for the first two lines of input because they have non-blank data after the third byte: SELECT c1 ,CLOB(c1) AS cc1 ,CLOB(c1,3) AS cc2 FROM scalar;
ANSWER =================== C1 CC1 CC2 ------ ------ --ABCDEF ABCDEF ABC ABCD ABCD ABC AB AB AB
Figure 356, CLOB function examples NOTE: The DB2BATCH command processor dies a nasty death whenever it encounters a CLOB field in the output. If possible, convert to VARCHAR first to avoid this problem.
COALESCE
Returns the first non-null value in a list of input expressions (reading from left to right). Each expression is separated from the prior by a comma. All input expressions must be compatible. VALUE is a synonym for COALESCE. SELECT
id ,comm ,COALESCE(comm,0) FROM staff WHERE id < 30 ORDER BY id;
ANSWER ================== ID COMM 3 -- ------ -----10 0.00 20 612.45 612.45
Figure 357, COALESCE function example A CASE expression can be written to do exactly the same thing as the COALESCE function. The following SQL statement shows two logically equivalent ways to replace nulls:
136
Scalar Functions, Definitions
DB2 V9.7 Cookbook ©
WITH temp1(c1,c2,c3) AS (VALUES (CAST(NULL AS SMALLINT) ,CAST(NULL AS SMALLINT) ,CAST(10 AS SMALLINT))) SELECT COALESCE(c1,c2,c3) AS cc1 ,CASE WHEN c1 IS NOT NULL THEN c1 WHEN c2 IS NOT NULL THEN c2 WHEN c3 IS NOT NULL THEN c3 END AS cc2 FROM TEMP1;
ANSWER ======== CC1 CC2 --- --10 10
Figure 358, COALESCE and equivalent CASE expression Be aware that a field can return a null value, even when it is defined as not null. This occurs if a column function is applied against the field, and no row is returned: SELECT COUNT(*) AS #rows ,MIN(id) AS min_id ,COALESCE(MIN(id),-1) AS ccc_id FROM staff WHERE id < 5;
ANSWER =================== #ROWS MIN_ID CCC_ID ----- ------ -----0 -1
Figure 359, NOT NULL field returning null value COLLATION_KEY_BIT
Returns a VARCHAR FOR BIT DATA string that is the collation sequence of the first argument in the function. There three parameters:
String to be evaluated.
Collation sequence to use (must be valid).
Length of output (optional).
The following query displays three collation sequences:
All flavors of a given character as the same (i.e. "a" = "A" = "Ä").
Upper and lower case characters are equal, but sort lower than accented characters.
All variations of a character have a different collation value.
Now for the query: WITH temp1 (c1) As (VALUES ('a'),('A'),('Á'),('Ä'),('b')) SELECT c1 ,COLLATION_KEY_BIT(c1,'UCA400R1_S1',9) AS "a=A=Á=Ä" ,COLLATION_KEY_BIT(c1,'UCA400R1_S2',9) AS "a=A=" predicate? As is shown below, both of our test queries treat an empty set as a match: SELECT * FROM table1 WHERE NOT EXISTS (SELECT * FROM table2 WHERE t2c >= 'X' AND t2c t1a);
ANSWERS ======= T1A T1B --- --A AA B BB C CC
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
SELECT * FROM table1 WHERE t1a = ALL (SELECT t2c FROM table2 WHERE t2c >= 'X');
Figure 699, NOT EXISTS vs. ALL, ignore nulls, no match
Sub-Query
249
Graeme Birchall ©
One might think that the above two queries are logically equivalent, but they are not. As is shown below, they return different results when the sub-query answer set can include nulls: SELECT * FROM table1 WHERE NOT EXISTS (SELECT * FROM table2 WHERE t2c t1a);
ANSWER ======= T1A T1B --- --A AA
SELECT * FROM table1 WHERE t1a = ALL (SELECT t2c FROM table2);
ANSWER ======= no rows
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 700, NOT EXISTS vs. ALL, process nulls A sub-query can only return true or false, but a DB2 field value can either match (i.e. be true), or not match (i.e. be false), or be unknown. It is the differing treatment of unknown values that is causing the above two queries to differ:
In the ALL sub-query, each value in T1A is checked against all of the values in T2C. The null value is checked, deemed to differ, and so the sub-query always returns false.
In the NOT EXISTS sub-query, each value in T1A is used to find those T2C values that are not equal. For the T1A values "B" and "C", the T2C value "A" does not equal, so the NOT EXISTS check will fail. But for the T1A value "A", there are no "not equal" values in T2C, because a null value does not "not equal" a literal. So the NOT EXISTS check will pass.
The following three queries list those T2C values that do "not equal" a given T1A value: SELECT * FROM table2 WHERE t2c 'A';
SELECT * FROM table2 WHERE t2c 'B';
SELECT * FROM table2 WHERE t2c 'C';
ANSWER =========== T2A T2B T2C --- --- --no rows
ANSWER =========== T2A T2B T2C --- --- --A A A
ANSWER =========== T2A T2B T2C --- --- --A A A
Figure 701, List of values in T2C T1A value To make a NOT EXISTS sub-query that is logically equivalent to the ALL sub-query that we have used above, one can add an additional check for null T2C values: SELECT * FROM table1 WHERE NOT EXISTS (SELECT * FROM table2 WHERE t2c t1a OR t2c IS NULL);
ANSWER ======= no rows
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 702, NOT EXISTS - same as ALL One problem with the above query is that it is not exactly obvious. Another is that the two T2C predicates will have to be fenced in with parenthesis if other predicates (on TABLE2) exist. For these reasons, use an ALL sub-query when that is what you mean to do.
250
Sub-query Flavors
DB2 V9.7 Cookbook ©
IN Keyword Sub-Query
The IN sub-query check is similar to the ANY and SOME checks:
If any row in the sub-query result matches, the answer is true.
If the sub-query result is empty, the answer is false.
If no row in the sub-query result matches, the answer is also false.
If all of the values in the sub-query result are null, the answer is false.
Below is an example that compares the T1A and T2A columns. Two rows match: SELECT * FROM table1 WHERE t1a IN (SELECT t2a FROM table2);
ANSWER ======= T1A T1B --- -A AA B BB
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 703, IN sub-query example, two matches In the next example, no rows match because the sub-query result is an empty set: SELECT * FROM table1 WHERE t1a IN (SELECT t2a FROM table2 WHERE t2a >= 'X');
ANSWER ====== 0 rows
Figure 704, IN sub-query example, no matches The IN, ANY, SOME, and ALL checks all look for a match. Because one null value does not equal another null value, having a null expression in the "top" table causes the sub-query to always returns false: SELECT * FROM table2 WHERE t2c IN (SELECT t2c FROM table2);
ANSWERS =========== T2A T2B T2C --- --- --A A A
SELECT * FROM table2 WHERE t2c = ANY (SELECT t2c FROM table2);
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 705, IN and = ANY sub-query examples, with nulls NOT IN Keyword Sub-Queries
Sub-queries that look for the non-existence of a row work largely as one would expect, except when a null value in involved. To illustrate, consider the following query, where we want to see if the current T1A value is not in the set of T2C values: SELECT * FROM table1 WHERE t1a NOT IN (SELECT t2c FROM table2);
ANSWER ====== 0 rows
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 706, NOT IN sub-query example, no matches
Sub-Query
251
Graeme Birchall ©
Observe that the T1A values "B" and "C" are obviously not in T2C, yet they are not returned. The sub-query result set contains the value null, which causes the NOT IN check to return unknown, which equates to false. The next example removes the null values from the sub-query result, which then enables the NOT IN check to find the non-matching values: SELECT * FROM table1 WHERE t1a NOT IN (SELECT t2c FROM table2 WHERE t2c IS NOT NULL);
ANSWER ======= T1A T1B --- -B BB C CC
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 707, NOT IN sub-query example, matches Another way to find the non-matching values while ignoring any null rows in the sub-query, is to use an EXISTS check in a correlated sub-query: SELECT * FROM table1 WHERE NOT EXISTS (SELECT * FROM table2 WHERE t1a = t2c);
ANSWER ======= T1A T1B --- -B BB C CC
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 708, NOT EXISTS sub-query example, matches Correlated vs. Uncorrelated Sub-Queries
An uncorrelated sub-query is one where the predicates in the sub-query part of SQL statement have no direct relationship to the current row being processed in the "top" table (hence uncorrelated). The following sub-query is uncorrelated: SELECT * FROM table1 WHERE t1a IN (SELECT t2a FROM table2);
ANSWER ======= T1A T1B --- -A AA B BB
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 709, Uncorrelated sub-query A correlated sub-query is one where the predicates in the sub-query part of the SQL statement cannot be resolved without reference to the row currently being processed in the "top" table (hence correlated). The following query is correlated: SELECT * FROM table1 WHERE t1a IN (SELECT t2a FROM table2 WHERE t1a = t2a);
ANSWER ======= T1A T1B --- -A AA B BB
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 710, Correlated sub-query Below is another correlated sub-query. Because the same table is being referred to twice, correlation names have to be used to delineate which column belongs to which table:
252
Sub-query Flavors
DB2 V9.7 Cookbook ©
SELECT * FROM table2 WHERE EXISTS (SELECT FROM WHERE
ANSWER =========== T2A T2B T2C --- --- --A A A
aa * table2 bb aa.t2a = bb.t2b);
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 711,Correlated sub-query, with correlation names Which is Faster
In general, if there is a suitable index on the sub-query table, use a correlated sub-query. Else, use an uncorrelated sub-query. However, there are several very important exceptions to this rule, and some queries can only be written one way. NOTE: The DB2 optimizer is not as good at choosing the best access path for sub-queries as it is with joins. Be prepared to spend some time doing tuning.
Multi-Field Sub-Queries
Imagine that you want to compare multiple items in your sub-query. The following examples use an IN expression and a correlated EXISTS sub-query to do two equality checks: SELECT * FROM table1 WHERE (t1a,t1b) IN (SELECT t2a, t2b FROM table2);
SELECT * FROM table1 WHERE EXISTS (SELECT FROM WHERE AND
ANSWER ====== 0 rows
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
ANSWER ====== 0 rows * table2 t1a = t2a t1b = t2b);
Figure 712, Multi-field sub-queries, equal checks Observe that to do a multiple-value IN check, you put the list of expressions to be compared in parenthesis, and then select the same number of items in the sub-query. An IN phrase is limited because it can only do an equality check. By contrast, use whatever predicates you want in an EXISTS correlated sub-query to do other types of comparison: SELECT * FROM table1 WHERE EXISTS (SELECT FROM WHERE AND
* table2 t1a = t2a t1b >= t2b);
ANSWER ======= T1A T1B --- -A AA B BB
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null
Figure 713, Multi-field sub-query, with non-equal check Nested Sub-Queries
Some business questions may require that the related SQL statement be written as a series of nested sub-queries. In the following example, we are after all employees in the EMPLOYEE table who have a salary that is greater than the maximum salary of all those other employees that do not work on a project with a name beginning 'MA'.
Sub-Query
253
Graeme Birchall ©
SELECT empno ,lastname ,salary FROM employee WHERE salary > (SELECT MAX(salary) FROM employee WHERE empno NOT IN (SELECT empno FROM emp_act WHERE projno LIKE 'MA%')) ORDER BY 1;
ANSWER ========================= EMPNO LASTNAME SALARY ------ --------- -------000010 HAAS 52750.00 000110 LUCCHESSI 46500.00
Figure 714, Nested Sub-Queries
Usage Examples In this section we will use various sub-queries to compare our two test tables - looking for those rows where none, any, ten, or all values match. Beware of Nulls
The presence of null values greatly complicates sub-query usage. Not allowing for them when they are present can cause one to get what is arguably a wrong answer. And do not assume that just because you don't have any nullable fields that you will never therefore encounter a null value. The DEPTNO table in the Department table is defined as not null, but in the following query, the maximum DEPTNO that is returned will be null: SELECT
COUNT(*) AS #rows ,MAX(deptno) AS maxdpt FROM department WHERE deptname LIKE 'Z%' ORDER BY 1;
ANSWER ============= #ROWS MAXDEPT ----- ------0 null
Figure 715, Getting a null value from a not null field True if NONE Match
Find all rows in TABLE1 where there are no rows in TABLE2 that have a T2C value equal to the current T1A value in the TABLE1 table: SELECT * FROM table1 WHERE 0 = (SELECT FROM WHERE
t1 COUNT(*) table2 t2 t1.t1a = t2.t2c);
SELECT * FROM table1 t1 WHERE NOT EXISTS (SELECT * FROM table2 t2 WHERE t1.t1a = t2.t2c); SELECT * FROM table1 WHERE t1a NOT IN (SELECT t2c FROM table2 WHERE t2c IS NOT NULL);
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null ANSWER ======= T1A T1B --- --B BB C CC
Figure 716, Sub-queries, true if none match
254
Usage Examples
DB2 V9.7 Cookbook ©
Observe that in the last statement above we eliminated the null rows from the sub-query. Had this not been done, the NOT IN check would have found them and then returned a result of "unknown" (i.e. false) for all of rows in the TABLE1A table. Using a Join
Another way to answer the same problem is to use a left outer join, going from TABLE1 to TABLE2 while matching on the T1A and T2C fields. Get only those rows (from TABLE1) where the corresponding T2C value is null: SELECT t1.* FROM table1 t1 LEFT OUTER JOIN table2 t2 ON t1.t1a = t2.t2c WHERE t2.t2c IS NULL;
ANSWER ======= T1A T1B --- --B BB C CC
Figure 717, Outer join, true if none match True if ANY Match
Find all rows in TABLE1 where there are one, or more, rows in TABLE2 that have a T2C value equal to the current T1A value: SELECT * FROM table1 WHERE EXISTS (SELECT FROM WHERE SELECT * FROM table1 WHERE 1 = 'X');
ANSWER ======= T1A T1B --- --A AA B BB C CC
SELECT * FROM table1 WHERE NOT EXISTS (SELECT * FROM table2 WHERE t1a t2b AND t2b >= 'X');
Figure 724, Sub-queries, true if all match, empty set False if no Matching Rows
The next two queries differ from the above in how they address empty sets. The queries will return a row from TABLE1 if the current T1A value matches all of the T2B values found in the sub-query, but they will not return a row if no matching values are found: SELECT * FROM table1 WHERE t1a = ALL (SELECT t2b FROM table2 WHERE t2b >= 'X') AND 0 (SELECT COUNT(*) FROM table2 WHERE t2b >= 'X'); SELECT * FROM table1 WHERE t1a IN (SELECT FROM WHERE HAVING
TABLE1 +-------+ |T1A|T1B| |---|---| |A |AA | |B |BB | |C |CC | +-------+
TABLE2 +-----------+ |T2A|T2B|T2C| |---|---|---| |A |A |A | |B |A | - | +-----------+ "-" = null ANSWER ====== 0 rows
MAX(t2b) table2 t2b >= 'X' COUNT(DISTINCT t2b) = 1);
Figure 725, Sub-queries, true if all match, and at least one value found Both of the above statements have flaws: The first processes the TABLE2 table twice, which not only involves double work, but also requires that the sub-query predicates be duplicated. The second statement is just plain strange.
258
Usage Examples
DB2 V9.7 Cookbook ©
Union, Intersect, and Except A UNION, EXCEPT, or INTERCEPT expression combines sets of columns into new sets of columns. An illustration of what each operation does with a given set of data is shown below: R1 UNION R2 R1 -A A A B B C C C E
R2 -A A B B B C D
----A B C D E
R1 UNION ALL R2 ----A A A A A B B B B B C C C C D E
R1 INTERSECT R2 --------A B C
R1 INTERSECT ALL R2 ----A A B B C
R1 EXCEPT R2 -----E
R1 EXCEPT ALL R2 -----A C C E
R1 MINUS R2 ----E
Figure 726, Examples of Union, Except, and Intersect WARNING: Unlike the UNION and INTERSECT operations, the EXCEPT statement is not commutative. This means that "A EXCEPT B" is not the same as "B EXCEPT A".
Syntax Diagram SELECT statement VALUES statement
UNION UNION ALL INTERSECT INTERSECT ALL EXCEPT EXCEPT ALL MINUS
SELECT statement VALUES statement
Figure 727, Union, Except, and Intersect syntax Sample Views CREATE VIEW AS VALUES CREATE VIEW AS VALUES
R1 (R1) ('A'),('A'),('A'),('B'),('B'),('C'),('C'),('C'),('E'); R2 (R2) ('A'),('A'),('B'),('B'),('B'),('C'),('D'); ANSWER ====== SELECT R1 R1 R2 FROM R1 -- -ORDER BY R1; A A A A SELECT R2 A B FROM R2 B B ORDER BY R2; B B C C C D C E
Figure 728, Query sample views
Union, Intersect, and Except
259
Graeme Birchall ©
Usage Notes Union & Union All
A UNION operation combines two sets of columns and removes duplicates. The UNION ALL expression does the same but does not remove the duplicates. SELECT FROM UNION SELECT FROM ORDER BY
R1 R1
R1 -A A A B B C C C E
R2 R2 1;
SELECT R1 FROM R1 UNION ALL SELECT R2 FROM R2 ORDER BY 1;
R2 -A A B B B C D
UNION ===== A B C D E
UNION ALL ========= A A A A A B B B B B C C C C D E
Figure 729, Union and Union All SQL NOTE: Recursive SQL requires that there be a UNION ALL phrase between the two main parts of the statement. The UNION ALL, unlike the UNION, allows for duplicate output rows which is what often comes out of recursive processing.
Intersect & Intersect All
An INTERSECT operation retrieves the matching set of distinct values (not rows) from two columns. The INTERSECT ALL returns the set of matching individual rows. SELECT R1 FROM R1 INTERSECT SELECT R2 FROM R2 ORDER BY 1; SELECT R1 FROM R1 INTERSECT ALL SELECT R2 FROM R2 ORDER BY 1;
R1 -A A A B B C C C E
R2 -A A B B B C D
INTERSECT ========= A B C
INTERSECT ALL ============= A A B B C
Figure 730, Intersect and Intersect All SQL An INTERSECT and/or EXCEPT operation is done by matching ALL of the columns in the top and bottom result-sets. In other words, these are row, not column, operations. It is not possible to only match on the keys, yet at the same time, also fetch non-key columns. To do this, one needs to use a sub-query. Except, Except All, & Minus
An EXCEPT operation retrieves the set of distinct data values (not rows) that exist in the first the table but not in the second. The EXCEPT ALL returns the set of individual rows that exist only in the first table. The word MINUS is a synonym for EXCEPT.
260
Usage Notes
DB2 V9.7 Cookbook ©
SELECT FROM EXCEPT SELECT FROM ORDER BY
R1 R1 R1 -A A A B B C C C E
R2 R2 1;
SELECT R1 FROM R1 EXCEPT ALL SELECT R2 FROM R2 ORDER BY 1;
R2 -A A B B B C D
R1 EXCEPT R2 ===== E
R1 EXCEPT ALL R2 ========== A C C E
Figure 731, Except and Except All SQL (R1 on top) Because the EXCEPT/MINUS operation is not commutative, using it in the reverse direction (i.e. R2 to R1 instead of R1 to R2) will give a different result: SELECT FROM EXCEPT SELECT FROM ORDER BY
R2 R2 R1 -A A A B B C C C E
R1 R1 1;
SELECT R2 FROM R2 EXCEPT ALL SELECT R1 FROM R1 ORDER BY 1;
R2 -A A B B B C D
R2 EXCEPT R1 ===== D
R2 EXCEPT ALL R1 ========== B D
Figure 732, Except and Except All SQL (R2 on top) NOTE: Only the EXCEPT/MINUS operation is not commutative. Both the UNION and the INTERSECT operations work the same regardless of which table is on top or on bottom.
Precedence Rules
When multiple operations are done in the same SQL statement, there are precedence rules:
Operations in parenthesis are done first.
INTERSECT operations are done before either UNION or EXCEPT.
Operations of equal worth are done from top to bottom.
The next example illustrates how parenthesis can be used change the processing order: SELECT FROM UNION SELECT FROM EXCEPT SELECT FROM ORDER BY ANSWER ====== E
R1 R1 R2 R2 R2 R2 1;
(SELECT FROM UNION SELECT FROM )EXCEPT SELECT FROM ORDER BY
R1 R1 R2 R2 R2 R2 1;
ANSWER ====== E
SELECT FROM UNION (SELECT FROM EXCEPT SELECT FROM )ORDER BY
R1 R1 R2 R2 R2 R2 1;
R1 -A A A B B C C C E
R2 -A A B B B C D
ANSWER ====== A B C E
Figure 733, Use of parenthesis in Union
Union, Intersect, and Except
261
Graeme Birchall ©
Unions and Views
Imagine that one has a series of tables that track sales data, with one table for each year. One can define a view that is the UNION ALL of these tables, so that a user would see them as a single object. Such a view can support inserts, updates, and deletes, as long as each table in the view has a constraint that distinguishes it from all the others. Below is an example: CREATE TABLE sales_data_2002 (sales_date DATE NOT NULL ,daily_seq# INTEGER NOT NULL ,cust_id INTEGER NOT NULL ,amount DEC(10,2) NOT NULL ,invoice# INTEGER NOT NULL ,sales_rep CHAR(10) NOT NULL ,CONSTRAINT C CHECK (YEAR(sales_date) = 2002) ,PRIMARY KEY (sales_date, daily_seq#)); CREATE TABLE sales_data_2003 (sales_date DATE NOT NULL ,daily_seq# INTEGER NOT NULL ,cust_id INTEGER NOT NULL ,amount DEC(10,2) NOT NULL ,invoice# INTEGER NOT NULL ,sales_rep CHAR(10) NOT NULL ,CONSTRAINT C CHECK (YEAR(sales_date) = 2003) ,PRIMARY KEY (sales_date, daily_seq#)); CREATE VIEW sales_data AS SELECT * FROM sales_data_2002 UNION ALL SELECT * FROM sales_data_2003;
Figure 734, Define view to combine yearly tables Below is some SQL that changes the contents of the above view: INSERT INTO sales_data VALUES ('2002-11-22',1,123,100.10,996,'SUE') ,('2002-11-22',2,123,100.10,997,'JOHN') ,('2003-01-01',1,123,100.10,998,'FRED') ,('2003-01-01',2,123,100.10,999,'FRED'); UPDATE sales_data SET amount = amount / 2 WHERE sales_rep = 'JOHN'; DELETE FROM sales_data WHERE sales_date = '2003-01-01' AND daily_seq# = 2;
Figure 735, Insert, update, and delete using view Below is the view contents, after the above is run: SALES_DATE ---------01/01/2003 11/22/2002 11/22/2002
DAILY_SEQ# ---------1 1 2
CUST_ID ------123 123 123
AMOUNT -----100.10 100.10 50.05
INVOICE# -------998 996 997
SALES_REP --------FRED SUE JOHN
Figure 736, View contents after insert, update, delete
262
Usage Notes
DB2 V9.7 Cookbook ©
Materialized Query Tables Introduction
A materialized query table contains the results of a query. The DB2 optimizer knows this and can, if appropriate, redirect a query that is against the source table(s) to use the materialized query table instead. This can make the query run much faster. The following statement defines a materialized query table: CREATE TABLE staff_summary AS (SELECT dept ,COUNT(*) AS count_rows ,SUM(id) AS sum_id FROM staff GROUP BY dept) DATA INITIALLY DEFERRED REFRESH IMMEDIATE;
Figure 737, Sample materialized query table DDL Below on the left is a query that is very similar to the one used in the above CREATE. The DB2 optimizer can convert this query into the optimized equivalent on the right, which uses the materialized query table. Because (in this case) the data in the materialized query table is maintained in sync with the source table, both statements will return the same answer. ORIGINAL QUERY ============== SELECT dept ,AVG(id) FROM staff GROUP BY dept
OPTIMIZED QUERY ================================= SELECT Q1.dept AS "dept" ,Q1.sum_id / Q1.count_rows FROM staff_summary AS Q1
Figure 738, Original and optimized queries When used appropriately, materialized query tables can cause dramatic improvements in query performance. For example, if in the above STAFF table there was, on average, about 5,000 rows per individual department, referencing the STAFF_SUMMARY table instead of the STAFF table in the sample query might be about 1,000 times faster. DB2 Optimizer Issues
In order for a materialized query table to be considered for use by the DB2 optimizer, the following has to be true:
The table has to be refreshed at least once.
The table MAINTAINED BY parameter and the related DB2 special registers must correspond. For example, if the table is USER maintained, then the CURRENT REFRESH AGE special register must be set to ANY, and the CURRENT MAINTAINED TABLE TYPES FOR OPTIMIZATION special register must be set to USER or ALL.
See page 266 for more details on these registers.
Usage Notes A materialized query table is defined using a variation of the standard CREATE TABLE statement. Instead of providing an element list, one supplies a SELECT statement, and defines the refresh option.
Materialized Query Tables
263
Graeme Birchall ©
CREATE
TABLE
table-name
AS
SUMMARY ( select stmt )
DATA INITIALLY DEFERRED
REFRESH
DEFERRED IMMEDIATE
ENABLE QUERY OPTIMIZATION DISABLE QUERY OPTIMIZATION MAINTAINED BY SYSTEM MAINTAINED BY
USER FEDERATED_TOOOL
Figure 739, Materialized query table DDL, syntax diagram Syntax Options Refresh
REFRESH DEFERRED: The data is refreshed whenever one does a REFRESH TABLE. At this point, DB2 will first delete all of the existing rows in the table, then run the select statement defined in the CREATE to (you guessed it) repopulate.
REFRESH IMMEDIATE: Once created, this type of table has to be refreshed once using the REFRESH statement. From then on, DB2 will maintain the materialized query table in sync with the source table as changes are made to the latter.
Materialized query tables that are defined REFRESH IMMEDIATE are obviously more useful in that the data in them is always current. But they may cost quite a bit to maintain, and not all queries can be defined thus. Query Optimization
ENABLE: The table is used for query optimization when appropriate. This is the default. The table can also be queried directly.
DISABLE: The table will not be used for query optimization. It can be queried directly.
Maintained By
SYSTEM: The data in the materialized query table is maintained by the system. This is the default.
USER: The user is allowed to perform insert, update, and delete operations against the materialized query table. The table cannot be refreshed. This type of table can be used when you want to maintain your own materialized query table (e.g. using triggers) to support features not provided by DB2. The table can also be defined to enable query optimization, but the optimizer will probably never use it as a substitute for a real table.
FEDERATED_TOOL: The data in the materialized query table is maintained by the replication tool. Only a REFRESH DEFERRED table can be maintained using this option.
Options vs. Actions
The following table compares materialized query table options to subsequent actions:
264
Usage Notes
DB2 V9.7 Cookbook ©
MATERIALIZED QUERY TABLE ========================== REFRESH MAINTAINED BY ========= ============= DEFERRED SYSTEM USER IMMEDIATE SYSTEM
ALLOWABLE ACTIONS ON TABLE ===================================== REFRESH TABLE INSERT/UPDATE/DELETE ============= ==================== yes no no yes yes no
Figure 740, Materialized query table options vs. allowable actions Select Statement
Various restrictions apply to the select statement that is used to define the materialized query table. In general, materialized query tables defined refresh-immediate need simpler queries than those defined refresh-deferred. Refresh Deferred Tables
The query must be a valid SELECT statement.
Every column selected must have a name.
An ORDER BY is not allowed.
Reference to a typed table or typed view is not allowed.
Reference to declared temporary table is not allowed.
Reference to a nickname or materialized query table is not allowed.
Reference to a system catalogue table is not allowed. Reference to an explain table is allowed, but is impudent.
Reference to NODENUMBER, PARTITION, or any other function that depends on physical characteristics, is not allowed.
Reference to a datalink type is not allowed.
Functions that have an external action are not allowed.
Scalar functions, or functions written in SQL, are not allowed. So SUM(SALARY) is fine, but SUM(INT(SALARY)) is not allowed.
Refresh Immediate Tables
All of the above restrictions apply, plus the following:
If the query references more than one table or view, it must define as inner join, yet not use the INNER JOIN syntax (i.e. must use old style).
If there is a GROUP BY, the SELECT list must have a COUNT(*) or COUNT_BIG(*) column.
Besides the COUNT and COUNT_BIG, the only other column functions supported are SUM and GROUPING - all with the DISTINCT phrase. Any field that allows nulls, and that is summed, but also have a COUNT(column name) function defined.
Any field in the GROUP BY list must be in the SELECT list.
The table must have at least one unique index defined, and the SELECT list must include (amongst other things) all the columns of this index.
Materialized Query Tables
265
Graeme Birchall ©
Grouping sets, CUBE an ROLLUP are allowed. The GROUP BY items and associated GROUPING column functions in the select list must for a unique key of the result set.
The HAVING clause is not allowed.
The DISTINCT clause is not allowed.
Non-deterministic functions are not allowed.
Special registers are not allowed.
If REPLICATED is specified, the table must have a unique key.
Optimizer Options
A materialized query table that has been defined ENABLE QUERY OPTIMIZATION, and has been refreshed, is a candidate for use by the DB2 optimizer if, and only if, three DB2 special registers are set to match the table status:
CURRENT MAINTAINED TABLE TYPES FOR OPTIMIZATION.
CURRENT QUERY OPTIMIZATION.
CURRENT REFRESH AGE.
Each of the above are discussed below. CURRENT REFRESH AGE
The refresh age special register tells the DB2 optimizer how up-to-date the data in an materialized query table has to be in order to be considered. There are only two possible values:
0: Only use those materialized query tables that are defined as refresh-immediate are eligible. This is the default.
99,999,999,999,999: Consider all valid materialized query tables. This is the same as ANY. NOTE: The above number is a 26-digit decimal value that is a timestamp duration, but without the microsecond component. The value ANY is logically equivalent.
The database default value can be changed using the following command: UPDATE DATABASE CONFIGURATION USING dft_refresh_age ANY;
Figure 741, Changing default refresh age for database The database default value can be overridden within a thread using the SET REFRESH AGE statement. Here is the syntax: SET CURRENT REFRESH AGE
=
number ANY host-var
Figure 742, Set refresh age command, syntax Below are some examples of the SET command: SET CURRENT REFRESH AGE 0; SET CURRENT REFRESH AGE = ANY; SET CURRENT REFRESH AGE = 99999999999999;
Figure 743, Set refresh age command, examples
266
Usage Notes
DB2 V9.7 Cookbook ©
CURRENT MAINTAINED TYPES
The current maintained types special register tells the DB2 optimizer what types of materialized query table that are defined refresh deferred are to be considered - assuming that the refresh-age parameter is not set to zero:
ALL: All refresh-deferred materialized query tables are to be considered. If this option is chosen, no other option can be used.
NONE: No refresh-deferred materialized query tables are to be considered. If this option is chosen, no other option can be used.
SYSTEM: System-maintained refresh-deferred materialized query tables are to be considered. This is the default.
USER: User-maintained refresh-deferred materialized query tables are to be considered.
FEDERATED TOOL: Federated-tool-maintained refresh-deferred materialized query tables are to be considered, but only if the CURRENT QUERY OPTIMIZATION special register is 2 or greater than 5.
CURRENT MAINTAINED TABLE TYPES FOR OPTIMIZATION: The existing values for this special register are used.
The database default value can be changed using the following command: UPDATE DATABASE CONFIGURATION USING dft_refresh_age ANY;
Figure 744, Changing default maintained type for database The database default value can be overridden within a thread using the SET REFRESH AGE statement. Here is the syntax: SET CURRENT MAINTAINED =
TABLE
TYPES
FOR OPTIMIZATION
ALL NONE ALL , FEDERATED_TOOL SYSTEM USER
FOR OPTIMIZATION TABLE
CURRENT MAINTANED
TYPES
Figure 745,Set maintained type command, syntax Below are some examples of the SET command: SET CURRENT MAINTAINED TYPES = ALL; SET CURRENT MAINTAINED TABLE TYPES = SYSTEM; SET CURRENT MAINTAINED TABLE TYPES FOR OPTIMIZATION = USER, SYSTEM;
Figure 746, Set maintained type command, examples CURRENT QUERY OPTIMIZATION
The current query optimization special register tells the DB2 optimizer what set of optimization techniques to use. The value can range from zero to nine - except for four or eight. A value of five or above will cause the optimizer to consider using materialized query tables.
Materialized Query Tables
267
Graeme Birchall ©
The database default value can be changed using the following command: UPDATE DATABASE CONFIGURATION USING DFT_QUERYOPT 5;
Figure 747, Changing default maintained type for database The database default value can be overridden within a thread using the SET CURRENT QUERY OPTIMIZATION statement. Here is the syntax: = SET CURRENT QUERY OPTIMIZATION
number host-variable
Figure 748,Set maintained type command, syntax Below are an example of the SET command: SET CURRENT QUERY OPTIMIZATION = 9;
figure 749, Set query optimization, example What Matches What
Assuming that the current query optimization special register is set to five or above, the DB2 optimizer will consider using a materialized query table (instead of the base table) when any of the following conditions are true: MQT DEFINITION ========================== REFRESH MAINTAINED-BY ========= ============== IMMEDIATE SYSTEM DEFERRED SYSETM DEFERRED USER DEFERRED FEDERATED-TOOL
DATABASE/APPLICATION STATUS =================================== REFRESH-AGE MAINTAINED-TYPE =========== ===================== ANY ALL or SYSTEM ANY ALL or USER ANY ALL or FEDERATED-TOOL
DB2 USE MQT === Yes Yes Yes Yes
Figure 750, When DB2 will consider using a materialized query table Selecting Special Registers
One can select the relevant special register to see what the values are: SELECT FROM
CURRENT REFRESH AGE ,CURRENT TIMESTAMP ,CURRENT QUERY OPTIMIZATION sysibm.sysdummy1;
AS age_ts AS current_ts AS q_opt
Figure 751, Selecting special registers Refresh Deferred Tables
A materialized query table defined REFRESH DEFERRED can be periodically updated using the REFRESH TABLE command. Below is an example of a such a table that has one row per qualifying department in the STAFF table:
268
Usage Notes
DB2 V9.7 Cookbook ©
CREATE TABLE staff_names AS (SELECT dept ,COUNT(*) AS ,SUM(salary) AS ,AVG(salary) AS ,MAX(salary) AS ,MIN(salary) AS ,STDDEV(salary) AS ,VARIANCE(salary) AS ,CURRENT TIMESTAMP AS FROM staff WHERE TRANSLATE(name) LIKE AND salary > GROUP BY dept HAVING COUNT(*) = 1 )DATA INITIALLY DEFERRED REFRESH
count_rows sum_salary avg_salary max_salary min_salary std_salary var_salary last_change '%A%' 10000 DEFERRED;
Figure 752, Refresh deferred materialized query table DDL Refresh Immediate Tables
A materialized query table defined REFRESH IMMEDIATE is automatically maintained in sync with the source table by DB2. As with any materialized query table, it is defined by referring to a query. Below is a table that refers to a single source table: CREATE TABLE emp_summary AS (SELECT emp.workdept ,COUNT(*) AS num_rows ,COUNT(emp.salary) AS num_salary ,SUM(emp.salary) AS sum_salary ,COUNT(emp.comm) AS num_comm ,SUM(emp.comm) AS sum_comm FROM employee emp GROUP BY emp.workdept )DATA INITIALLY DEFERRED REFRESH IMMEDIATE;
Figure 753, Refresh immediate materialized query table DDL Below is a query that can use the above materialized query table in place of the base table: SELECT
emp.workdept ,DEC(SUM(emp.salary),8,2) ,DEC(AVG(emp.salary),7,2) ,SMALLINT(COUNT(emp.comm)) ,SMALLINT(COUNT(*)) FROM employee emp WHERE emp.workdept > 'C' GROUP BY emp.workdept HAVING COUNT(*) 5 AND SUM(emp.salary) > 50000 ORDER BY sum_sal DESC;
AS AS AS AS
sum_sal avg_sal #comms #emps
Figure 754, Query that uses materialized query table (1 of 3) The next query can also use the materialized query table. This time, the data returned from the materialized query table is qualified by checking against a sub-query: SELECT
emp.workdept ,COUNT(*) AS #rows FROM employee emp WHERE emp.workdept IN (SELECT deptno FROM department WHERE deptname LIKE '%S%') GROUP BY emp.workdept HAVING SUM(salary) > 50000;
Figure 755, Query that uses materialized query table (2 of 3)
Materialized Query Tables
269
Graeme Birchall ©
This last example uses the materialized query table in a nested table expression: SELECT
#emps ,DEC(SUM(sum_sal),9,2) AS sal_sal ,SMALLINT(COUNT(*)) AS #depts FROM (SELECT emp.workdept ,DEC(SUM(emp.salary),8,2) ,MAX(emp.salary) ,SMALLINT(COUNT(*)) FROM employee emp GROUP BY emp.workdept )AS XXX GROUP BY #emps HAVING COUNT(*) > 1 ORDER BY #emps FETCH FIRST 3 ROWS ONLY OPTIMIZE FOR 3 ROWS;
AS sum_sal AS max_sal AS #emps
Figure 756, Query that uses materialized query table (3 of 3) Using Materialized Query Tables to Duplicate Data
All of the above materialized query tables have contained a GROUP BY in their definition. But this is not necessary. To illustrate, we will first create a simple table: CREATE TABLE staff_all (id SMALLINT ,name VARCHAR(9) ,job CHAR(5) ,salary DECIMAL(7,2) ,PRIMARY KEY(id));
NOT NULL NOT NULL
Figure 757, Create source table As long as the above table has a primary key, which it does, we can define a duplicate of the above using the following code: CREATE TABLE staff_all_dup AS (SELECT * FROM staff_all) DATA INITIALLY DEFERRED REFRESH IMMEDIATE;
Figure 758, Create duplicate data table We can also decide to duplicate only certain rows: CREATE TABLE staff_all_dup_some AS (SELECT * FROM staff_all WHERE id < 30) DATA INITIALLY DEFERRED REFRESH IMMEDIATE;
Figure 759, Create table - duplicate certain rows only Imagine that we had another table that listed all those staff that we are about to fire: CREATE TABLE staff_to_fire (id SMALLINT NOT NULL ,name VARCHAR(9) NOT NULL ,dept SMALLINT ,PRIMARY KEY(id));
Figure 760, Create source table We can create materialized query table that joins the above two staff tables as long as the following is true:
Both tables have identical primary keys (i.e. same number of columns).
The join is an inner join on the common primary key fields.
270
Usage Notes
DB2 V9.7 Cookbook ©
All primary key columns are listed in the SELECT.
Now for an example: CREATE TABLE staff_combo AS (SELECT aaa.id AS id1 ,aaa.job AS job ,fff.id as id2 ,fff.dept AS dept FROM staff_all aaa ,staff_to_fire fff WHERE aaa.id = fff.id) DATA INITIALLY DEFERRED REFRESH IMMEDIATE;
Figure 761, Materialized query table on join See page 272 for more examples of join usage. Queries that don't use Materialized Query Table
Below is a query that can not use the EMP_SUMMARY table because of the reference to the MAX function. Ironically, this query is exactly the same as the nested table expression above, but in the prior example the MAX is ignored because it is never actually selected: SELECT
emp.workdept ,DEC(SUM(emp.salary),8,2) ,MAX(emp.salary) FROM employee emp GROUP BY emp.workdept;
AS sum_sal AS max_sal
Figure 762, Query that doesn't use materialized query table (1 of 2) The following query can't use the materialized query table because of the DISTINCT clause: SELECT
emp.workdept ,DEC(SUM(emp.salary),8,2) ,COUNT(DISTINCT salary) FROM employee emp GROUP BY emp.workdept;
AS sum_sal AS #salaries
Figure 763, Query that doesn't use materialized query table (2 of 2) Usage Notes and Restrictions
A materialized query table must be refreshed before it can be queried. If the table is defined refresh immediate, then the table will be maintained automatically after the initial refresh.
Make sure to commit after doing a refresh. The refresh does not have an implied commit.
Run RUNSTATS after refreshing a materialized query table.
One can not load data into materialized query tables.
One can not directly update materialized query tables.
To refresh a materialized query table, use either of the following commands: REFRESH TABLE emp_summary; COMMIT; SET INTEGRITY FOR emp_summary iMMEDIATE CHECKED; COMMIT;
Figure 764, Materialized query table refresh commands
Materialized Query Tables
271
Graeme Birchall ©
Multi-table Materialized Query Tables
Single-table materialized query tables save having to look at individual rows to resolve a GROUP BY. Multi-table materialized query tables do this, and also avoid having to resolve a join. CREATE TABLE dept_emp_summary AS (SELECT emp.workdept ,dpt.deptname ,COUNT(*) AS num_rows ,COUNT(emp.salary) AS num_salary ,SUM(emp.salary) AS sum_salary ,COUNT(emp.comm) AS num_comm ,SUM(emp.comm) AS sum_comm FROM employee emp ,department dpt WHERE dpt.deptno = emp.workdept GROUP BY emp.workdept ,dpt.deptname )DATA INITIALLY DEFERRED REFRESH IMMEDIATE;
Figure 765, Multi-table materialized query table DDL The following query is resolved using the above materialized query table: SELECT
d.deptname ,d.deptno ,DEC(AVG(e.salary),7,2) AS avg_sal ,SMALLINT(COUNT(*)) AS #emps FROM department d ,employee e WHERE e.workdept = d.deptno AND d.deptname LIKE '%S%' GROUP BY d.deptname ,d.deptno HAVING SUM(e.comm) > 4000 ORDER BY avg_sal DESC;
Figure 766, Query that uses materialized query table Here is the SQL that DB2 generated internally to get the answer: SELECT
FROM
Q2.$C0 ,Q2.$C1 ,Q2.$C2 ,Q2.$C3 (SELECT
AS "deptname" AS "deptno" AS "avg_sal" AS "#emps" Q1.deptname ,Q1.workdept ,DEC((Q1.sum_salary / Q1.num_salary),7,2) ,SMALLINT(Q1.num_rows) dept_emp_summary AS Q1 (Q1.deptname LIKE '%S%') (4000 < Q1.sum_comm)
AS AS AS AS
$C0 $C1 $C2 $C3
FROM WHERE AND )AS Q2 ORDER BY Q2.$C2 DESC;
Figure 767, DB2 generated query to use materialized query table Rules and Restrictions
The join must be an inner join, and it must be written in the old style syntax.
Every table accessed in the join (except one?) must have a unique index.
The join must not be a Cartesian product.
The GROUP BY must include all of the fields that define the unique key for every table (except one?) in the join.
272
Usage Notes
DB2 V9.7 Cookbook ©
Three-table Example
CREATE TABLE dpt_emp_act_sumry AS (SELECT emp.workdept ,dpt.deptname ,emp.empno ,emp.firstnme ,SUM(act.emptime) AS sum_time ,COUNT(act.emptime) AS num_time ,COUNT(*) AS num_rows FROM department dpt ,employee emp ,emp_act act WHERE dpt.deptno = emp.workdept AND emp.empno = act.empno GROUP BY emp.workdept ,dpt.deptname ,emp.empno ,emp.firstnme )DATA INITIALLY DEFERRED REFRESH IMMEDIATE;
Figure 768, Three-table materialized query table DDL Now for a query that will use the above: SELECT FROM WHERE AND AND AND GROUP ORDER
d.deptno ,d.deptname ,DEC(AVG(a.emptime),5,2) AS avg_time department d ,employee e ,emp_act a d.deptno = e.workdept e.empno = a.empno d.deptname LIKE '%S%' e.firstnme LIKE '%S%' BY d.deptno ,d.deptname BY 3 DESC;
Figure 769, Query that uses materialized query table And here is the DB2 generated SQL: SELECT
Q4.$C0 AS "deptno" ,Q4.$C1 AS "deptname" ,Q4.$C2 AS "avg_time" FROM (SELECT Q3.$C3 AS $C0 ,Q3.$C2 AS $C1 ,DEC((Q3.$C1 / Q3.$C0),5,2) AS $C2 FROM (SELECT SUM(Q2.$C2) AS $C0 ,SUM(Q2.$C3) AS $C1 ,Q2.$C0 AS $C2 ,Q2.$C1 AS $C3 FROM (SELECT Q1.deptname AS ,Q1.workdept AS ,Q1.num_time AS ,Q1.sum_time AS FROM dpt_emp_act_sumry AS Q1 WHERE (Q1.firstnme LIKE '%S%') AND (Q1.DEPTNAME LIKE '%S%') )AS Q2 GROUP BY Q2.$C1 ,Q2.$C0 )AS Q3 )AS Q4 ORDER BY Q4.$C2 DESC;
$C0 $C1 $C2 $C3
Figure 770, DB2 generated query to use materialized query table
Materialized Query Tables
273
Graeme Birchall ©
Indexes on Materialized Query Tables
To really make things fly, one can add indexes to the materialized query table columns. DB2 will then use these indexes to locate the required data. Certain restrictions apply:
Unique indexes are not allowed.
The materialized query table must not be in a "check pending" status when the index is defined. Run a refresh to address this problem.
Below are some indexes for the DPT_EMP_ACT_SUMRY table that was defined above: CREATE INDEX dpt_emp_act_sumx1 ON dpt_emp_act_sumry (workdept ,deptname ,empno ,firstnme); CREATE INDEX dpt_emp_act_sumx2 ON dpt_emp_act_sumry (num_rows);
Figure 771, Indexes for DPT_EMP_ACT_SUMRY materialized query table The next query will use the first index (i.e. on WORKDEPT): SELECT
FROM WHERE AND AND GROUP
ORDER
d.deptno ,d.deptname ,e.empno ,e.firstnme ,INT(AVG(a.emptime)) AS avg_time department d ,employee e ,emp_act a d.deptno = e.workdept e.empno = a.empno d.deptno LIKE 'D%' BY d.deptno ,d.deptname ,e.empno ,e.firstnme BY 1,2,3,4;
Figure 772, Sample query that use WORKDEPT index The next query will use the second index (i.e. on NUM_ROWS): SELECT
d.deptno ,d.deptname ,e.empno ,e.firstnme ,COUNT(*) FROM department ,employee ,emp_act WHERE d.deptno AND e.empno GROUP BY d.deptno ,d.deptname ,e.empno ,e.firstnme HAVING COUNT(*) > ORDER BY 1,2,3,4;
AS #acts d e a = e.workdept = a.empno
4
Figure 773, Sample query that uses NUM_ROWS index
274
Usage Notes
DB2 V9.7 Cookbook ©
Organizing by Dimensions
The following materialized query table is organized (clustered) by the two columns that are referred to in the GROUP BY. Under the covers, DB2 will also create a dimension index on each column, and a block index on both columns combined: CREATE TABLE emp_sum AS (SELECT workdept ,job ,SUM(salary) AS sum_sal ,COUNT(*) AS #emps ,GROUPING(workdept) AS grp_dpt ,GROUPING(job) AS grp_job FROM employee GROUP BY CUBE(workdept ,job)) DATA INITIALLY DEFERRED REFRESH DEFERRED ORGANIZE BY DIMENSIONS (workdept, job) IN tsempsum;
Figure 774, Materialized query table organized by dimensions WARNING: Multi-dimensional tables may perform very poorly when created in the default tablespace, or in a system-maintained tablespace. Use a database-maintained tablespace with the right extent size, and/or run the DB2EMPFA command.
Don't forget to run RUNSTATS! Using Staging Tables
A staging table can be used to incrementally maintain a materialized query table that has been defined refresh deferred. Using a staging table can result in a significant performance saving (during the refresh) if the source table is very large, and is not changed very often. NOTE: To use a staging table, the SQL statement used to define the target materialized query table must follow the rules that apply for a table that is defined refresh immediate even though it is defined refresh deferred.
The staging table CREATE statement has the following components:
The name of the staging table.
A list of columns (with no attributes) in the target materialized query table. The column names do not have to match those in the target table.
Either two or three additional columns with specific names- as provided by DB2.
The name of the target materialized query table.
To illustrate, below is a typical materialized query table: CREATE TABLE emp_sumry AS (SELECT workdept AS ,COUNT(*) AS ,COUNT(salary) AS ,SUM(salary) AS FROM employee emp GROUP BY emp.workdept )DATA INITIALLY DEFERRED REFRESH
dept #rows #sal sum_sal DEFERRED;
Figure 775, Sample materialized query table Here is a staging table for the above:
Materialized Query Tables
275
Graeme Birchall ©
CREATE TABLE emp_sumry_s (dept ,num_rows ,num_sal ,sum_sal ,GLOBALTRANSID ,GLOBALTRANSTIME )FOR emp_sumry PROPAGATE IMMEDIATE;
Figure 776, Staging table for the above materialized query table Additional Columns
The two, or three, additional columns that every staging table must have are as follows:
GLOBALTRANSID: The global transaction ID for each propagated row.
GLOBALTRANSTIME: The transaction timestamp
OPERATIONTYPE: The operation type (i.e. insert, update, or delete). This column is needed if the target materialized query table does not contain a GROUP BY statement.
Using a Staging Table
To activate the staging table one must first use the SET INTEGRITY command to remove the check pending flag, and then do a full refresh of the target materialized query table. After this is done, the staging table will record all changes to the source table. Use the refresh incremental command to apply the changes recorded in the staging table to the target materialized query table. SET INTEGRITY FOR emp_sumry_s STAGING IMMEDIATE UNCHECKED; REFRESH TABLE emp_sumry; > REFRESH TABLE emp_sumry INCREMENTAL;
Figure 777, Enabling and the using a staging table A multi-row update (or insert, or delete) uses the same CURRENT TIMESTAMP for all rows changed, and for all invoked triggers. Therefore, the #CHANGING_SQL field is only incremented when a new timestamp value is detected.
276
Usage Notes
DB2 V9.7 Cookbook ©
Identity Columns and Sequences Imagine that one has an INVOICE table that records invoices generated. Also imagine that one wants every new invoice that goes into this table to get an invoice number value that is part of a unique and unbroken sequence of ascending values - assigned in the order that the invoices are generated. So if the highest invoice number is currently 12345, then the next invoice will get 12346, and then 12347, and so on. There are three ways to do this, up to a point:
Use an identity column, which generates a unique value per row in a table.
Use a sequence, which generates a unique value per one or more tables.
Do it yourself, using an insert trigger to generate the unique values.
You may need to know what values were generated during each insert. There are several ways to do this:
For all of the above techniques, embed the insert inside a select statement (see figure 795 and/or page 71). This is probably the best solution.
For identity columns, use the IDENTITY_VAL_LOCAL function (see page284).
For sequences, make a NEXTVAL or PREVVAL call (see page 287).
Living With Gaps
The only way that one can be absolutely certain not to have a gap in the sequence of values generated is to create your own using an insert trigger. However, this solution is probably the least efficient of those listed here, and it certainly has the least concurrency. There is almost never a valid business reason for requiring an unbroken sequence of values. So the best thing to do, if your users ask for such a feature, is to beat them up. Living With Sequence Errors
For efficiency reasons, identity column and sequence values are usually handed out (to users doing inserts) in block of values, where the block size is defined using the CACHE option. If a user inserts a row, and then dithers for a bit before inserting another, it is possible that some other user (with a higher value) will insert first. In this case, the identity column or sequence value will be a good approximation of the insert sequence, but not right on. If the users need to know the precise order with which rows were inserted, then either set the cache size to one, which will cost, or include a current timestamp value.
Identity Columns One can define a column in a DB2 table as an "identity column". This column, which must be numeric (note: fractional fields not allowed), will be incremented by a fixed constant each time a new row is inserted. Below is a syntax diagram for that part of a CREATE TABLE statement that refers to an identity column definition:
Identity Columns and Sequences
277
Graeme Birchall ©
column name
data type
GENERATED
ALWAYS BY DEFAULT
AS IDENTITY (
1 numeric constant
START WITH
)
1 numeric constant
INCREMENT BY NO MINVALUE MINVALUE
numeric constant
NO MAXVALUE MAXVALUE
numeric constant
NO CYCLE CYCLE CACHE 20 NO CACHE CACHE integer constant NO ORDER ORDER
Figure 778, Identity Column syntax Below is an example of a typical invoice table that uses an identity column that starts at one, and then goes ever upwards: CREATE TABLE invoice_data (invoice# INTEGER NOT GENERATED ALWAYS AS IDENTITY (START WITH 1 ,INCREMENT BY 1 ,NO MAXVALUE ,NO CYCLE ,ORDER) ,sale_date DATE NOT ,customer_id CHAR(20) NOT ,product_id INTEGER NOT ,quantity INTEGER NOT ,price DECIMAL(18,2) NOT ,PRIMARY KEY (invoice#));
NULL
NULL NULL NULL NULL NULL
Figure 779, Identity column, sample table Rules and Restrictions
Identity columns come in one of two general flavors:
The value is always generated by DB2.
The value is generated by DB2 only if the user does not provide a value (i.e. by default). This configuration is typically used when the input is coming from an external source (e.g. data propagation).
Rules
There can only be one identity column per table.
The field cannot be updated if it is defined "generated always".
278
Identity Columns
DB2 V9.7 Cookbook ©
The column type must be numeric and must not allow fractional values. Any integer type is OK. Decimal is also fine, as long as the scale is zero. Floating point is a no-no.
The identity column value is generated before any BEFORE triggers are applied. Use a trigger transition variable to see the value.
A unique index is not required on the identity column, but it is a good idea. Certainly, if the value is being created by DB2, then a non-unique index is a fairly stupid idea.
Unlike triggers, identity column logic is invoked and used during a LOAD. However, a load-replace will not reset the identity column value. Use the RESTART command (see below) to do this. An identity column is not affected by a REORG.
Syntax Notes
START WITH defines the start value, which can be any valid integer value. If no start value is provided, then the default is the MINVALUE for ascending sequences, and the MAXVALUE for descending sequences. If this value is also not provided, then the default is 1.
INCREMENT BY defines the interval between consecutive values. This can be any valid integer value, though using zero is pretty silly. The default is 1.
MINVALUE defines (for ascending sequences) the value that the sequence will start at if no start value is provided. It is also the value that an ascending sequence will begin again at after it reaches the maximum and loops around. If no minimum value is provided, then after reaching the maximum the sequence will begin again at the start value. If that is also not defined, then the sequence will begin again at 1, which is the default start value.
For descending sequences, it is the minimum value that will be used before the sequence loops around, and starts again at the maximum value.
MAXVALUE defines (for ascending sequences) the value that a sequence will stop at, and then go back to the minimum value. For descending sequences, it is the start value (if no start value is provided), and also the restart value - if the sequence reaches the minimum and loops around.
CYCLE defines whether the sequence should cycle about when it reaches the maximum value (for an ascending sequences), or whether it should stop. The default is no cycle.
CACHE defines whether or not to allocate sequences values in chunks, and thus to save on log writes. The default is no cache, which means that every row inserted causes a log write (to save the current value).
If a cache value (from 2 to 20) is provided, then the new values are assigned to a common pool in blocks. Each insert user takes from the pool, and only when all of the values are used is a new block (of values) allocated and a log write done. If the table is deactivated, either normally or otherwise, then the values in the current block are discarded, resulting in gaps in the sequence. Gaps in the sequence of values also occur when an insert is subsequently rolled back, so they cannot be avoided. But don't use the cache if you want to try and avoid them.
ORDER defines whether all new rows inserted are assigned a sequence number in the order that they were inserted. The default is no, which means that occasionally a row that is inserted after another may get a slightly lower sequence number. This is the default.
Identity Columns and Sequences
279
Graeme Birchall ©
Identity Column Examples
The following example uses all of the defaults to start an identity column at one, and then to go up in increments of one. The inserts will eventually die when they reach the maximum allowed value for the field type (i.e. for small integer = 32K). CREATE TABLE test_data (key# SMALLINT NOT NULL GENERATED ALWAYS AS IDENTITY ,dat1 SMALLINT NOT NULL ,ts1 TIMESTAMP NOT NULL ,PRIMARY KEY(key#));
KEY# FIELD - VALUES ASSIGNED ============================ 1 2 3 4 5 6 7 8 9 10 11 etc.
Figure 780, Identity column, ascending sequence The next example defines an identity column that goes down in increments of -3: CREATE TABLE test_data (key# SMALLINT NOT NULL GENERATED ALWAYS AS IDENTITY (START WITH 6 ,INCREMENT BY -3 ,NO CYCLE ,NO CACHE ,ORDER) ,dat1 SMALLINT NOT NULL ,ts1 TIMESTAMP NOT NULL ,PRIMARY KEY(key#));
KEY# FIELD - VALUES ASSIGNED ============================ 6 3 0 -3 -6 -9 -12 -15 etc.
Figure 781, Identity column, descending sequence The next example, which is amazingly stupid, goes nowhere fast. A primary key cannot be defined on this table: CREATE TABLE test_data (key# SMALLINT NOT NULL GENERATED ALWAYS AS IDENTITY (START WITH 123 ,MAXVALUE 124 ,INCREMENT BY 0 ,NO CYCLE ,NO ORDER) ,dat1 SMALLINT NOT NULL ,ts1 TIMESTAMP NOT NULL);
KEY# VALUES ASSIGNED ============================ 123 123 123 123 123 123 etc.
Figure 782, Identity column, dumb sequence The next example uses every odd number up to the maximum (i.e. 6), then loops back to the minimum value, and goes through the even numbers, ad-infinitum: CREATE TABLE test_data (key# SMALLINT NOT NULL GENERATED ALWAYS AS IDENTITY (START WITH 1 ,INCREMENT BY 2 ,MAXVALUE 6 ,MINVALUE 2 ,CYCLE ,NO CACHE ,ORDER) ,dat1 SMALLINT NOT NULL ,ts1 TIMESTAMP NOT NULL);
KEY# VALUES ASSIGNED ============================ 1 3 5 2 4 6 2 4 6 2 4 6 etc.
Figure 783, Identity column, odd values, then even, then stuck Usage Examples
Below is the DDL for a simplified invoice table where the primary key is an identity column. Observe that the invoice# is always generated by DB2:
280
Identity Columns
DB2 V9.7 Cookbook ©
CREATE TABLE invoice_data (invoice# INTEGER NOT GENERATED ALWAYS AS IDENTITY (START WITH 100 ,INCREMENT BY 1 ,NO CYCLE ,ORDER) ,sale_date DATE NOT ,customer_id CHAR(20) NOT ,product_id INTEGER NOT ,quantity INTEGER NOT ,price DECIMAL(18,2) NOT ,PRIMARY KEY (invoice#));
NULL
NULL NULL NULL NULL NULL
Figure 784, Identity column, definition One cannot provide a value for the invoice# when inserting into the above table. Therefore, one must either use a default placeholder, or leave the column out of the insert. An example of both techniques is given below. The second insert also selects the generated values: INSERT INTO invoice_data VALUES (DEFAULT,'2001-11-22','ABC',123,100,10); SELECT invoice# FROM FINAL TABLE (INSERT INTO invoice_data (sale_date,customer_id,product_id,quantity,price) VALUES ('2002-11-22','DEF',123,100,10) ,('2003-11-22','GHI',123,100,10));
ANSWER ======== INVOICE# -------101 102
Figure 785, Invoice table, sample inserts Below is the state of the table after the above two inserts: INVOICE# -------100 101 102
SALE_DATE ---------2001-11-22 2002-11-22 2003-11-22
CUSTOMER_ID ----------ABC DEF GHI
PRODUCT_ID --- -----123 123 123
QUANTITY -------100 100 100
PRICE ----10.00 10.00 10.00
Figure 786, Invoice table, after inserts Altering Identity Column Options
Imagine that the application is happily collecting invoices in the above table, but your silly boss is unhappy because not enough invoices, as measured by the ever-ascending invoice# value, are being generated per unit of time. We can improve things without actually fixing any difficult business problems by simply altering the invoice# current value and the increment using the ALTER TABLE ... RESTART command: ALTER TABLE invoice_data ALTER COLUMN invoice# RESTART WITH 1000 SET INCREMENT BY 2;
Figure 787, Invoice table, restart identity column value Now imagine that we insert two more rows thus: INSERT INTO invoice_data VALUES (DEFAULT,'2004-11-24','XXX',123,100,10) ,(DEFAULT,'2004-11-25','YYY',123,100,10);
Figure 788, Invoice table, more sample inserts Our mindless management will now see this data:
Identity Columns and Sequences
281
Graeme Birchall ©
INVOICE# -------100 101 102 1000 1002
SALE_DATE ---------2001-11-22 2002-11-22 2003-11-22 2004-11-24 2004-11-25
CUSTOMER_ID ----------ABC DEF GHI XXX YYY
PRODUCT_ID ---------123 123 123 123 123
QUANTITY -------100 100 100 100 100
PRICE ----10.00 10.00 10.00 10.00 10.00
Figure 789, Invoice table, after second inserts Alter Usage Notes
The identity column options can be changed using the ALTER TABLE command: RESTART
numeric constant
SET INCREMENT BY
numeric constant
SET
NO MINVALUE MINVALUE numeric constant
SET
NO MAXVALUE MAXVALUE numeric constant
SET
NO CYCLE CYCLE
SET
NO ORDER ORDER
Figure 790, Identity Column alter syntax Restarting the identity column start number to a lower number, or to a higher number if the increment is a negative value, can result in the column getting duplicate values. This can also occur if the increment value is changed from positive to negative, or vice-versa. If no value is provided for the restart option, the sequence restarts at the previously defined start value. Gaps in Identity Column Values
If an identity column is generated always, and no cache is used, and the increment value is 1, then there will usually be no gaps in the sequence of assigned values. But gaps can occur if an insert is subsequently rolled out instead of committed. In the following example, there will be no row in the table with customer number "1" after the rollback: CREATE TABLE customers (cust# INTEGER NOT NULL GENERATED ALWAYS AS IDENTITY (NO CACHE) ,cname CHAR(10) NOT NULL ,ctype CHAR(03) NOT NULL ,PRIMARY KEY (cust#)); COMMIT; SELECT cust# FROM FINAL TABLE (INSERT INTO customers VALUES (DEFAULT,'FRED','XXX')); ROLLBACK;
ANSWER ====== CUST# ----1
SELECT FROM (INSERT VALUES COMMIT;
ANSWER ====== CUST# ----2
cust# FINAL TABLE INTO customers (DEFAULT,'FRED','XXX'));
Figure 791, Gaps in Values, example
282
Identity Columns
DB2 V9.7 Cookbook ©
Find Gaps in Values
The following query can be used to list the missing values in a table. It starts by getting the minimum and maximum values. It next generates every value in between. Finally, it checks the generated values against the source tables. Non-matches are selected. WITH INPUT generate_values (min_val, max_val, num_val, cur_val) AS ===== (SELECT MIN(dat1) DAT1 ,MAX(dat1) ---,COUNT(*) 1 ,MIN(dat1) 2 FROM test_data td1 3 UNION ALL 4 SELECT min_val 6 ,max_val 7 ,num_val 9 ,cur_val + 1 10 FROM generate_values gv1 WHERE cur_val < max_val ) ANSWER SELECT * =============================== FROM generate_values gv2 MIN_VAL MAX_VAL NUM_VAL CUR_VAL WHERE NOT EXISTS ------- ------- ------- ------(SELECT * 1 10 8 5 FROM test_data td2 1 10 8 8 WHERE td2.dat1 = cur_val) ORDER BY cur_val;
Figure 792, Find gaps in values The above query may be inefficient if there is no suitable index on the DAT1 column. The next query gets around this problem by using an EXCEPT instead of a sub-query: WITH generate_values (min_val, max_val, num_val, cur_val) AS (SELECT MIN(dat1) ,MAX(dat1) ,COUNT(*) ,MIN(dat1) FROM test_data td1 UNION ALL SELECT min_val ,max_val ,num_val ,cur_val + 1 FROM generate_values gv1 WHERE cur_val < max_val ) SELECT cur_val FROM generate_values gv2 EXCEPT ALL SELECT dat1 FROM test_data td2 ORDER BY 1;
INPUT ===== DAT1 ---1 2 3 4 6 7 9 10
ANSWER ======= CUR_VAL ------5 8
Figure 793, Find gaps in values The next query uses a totally different methodology. It assigns a rank to every value, and then looks for places where the rank and value get out of step:
Identity Columns and Sequences
283
Graeme Birchall ©
WITH assign_ranks AS (SELECT dat1 ,DENSE_RANK() OVER(ORDER BY dat1) AS rank# FROM test_data ), locate_gaps AS (SELECT dat1 - rank# AS diff ,min(dat1) AS min_val ,max(dat1) AS max_val ,ROW_NUMBER() OVER(ORDER BY dat1 - rank#) AS gap# FROM assign_ranks ar1 GROUP BY dat1 - rank# ) SELECT lg1.gap# AS gap# ,lg1.max_val AS prev_val ,lg2.min_val AS next_val ,lg2.min_val - lg1.max_val AS diff FROM locate_gaps lg1 ,locate_gaps lg2 ANSWER WHERE lg2.gap# = lg1.gap# + 1 =========================== ORDER BY lg1.gap#; GAP# PREV_VAL NEXT_VAL DIFF ---- -------- -------- ---1 4 6 2 2 7 9 2
Figure 794, Find gaps in values IDENTITY_VAL_LOCAL Function
There are two ways to find out what values were generated when one inserted a row into a table with an identity column:
Embed the insert within a select statement (see figure 795).
Call the IDENTITY_VAL_LOCAL function.
Certain rules apply to IDENTITY_VAL_LOCAL function usage:
The value returned from is a decimal (31.0) field.
The function returns null if the user has not done a single-row insert in the current unit of work. Therefore, the function has to be invoked before one does a commit. Having said this, in some versions of DB2 it seems to work fine after a commit.
If the user inserts multiple rows into table(s) having identity columns in the same unit of work, the result will be the value obtained from the last single-row insert. The result will be null if there was none.
Multiple-row inserts are ignored by the function. So if the user first inserts one row, and then separately inserts two rows (in a single SQL statement), the function will return the identity column value generated during the first insert.
The function cannot be called in a trigger or SQL function. To get the current identity column value in an insert trigger, use the trigger transition variable for the column. The value, and thus the transition variable, is defined before the trigger is begun.
If invoked inside an insert statement (i.e. as an input value), the value will be taken from the most recent (previous) single-row insert done in the same unit of work. The result will be null if there was none.
The value returned by the function is unpredictable if the prior single-row insert failed. It may be the value from the insert before, or it may be the value given to the failed insert.
284
Identity Columns
DB2 V9.7 Cookbook ©
The function is non-deterministic, which means that the result is determined at fetch time (i.e. not at open) when used in a cursor. So if one fetches a row from a cursor, and then does an insert, the next fetch may get a different value from the prior.
The value returned by the function may not equal the value in the table - if either a trigger or an update has changed the field since the value was generated. This can only occur if the identity column is defined as being "generated by default". An identity column that is "generated always" cannot be updated.
When multiple users are inserting into the same table concurrently, each will see their own most recent identity column value. They cannot see each other's.
If the above sounds unduly complex, it is because it is. It is often much easier to simply get the values by embedding the insert inside a select: SELECT
MIN(cust#) AS minc ,MAX(cust#) AS maxc ,COUNT(*) AS rows FROM FINAL TABLE (INSERT INTO customers VALUES (DEFAULT,'FRED','xxx') ,(DEFAULT,'DAVE','yyy') ,(DEFAULT,'JOHN','zzz'));
ANSWER ============== MINC MAXC ROWS ---- ---- ---3 5 3
Figure 795, Selecting identity column values inserted Below are two examples of the function in use. Observe that the second invocation (done after the commit) returned a value, even though it is supposed to return null: CREATE TABLE invoice_table (invoice# INTEGER NOT GENERATED ALWAYS AS IDENTITY ,sale_date DATE NOT ,customer_id CHAR(20) NOT ,product_id INTEGER NOT ,quantity INTEGER NOT ,price DECIMAL(18,2) NOT ,PRIMARY KEY (invoice#)); COMMIT;
NULL NULL NULL NULL NULL NULL
INSERT INTO invoice_table VALUES (DEFAULT,'2000-11-22','ABC',123,100,10); WITH temp (id) AS (VALUES (IDENTITY_VAL_LOCAL())) SELECT * FROM temp;
16000
Figure 836, Join fullselect to real table Table Function Usage
If the fullselect query has a reference to a row in a table that is outside of the fullselect, then it needs to be written as a TABLE function call. In the next example, the preceding "A" table is referenced in the fullselect, and so the TABLE function call is required: SELECT
a.id ,a.dept ,a.salary ,b.deptsal FROM staff a ,TABLE (SELECT b.dept ,SUM(b.salary) AS deptsal FROM staff b WHERE b.dept = a.dept GROUP BY b.dept )AS b WHERE a.id < 40 ORDER BY a.id;
ANSWER ========================= ID DEPT SALARY DEPTSAL -- ---- -------- -------10 20 18357.50 64286.10 20 20 78171.25 64286.10 30 38 77506.75 77285.55
Figure 837, Fullselect with external table reference Below is the same query written without the reference to the "A" table in the fullselect, and thus without a TABLE function call:
Temporary Tables
303
Graeme Birchall ©
SELECT
a.id ,a.dept ,a.salary ,b.deptsal FROM staff a ,(SELECT b.dept ,SUM(b.salary) AS deptsal FROM staff b GROUP BY b.dept )AS b WHERE a.id < 40 AND b.dept = a.dept ORDER BY a.id;
ANSWER ========================= ID DEPT SALARY DEPTSAL -- ---- -------- -------10 20 18357.50 64286.10 20 20 78171.25 64286.10 30 38 77506.75 77285.55
Figure 838, Fullselect without external table reference Any externally referenced table in a fullselect must be defined in the query syntax (starting at the first FROM statement) before the fullselect. Thus, in the first example above, if the "A" table had been listed after the "B" table, then the query would have been invalid. Full-Select in SELECT Phrase
A fullselect that returns a single column and row can be used in the SELECT part of a query: SELECT
id ,salary ,(SELECT MAX(salary) FROM staff ) AS maxsal FROM staff a WHERE id < 60 ORDER BY id;
ANSWER ==================== ID SALARY MAXSAL -- -------- -------10 18357.50 22959.20 20 78171.25 22959.20 30 77506.75 22959.20 40 18006.00 22959.20 50 20659.80 22959.20
Figure 839, Use an uncorrelated Full-Select in a SELECT list A fullselect in the SELECT part of a statement must return only a single row, but it need not always be the same row. In the following example, the ID and SALARY of each employee is obtained - along with the max SALARY for the employee's department. SELECT
id ,salary ,(SELECT MAX(salary) FROM staff b WHERE a.dept = b.dept ) AS maxsal FROM staff a WHERE id < 60 ORDER BY id;
ANSWER ==================== ID SALARY MAXSAL -- -------- -------10 18357.50 18357.50 20 78171.25 18357.50 30 77506.75 18006.00 40 18006.00 18006.00 50 20659.80 20659.80
Figure 840, Use a correlated Full-Select in a SELECT list SELECT id ,dept ,salary ,(SELECT MAX(salary) FROM staff b WHERE b.dept = a.dept) ,(SELECT MAX(salary) FROM staff) FROM staff a WHERE id < 60 ORDER BY id;
ANSWER ================================== ID DEPT SALARY 4 5 -- ---- -------- -------- -------10 20 18357.50 18357.50 22959.20 20 20 78171.25 18357.50 22959.20 30 38 77506.75 18006.00 22959.20 40 38 18006.00 18006.00 22959.20 50 15 20659.80 20659.80 22959.20
Figure 841, Use correlated and uncorrelated Full-Selects in a SELECT list INSERT Usage
The following query uses both an uncorrelated and correlated fullselect in the query that builds the set of rows to be inserted:
304
Temporary Tables - in Statement
DB2 V9.7 Cookbook ©
INSERT INTO staff SELECT id + 1 ,(SELECT MIN(name) FROM staff) ,(SELECT dept FROM staff s2 WHERE s2.id = s1.id - 100) ,'A',1,2,3 FROM staff s1 WHERE id = (SELECT MAX(id) FROM staff);
Figure 842, Fullselect in INSERT UPDATE Usage
The following example uses an uncorrelated fullselect to assign a set of workers the average salary in the company - plus two thousand dollars. UPDATE staff a SET salary = (SELECT AVG(salary)+ 2000 FROM staff) WHERE id < 60;
ANSWER: ======= ID DEPT -- ---10 20 20 20 30 38 40 38 50 15
SALARY ================= BEFORE AFTER -------- -------18357.50 18675.64 78171.25 18675.64 77506.75 18675.64 18006.00 18675.64 20659.80 18675.64
Figure 843, Use uncorrelated Full-Select to give workers company AVG salary (+$2000) The next statement uses a correlated fullselect to assign a set of workers the average salary for their department - plus two thousand dollars. Observe that when there is more than one worker in the same department, that they all get the same new salary. This is because the fullselect is resolved before the first update was done, not after each. UPDATE staff a SET salary = (SELECT AVG(salary) + 2000 FROM staff b WHERE a.dept = b.dept ) WHERE id < 60;
ANSWER: ======= ID DEPT -- ---10 20 20 20 30 38 40 38 50 15
SALARY ================= BEFORE AFTER -------- -------18357.50 18071.52 78171.25 18071.52 77506.75 17457.11 18006.00 17457.11 20659.80 17482.33
Figure 844, Use correlated Full-Select to give workers department AVG salary (+$2000) NOTE: A fullselect is always resolved just once. If it is queried using a correlated expression, then the data returned each time may differ, but the table remains unchanged.
The next update is the same as the prior, except that two fields are changed: UPDATE staff a SET (salary,years) = (SELECT AVG(salary) + 2000 ,MAX(years) FROM staff b WHERE a.dept = b.dept ) WHERE id < 60;
Figure 845, Update two fields by referencing Full-Select
Temporary Tables
305
Graeme Birchall ©
Declared Global Temporary Tables If we want to temporarily retain some rows for processing by subsequent SQL statements, we can use a Declared Global Temporary Table. A temporary table only exists until the thread is terminated (or sooner). It is not defined in the DB2 catalogue, and neither its definition nor its contents are visible to other users. Multiple users can declare the same temporary table at the same time. Each will be independently working with their own copy. DECLARE GLOBAL TEMPORARY TABLE , column-name
( LIKE AS
table-name )
column-definition
table-name view-name (
full-select
)
DEFINITION ONLY
COLUMN
INCLUDING EXCLUDING
DEFAULTS COLUMN ATTRIBUTES
EXCLUDING IDENTITY INCLUDING IDENTITY ON COMMIT DELETE ROWS ON COMMIT PRESERVE ROWS NOT LOGGED IN
ON ROLLBACK DELETE ROWS ON ROLLBACK PRESERVE ROWS
WITH REPLACE
tablespace-name ,
PARTITIONING KEY
(
column-name
)
USING HASHING
Figure 846, Declared Global Temporary Table syntax Usage Notes
For a complete description of this feature, see the SQL reference. Below are some key points:
The temporary table name can be any valid DB2 table name. The table qualifier, if provided, must be SESSION. If the qualifier is not provided, it is assumed to be SESSION.
If the temporary table has been previously defined in this session, the WITH REPLACE clause can be used to override it. Alternatively, one can DROP the prior instance.
An index can be defined on a global temporary table. The qualifier (i.e. SESSION) must be explicitly provided.
Any column type can be used in the table, except for: BLOB, CLOB, DBCLOB, LONG VARCHAR, LONG VARGRAPHIC, DATALINK, reference, and structured data types.
One can choose to preserve or delete (the default) the rows in the table when a commit occurs. Deleting the rows does not drop the table.
Standard identity column definitions can be used if desired.
Changes are not logged.
306
Declared Global Temporary Tables
DB2 V9.7 Cookbook ©
Sample SQL
Below is an example of declaring a global temporary table by listing the columns: DECLARE GLOBAL TEMPORARY TABLE session.fred (dept SMALLINT NOT NULL ,avg_salary DEC(7,2) NOT NULL ,num_emps SMALLINT NOT NULL) ON COMMIT DELETE ROWS;
Figure 847, Declare Global Temporary Table - define columns In the next example, the temporary table is defined to have exactly the same columns as the existing STAFF table: DECLARE GLOBAL TEMPORARY TABLE session.fred LIKE staff INCLUDING COLUMN DEFAULTS WITH REPLACE ON COMMIT PRESERVE ROWS;
Figure 848, Declare Global Temporary Table - like another table In the next example, the temporary table is defined to have a set of columns that are returned by a particular select statement. The statement is not actually run at definition time, so any predicates provided are irrelevant: DECLARE GLOBAL TEMPORARY TABLE session.fred AS (SELECT dept ,MAX(id) AS max_id ,SUM(salary) AS sum_sal FROM staff WHERE name 'IDIOT' GROUP BY dept) DEFINITION ONLY WITH REPLACE;
Figure 849, Declare Global Temporary Table - like query output Indexes can be added to temporary tables in order to improve performance and/or to enforce uniqueness: DECLARE GLOBAL TEMPORARY TABLE session.fred LIKE staff INCLUDING COLUMN DEFAULTS WITH REPLACE ON COMMIT DELETE ROWS; CREATE UNIQUE INDEX session.fredx ON Session.fred (id); INSERT INTO session.fred SELECT * FROM staff WHERE id < 200; SELECT FROM
COUNT(*) session.fred;
ANSWER ====== 19
COUNT(*) session.fred;
ANSWER ====== 0
COMMIT; SELECT FROM
Figure 850, Temporary table with index A temporary table has to be dropped to reuse the same name:
Temporary Tables
307
Graeme Birchall ©
DECLARE GLOBAL TEMPORARY TABLE session.fred (dept SMALLINT NOT NULL ,avg_salary DEC(7,2) NOT NULL ,num_emps SMALLINT NOT NULL) ON COMMIT DELETE ROWS; INSERT INTO session.fred SELECT dept ,AVG(salary) ,COUNT(*) FROM staff GROUP BY dept; SELECT FROM
COUNT(*) session.fred;
ANSWER ====== 8
DROP TABLE session.fred; DECLARE GLOBAL TEMPORARY TABLE session.fred (dept SMALLINT NOT NULL) ON COMMIT DELETE ROWS; SELECT FROM
COUNT(*) session.fred;
ANSWER ====== 0
Figure 851, Dropping a temporary table Tablespace
Before a user can create a declared global temporary table, a USER TEMPORARY tablespace that they have access to, has to be created. A typical definition follows: CREATE USER TEMPORARY TABLESPACE FRED MANAGED BY DATABASE USING (FILE 'C:\DB2\TEMPFRED\FRED1' 1000 ,FILE 'C:\DB2\TEMPFRED\FRED2' 1000 ,FILE 'C:\DB2\TEMPFRED\FRED3' 1000); GRANT USE OF TABLESPACE FRED TO PUBLIC;
Figure 852, Create USER TEMPORARY tablespace Do NOT use to Hold Output
In general, do not use a Declared Global Temporary Table to hold job output data, especially if the table is defined ON COMMIT PRESERVE ROWS. If the job fails halfway through, the contents of the temporary table will be lost. If, prior to the failure, the job had updated and then committed Production data, it may be impossible to recreate the lost output because the committed rows cannot be updated twice.
308
Declared Global Temporary Tables
DB2 V9.7 Cookbook ©
Recursive SQL Recursive SQL enables one to efficiently resolve all manner of complex logical structures that can be really tough to work with using other techniques. On the down side, it is a little tricky to understand at first and it is occasionally expensive. In this chapter we shall first show how recursive SQL works and then illustrate some of the really cute things that one use it for. Use Recursion To
Create sample data.
Select the first "n" rows.
Generate a simple parser.
Resolve a Bill of Materials hierarchy.
Normalize and/or denormalize data structures.
When (Not) to Use Recursion
A good SQL statement is one that gets the correct answer, is easy to understand, and is efficient. Let us assume that a particular statement is correct. If the statement uses recursive SQL, it is never going to be categorized as easy to understand (though the reading gets much easier with experience). However, given the question being posed, it is possible that a recursive SQL statement is the simplest way to get the required answer. Recursive SQL statements are neither inherently efficient nor inefficient. Because they often involve a join, it is very important that suitable indexes be provided. Given appropriate indexes, it is quite probable that a recursive SQL statement is the most efficient way to resolve a particular business problem. It all depends upon the nature of the question: If every row processed by the query is required in the answer set (e.g. Find all people who work for Bob), then a recursive statement is likely to very efficient. If only a few of the rows processed by the query are actually needed (e.g. Find all airline flights from Boston to Dallas, then show only the five fastest) then the cost of resolving a large data hierarchy (or network), most of which is immediately discarded, can be very prohibitive. If one wants to get only a small subset of rows in a large data structure, it is very important that of the unwanted data is excluded as soon as possible in the processing sequence. Some of the queries illustrated in this chapter have some rather complicated code in them to do just this. Also, always be on the lookout for infinitely looping data structures. Conclusion
Recursive SQL statements can be very efficient, if coded correctly, and if there are suitable indexes. When either of the above is not true, they can be very slow.
How Recursion Works Below is a description of a very simple application. The table on the left contains a normalized representation of the hierarchical structure on the right. Each row in the table defines a relationship displayed in the hierarchy. The PKEY field identifies a parent key, the CKEY
Recursive SQL
309
Graeme Birchall ©
field has related child keys, and the NUM field has the number of times the child occurs within the related parent. HIERARCHY +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |EEE | 44| |DDD |FFF | 5| |FFF |GGG | 5| +---------------+
AAA | +-----+-----+ | | | BBB CCC DDD | | +-+ +-+--+ | | | EEE FFF | | GGG
Figure 853, Sample Table description - Recursion List Dependents of AAA
We want to use SQL to get a list of all the dependents of AAA. This list should include not only those items like CCC that are directly related, but also values such as GGG, which are indirectly related. The easiest way to answer this question (in SQL) is to use a recursive SQL statement that goes thus: WITH parent (pkey, ckey) AS (SELECT pkey, ckey FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.pkey, C.ckey FROM hierarchy C ,parent P WHERE P.ckey = C.pkey ) SELECT pkey, ckey FROM parent;
ANSWER ========= PKEY CKEY ---- ---AAA BBB AAA CCC AAA DDD CCC EEE DDD EEE DDD FFF FFF GGG
< < < <
PROCESSING SEQUENCE ========== 1st pass "" "" 2nd pass 3rd pass "" 4th pass
Figure 854, SQL that does Recursion The above statement is best described by decomposing it into its individual components, and then following of sequence of events that occur:
The WITH statement at the top defines a temporary table called PARENT.
The upper part of the UNION ALL is only invoked once. It does an initial population of the PARENT table with the three rows that have an immediate parent key of AAA .
The lower part of the UNION ALL is run recursively until there are no more matches to the join. In the join, the current child value in the temporary PARENT table is joined to related parent values in the DATA table. Matching rows are placed at the front of the temporary PARENT table. This recursive processing will stop when all of the rows in the PARENT table have been joined to the DATA table.
The SELECT phrase at the bottom of the statement sends the contents of the PARENT table back to the user's program.
Another way to look at the above process is to think of the temporary PARENT table as a stack of data. This stack is initially populated by the query in the top part of the UNION ALL. Next, a cursor starts from the bottom of the stack and goes up. Each row obtained by the cursor is joined to the DATA table. Any matching rows obtained from the join are added to the top of the stack (i.e. in front of the cursor). When the cursor reaches the top of the stack, the statement is done. The following diagram illustrates this process:
310
How Recursion Works
DB2 V9.7 Cookbook ©
PKEY >
AAA
AAA
AAA
CCC
DDD DDD
FFF
CKEY >
BBB
CCC
DDD
EEE
EEE
GGG
FFF
Figure 855, Recursive processing sequence Notes & Restrictions
Recursive SQL requires that there be a UNION ALL phrase between the two main parts of the statement. The UNION ALL, unlike the UNION, allows for duplicate output rows, which is what often comes out of recursive processing.
If done right, recursive SQL is often fairly efficient. When it involves a join similar to the example shown above, it is important to make sure that this join is efficient. To this end, suitable indexes should be provided.
The output of a recursive SQL is a temporary table (usually). Therefore, all temporary table usage restrictions also apply to recursive SQL output. See the section titled "Common Table Expression" for details.
The output of one recursive expression can be used as input to another recursive expression in the same SQL statement. This can be very handy if one has multiple logical hierarchies to traverse (e.g. First find all of the states in the USA, then final all of the cities in each state).
Any recursive coding, in any language, can get into an infinite loop - either because of bad coding, or because the data being processed has a recursive value structure. To prevent your SQL running forever, see the section titled "Halting Recursive Processing" on page 320.
Sample Table DDL & DML CREATE TABLE hierarchy (pkey CHAR(03) NOT NULL ,ckey CHAR(03) NOT NULL ,num SMALLINT NOT NULL ,PRIMARY KEY(pkey, ckey) ,CONSTRAINT dt1 CHECK (pkey ckey) ,CONSTRAINT dt2 CHECK (num > 0)); COMMIT; CREATE UNIQUE INDEX hier_x1 ON hierarchy (ckey, pkey); COMMIT; INSERT INTO hierarchy VALUES ('AAA','BBB', 1), ('AAA','CCC', 5), ('AAA','DDD',20), ('CCC','EEE',33), ('DDD','EEE',44), ('DDD','FFF', 5), ('FFF','GGG', 5); COMMIT;
Figure 856, Sample Table DDL - Recursion
Recursive SQL
311
Graeme Birchall ©
Introductory Recursion This section will use recursive SQL statements to answer a series of simple business questions using the sample HIERARCHY table described on page 311. Be warned that things are going to get decidedly more complex as we proceed. List all Children #1
Find all the children of AAA. Don't worry about getting rid of duplicates, sorting the data, or any other of the finer details. WITH parent (ckey) AS (SELECT ckey FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.ckey FROM hierarchy C ,parent P WHERE P.ckey = C.pkey ) SELECT ckey FROM parent;
ANSWER ====== CKEY ---BBB CCC DDD EEE EEE FFF GGG
HIERARCHY +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |EEE | 44| |DDD |FFF | 5| |FFF |GGG | 5| +---------------+
Figure 857, List of children of AAA WARNING: Much of the SQL shown in this section will loop forever if the target database has a recursive data structure. See page 320 for details on how to prevent this.
The above SQL statement uses standard recursive processing. The first part of the UNION ALL seeds the temporary table PARENT. The second part recursively joins the temporary table to the source data table until there are no more matches. The final part of the query displays the result set. Imagine that the HIERARCHY table used above is very large and that we also want the above query to be as efficient as possible. In this case, two indexes are required; The first, on PKEY, enables the initial select to run efficiently. The second, on CKEY, makes the join in the recursive part of the query efficient. The second index is arguably more important than the first because the first is only used once, whereas the second index is used for each child of the toplevel parent. List all Children #2
Find all the children of AAA, include in this list the value AAA itself. To satisfy the latter requirement we will change the first SELECT statement (in the recursive code) to select the parent itself instead of the list of immediate children. A DISTINCT is provided in order to ensure that only one line containing the name of the parent (i.e. "AAA") is placed into the temporary PARENT table. NOTE: Before the introduction of recursive SQL processing, it often made sense to define the top-most level in a hierarchical data structure as being a parent-child of itself. For example, the HIERARCHY table might contain a row indicating that "AAA" is a child of "AAA". If the target table has data like this, add another predicate: C.PKEY C.CKEY to the recursive part of the SQL statement to stop the query from looping forever.
312
Introductory Recursion
DB2 V9.7 Cookbook ©
WITH parent (ckey) AS (SELECT DISTINCT pkey FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.ckey FROM hierarchy C ,parent P WHERE P.ckey = C.pkey ) SELECT ckey FROM parent;
ANSWER ====== CKEY ---AAA BBB CCC DDD EEE EEE FFF GGG
HIERARCHY +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |EEE | 44| |DDD |FFF | 5| |FFF |GGG | 5| +---------------+
Figure 858, List all children of AAA In most, but by no means all, business situations, the above SQL statement is more likely to be what the user really wanted than the SQL before. Ask before you code. List Distinct Children
Get a distinct list of all the children of AAA. This query differs from the prior only in the use of the DISTINCT phrase in the final select. WITH parent (ckey) AS (SELECT DISTINCT pkey FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.ckey FROM hierarchy C ,parent P WHERE P.ckey = C.pkey ) SELECT DISTINCT ckey FROM parent;
ANSWER ====== CKEY ---AAA BBB CCC DDD EEE FFF GGG
HIERARCHY +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |EEE | 44| |DDD |FFF | 5| |FFF |GGG | 5| +---------------+
Figure 859, List distinct children of AAA The next thing that we want to do is build a distinct list of children of AAA that we can then use to join to other tables. To do this, we simply define two temporary tables. The first does the recursion and is called PARENT. The second, called DISTINCT_PARENT, takes the output from the first and removes duplicates. WITH parent (ckey) AS (SELECT DISTINCT pkey FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.ckey FROM hierarchy C ,parent P WHERE P.ckey = C.pkey ), distinct_parent (ckey) AS (SELECT DISTINCT ckey FROM parent ) SELECT ckey FROM distinct_parent;
ANSWER ====== CKEY ---AAA BBB CCC DDD EEE FFF GGG
HIERARCHY +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |EEE | 44| |DDD |FFF | 5| |FFF |GGG | 5| +---------------+
Figure 860, List distinct children of AAA Show Item Level
Get a list of all the children of AAA. For each value returned, show its level in the logical hierarchy relative to AAA.
Recursive SQL
313
Graeme Birchall ©
WITH parent (ckey, lvl) AS (SELECT DISTINCT pkey, 0 FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.ckey, P.lvl +1 FROM hierarchy C ,parent P WHERE P.ckey = C.pkey ) SELECT ckey, lvl FROM parent;
ANSWER ======== CKEY LVL ---- --AAA 0 BBB 1 CCC 1 DDD 1 EEE 2 EEE 2 FFF 2 GGG 3
AAA | +-----+-----+ | | | BBB CCC DDD | | +-+ +-+--+ | | | EEE FFF | | GGG
Figure 861, Show item level in hierarchy The above statement has a derived integer field called LVL. In the initial population of the temporary table this level value is set to zero. When subsequent levels are reached, this value in incremented by one. Select Certain Levels
Get a list of all the children of AAA that are less than three levels below AAA. WITH parent (ckey, lvl) AS (SELECT DISTINCT pkey, 0 FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.ckey, P.lvl +1 FROM hierarchy C ,parent P WHERE P.ckey = C.pkey ) SELECT ckey, lvl FROM parent WHERE lvl < 3;
ANSWER ======== CKEY LVL ---- --AAA 0 BBB 1 CCC 1 DDD 1 EEE 2 EEE 2 FFF 2
HIERARCHY +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |EEE | 44| |DDD |FFF | 5| |FFF |GGG | 5| +---------------+
Figure 862, Select rows where LEVEL < 3 The above statement has two main deficiencies:
It will run forever if the database contains an infinite loop.
It may be inefficient because it resolves the whole hierarchy before discarding those levels that are not required.
To get around both of these problems, we can move the level check up into the body of the recursive statement. This will stop the recursion from continuing as soon as we reach the target level. We will have to add "+ 1" to the check to make it logically equivalent: WITH parent (ckey, lvl) AS (SELECT DISTINCT pkey, 0 FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.ckey, P.lvl +1 FROM hierarchy C ,parent P WHERE P.ckey = C.pkey AND P.lvl+1 < 3 ) SELECT ckey, lvl FROM parent;
ANSWER ======== CKEY LVL ---- --AAA 0 BBB 1 CCC 1 DDD 1 EEE 2 EEE 2 FFF 2
AAA | +-----+-----+ | | | BBB CCC DDD | | +-+ +-+--+ | | | EEE FFF | | GGG
Figure 863, Select rows where LEVEL < 3
314
Introductory Recursion
DB2 V9.7 Cookbook ©
The only difference between this statement and the one before is that the level check is now done in the recursive part of the statement. This new level-check predicate has a dual function: It gives us the answer that we want, and it stops the SQL from running forever if the database happens to contain an infinite loop (e.g. DDD was also a parent of AAA). One problem with this general statement design is that it can not be used to list only that data which pertains to a certain lower level (e.g. display only level 3 data). To answer this kind of question efficiently we can combine the above two queries, having appropriate predicates in both places (see next). Select Explicit Level
Get a list of all the children of AAA that are exactly two levels below AAA. WITH parent (ckey, lvl) AS (SELECT DISTINCT pkey, 0 FROM hierarchy WHERE pkey = 'AAA' UNION ALL SELECT C.ckey, P.lvl +1 FROM hierarchy C ,parent P WHERE P.ckey = C.pkey AND P.lvl+1 < 3 ) SELECT ckey, lvl FROM parent WHERE lvl = 2;
ANSWER ======== CKEY LVL ---- --EEE 2 EEE 2 FFF 2
HIERARCHY +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |EEE | 44| |DDD |FFF | 5| |FFF |GGG | 5| +---------------+
Figure 864, Select rows where LEVEL = 2 In the recursive part of the above statement all of the levels up to and including that which is required are obtained. All undesired lower levels are then removed in the final select. Trace a Path - Use Multiple Recursions
Multiple recursive joins can be included in a single query. The joins can run independently, or the output from one recursive join can be used as input to a subsequent. Such code enables one to do the following:
Expand multiple hierarchies in a single query. For example, one might first get a list of all departments (direct and indirect) in a particular organization, and then use the department list as a seed to find all employees (direct and indirect) in each department.
Go down, and then up, a given hierarchy in a single query. For example, one might want to find all of the children of AAA, and then all of the parents. The combined result is the list of objects that AAA is related to via a direct parent-child path.
Go down the same hierarchy twice, and then combine the results to find the matches, or the non-matches. This type of query might be used to, for example, see if two companies own shares in the same subsidiary.
The next example recursively searches the HIERARCHY table for all values that are either a child or a parent (direct or indirect) of the object DDD. The first part of the query gets the list of children, the second part gets the list of parents (but never the value DDD itself), and then the results are combined.
Recursive SQL
315
Graeme Birchall ©
WITH children (kkey, lvl) AS (SELECT ckey, 1 FROM hierarchy WHERE pkey = 'DDD' UNION ALL SELECT H.ckey, C.lvl + 1 FROM hierarchy H ,children C WHERE H.pkey = C.kkey ) ,parents (kkey, lvl) AS (SELECT pkey, -1 FROM hierarchy WHERE ckey = 'DDD' UNION ALL SELECT H.pkey, P.lvl - 1 FROM hierarchy H ,parents P WHERE H.ckey = P.kkey ) SELECT kkey ,lvl FROM children UNION ALL SELECT kkey ,lvl FROM parents;
ANSWER ======== KKEY LVL ---- --AAA -1 EEE 1 FFF 1 GGG 2
AAA | +-----+-----+ | | | BBB CCC DDD | | +-+ +-+--+ | | | EEE FFF | | GGG
Figure 865, Find all children and parents of DDD Extraneous Warning Message
Some recursive SQL statements generate the following warning when the DB2 parser has reason to suspect that the statement may run forever: SQL0347W The recursive common table expression "GRAEME.TEMP1" may contain an infinite loop. SQLSTATE=01605
The text that accompanies this message provides detailed instructions on how to code recursive SQL so as to avoid getting into an infinite loop. The trouble is that even if you do exactly as told you may still get the silly message. To illustrate, the following two SQL statements are almost identical. Yet the first gets a warning and the second does not: WITH temp1 (n1) AS (SELECT id FROM staff WHERE id = 10 UNION ALL SELECT n1 +10 FROM temp1 WHERE n1 < 50 ) SELECT * FROM temp1;
ANSWER ====== N1 -warn 10 20 30 40 50
Figure 866, Recursion - with warning message WITH temp1 (n1) AS (SELECT INT(id) FROM staff WHERE id = 10 UNION ALL SELECT n1 +10 FROM temp1 WHERE n1 < 50 ) SELECT * FROM temp1;
ANSWER ====== N1 -10 20 30 40 50
Figure 867, Recursion - without warning message
316
Introductory Recursion
DB2 V9.7 Cookbook ©
If you know what you are doing, ignore the message.
Logical Hierarchy Flavours Before getting into some of the really nasty stuff, we best give a brief overview of the various kinds of logical hierarchy that exist in the real world and how each is best represented in a relational database. Some typical data hierarchy flavours are shown below. Note that the three on the left form one, mutually exclusive, set and the two on the right another. Therefore, it is possible for a particular hierarchy to be both divergent and unbalanced (or balanced), but not both divergent and convergent. DIVERGENT =========
CONVERGENT ==========
RECURSIVE =========
BALANCED ========
AAA | +-+-+ | | BBB CCC | +-+-+ | | DDD EEE
AAA | +-+-+ | | BBB CCC | | +-+-+-+ | | DDD EEE
AAA+ | +-+-+ | | DDD EEE
AAA | +-+-+ | | BBB CCC | | | +---+ | | | DDD EEE FFF
UNBALANCED ========== AAA | +-+-+ | | BBB CCC | +-+-+ | | DDD EEE
Figure 868, Hierarchy Flavours Divergent Hierarchy
In this flavour of hierarchy, no object has more than one parent. Each object can have none, one, or more than one, dependent child objects. Physical objects (e.g. Geographic entities) tend to be represented in this type of hierarchy. This type of hierarchy will often incorporate the concept of different layers in the hierarchy referring to differing kinds of object - each with its own set of attributes. For example, a Geographic hierarchy might consist of countries, states, cities, and street addresses. A single table can be used to represent this kind of hierarchy in a fully normalized form. One field in the table will be the unique key, another will point to the related parent. Other fields in the table may pertain either to the object in question, or to the relationship between the object and its parent. For example, in the following table the PRICE field has the price of the object, and the NUM field has the number of times that the object occurs in the parent. OBJECTS_RELATES +---------------------+ |KEYO |PKEY |NUM|PRICE| |-----|-----|---|-----| |AAA | | | $10| |BBB |AAA | 1| $21| |CCC |AAA | 5| $23| |DDD |AAA | 20| $25| |EEE |DDD | 44| $33| |FFF |DDD | 5| $34| |GGG |FFF | 5| $44| +---------------------+
AAA | +-----+-----+ | | | BBB CCC DDD | +--+--+ | | EEE FFF | | GGG
Figure 869, Divergent Hierarchy - Table and Layout
Recursive SQL
317
Graeme Birchall ©
Some database designers like to make the arbitrary judgment that every object has a parent, and in those cases where there is no "real" parent, the object considered to be a parent of itself. In the above table, this would mean that AAA would be defined as a parent of AAA. Please appreciate that this judgment call does not affect the objects that the database represents, but it can have a dramatic impact on SQL usage and performance. Prior to the introduction of recursive SQL, defining top level objects as being self-parenting was sometimes a good idea because it enabled one to resolve a hierarchy using a simple join without unions. This same process is now best done with recursive SQL. Furthermore, if objects in the database are defined as self-parenting, the recursive SQL will get into an infinite loop unless extra predicates are provided. Convergent Hierarchy NUMBER OF TABLES: A convergent hierarchy has many-to-many relationships that require two tables for normalized data storage. The other hierarchy types require but a single table.
In this flavour of hierarchy, each object can have none, one, or more than one, parent and/or dependent child objects. Convergent hierarchies are often much more difficult to work with than similar divergent hierarchies. Logical entities, or man-made objects, (e.g. Company Divisions) often have this type of hierarchy. Two tables are required in order to represent this kind of hierarchy in a fully normalized form. One table describes the object, and the other describes the relationships between the objects. OBJECTS +-----------+ |KEYO |PRICE| |-----|-----| |AAA | $10| |BBB | $21| |CCC | $23| |DDD | $25| |EEE | $33| |FFF | $34| |GGG | $44| +-----------+
RELATIONSHIPS +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |EEE | 44| |DDD |FFF | 5| |FFF |GGG | 5| +---------------+
AAA | +-----+-----+ | | | BBB CCC DDD | | +-+ +-+--+ | | | EEE FFF | | GGG
Figure 870, Convergent Hierarchy - Tables and Layout One has to be very careful when resolving a convergent hierarchy to get the answer that the user actually wanted. To illustrate, if we wanted to know how many children AAA has in the above structure the "correct" answer could be six, seven, or eight. To be precise, we would need to know if EEE should be counted twice and if AAA is considered to be a child of itself. Recursive Hierarchy WARNING: Recursive data hierarchies will cause poorly written recursive SQL statements to run forever. See the section titled "Halting Recursive Processing" on page 320 for details on how to prevent this, and how to check that a hierarchy is not recursive.
In this flavour of hierarchy, each object can have none, one, or more than one parent. Also, each object can be a parent and/or a child of itself via another object, or via itself directly. In the business world, this type of hierarchy is almost always wrong. When it does exist, it is often because a standard convergent hierarchy has gone a bit haywire. This database design is exactly the same as the one for a convergent hierarchy. Two tables are (usually) required in order to represent the hierarchy in a fully normalized form. One table describes the object, and the other describes the relationships between the objects.
318
Logical Hierarchy Flavours
DB2 V9.7 Cookbook ©
OBJECTS +-----------+ |KEYO |PRICE| |-----|-----| |AAA | $10| |BBB | $21| |CCC | $23| |DDD | $25| |EEE | $33| |FFF | $34| |GGG | $44| +-----------+
RELATIONSHIPS +---------------+ |PKEY |CKEY |NUM| |-----|-----|---| |AAA |BBB | 1| |AAA |CCC | 5| |AAA |DDD | 20| |CCC |EEE | 33| |DDD |AAA | 99| |DDD |FFF | 5| |DDD |EEE | 44| |FFF |GGG | 5| +---------------+
AAA -+ | | +-+ +-+--+ | | | EEE FFF | | GGG
Figure 871, Recursive Hierarchy - Tables and Layout Prior to the introduction of recursive SQL, it took some non-trivial coding root out recursive data structures in convergent hierarchies. Now it is a no-brainer, see page 320 for details. Balanced & Unbalanced Hierarchies
In some logical hierarchies the distance, in terms of the number of intervening levels, from the top parent entity to its lowest-level child entities is the same for all legs of the hierarchy. Such a hierarchy is considered to be balanced. An unbalanced hierarchy is one where the distance from a top-level parent to a lowest-level child is potentially different for each leg of the hierarchy. AAA | +-----+-----+ | | | BBB CCC DDD | | | | | +-+-+ | | | | EEE FFF GGG HHH
>
AAA | +---+----+ | | | | CCC DDD | | | | +-+ +-+-+ | | | | FFF GGG HHH | | III
Figure 872, Balanced and Unbalanced Hierarchies Balanced hierarchies often incorporate the concept of levels, where a level is a subset of the values in the hierarchy that are all of the same time and are also the same distance from the top level parent. For example, in the balanced hierarchy above each of the three levels shown might refer to a different category of object (e.g. country, state, city). By contrast, in the unbalanced hierarchy above is probable that the objects being represented are all of the same general category (e.g. companies that own other companies). Divergent hierarchies are the most likely to be balanced. Furthermore, balanced and/or divergent hierarchies are the kind that are most often used to do data summation at various intermediate levels. For example, a hierarchy of countries, states, and cities, is likely to be summarized at any level. Data & Pointer Hierarchies
The difference between a data and a pointer hierarchy is not one of design, but of usage. In a pointer schema, the main application tables do not store a description of the logical hierarchy. Instead, they only store the base data. Separate to the main tables are one, or more, related tables that define which hierarchies each base data row belongs to.
Recursive SQL
319
Graeme Birchall ©
Typically, in a pointer hierarchy, the main data tables are much larger and more active than the hierarchical tables. A banking application is a classic example of this usage pattern. There is often one table that contains core customer information and several related tables that enable one to do analysis by customer category. A data hierarchy is an altogether different beast. An example would be a set of tables that contain information on all that parts that make up an aircraft. In this kind of application the most important information in the database is often that which pertains to the relationships between objects. These tend to be very complicated often incorporating the attributes: quantity, direction, and version. Recursive processing of a data hierarchy will often require that one does a lot more than just find all dependent keys. For example, to find the gross weight of an aircraft from such a database one will have to work with both the quantity and weight of all dependent objects. Those objects that span sub-assembles (e.g. a bolt connecting to engine to the wing) must not be counted twice, missed out, nor assigned to the wrong sub-grouping. As always, such questions are essentially easy to answer, the trick is to get the right answer.
Halting Recursive Processing One occasionally encounters recursive hierarchical data structures (i.e. where the parent item points to the child, which then points back to the parent). This section describes how to write recursive SQL statements that can process such structures without running forever. There are three general techniques that one can use:
Stop processing after reaching a certain number of levels.
Keep a record of where you have been, and if you ever come back, either fail or in some other way stop recursive processing.
Keep a record of where you have been, and if you ever come back, simply ignore that row and keep on resolving the rest of hierarchy.
Sample Table DDL & DML
The following table is a normalized representation of the recursive hierarchy on the right. Note that AAA and DDD are both a parent and a child of each other. TROUBLE +---------+ |PKEY|CKEY| |----|----| |AAA |BBB | |AAA |CCC | |AAA |DDD | |CCC |EEE | |DDD |AAA | |DDD |FFF | |DDD |EEE | |FFF |GGG | +---------+
AAA -+ | | +-+ +-+--+ | | | EEE FFF | | GGG
|DDD |AAA | | | | points back to |DDD |FFF | EEE FFF the hierarchy |DDD |EEE | | parent. |FFF |GGG | | +---------+ GGG
Figure 878, Show path, and rows in loop Now we can get rid of the level check, and instead use the LOCATE_BLOCK function to avoid loops in the data: WITH parent (pkey, ckey, lvl, path) AS ANSWER (SELECT DISTINCT ========================== pkey PKEY CKEY LVL PATH ,pkey ---- ----- -- -----------,0 AAA AAA 0 AAA ,VARCHAR(pkey,20) AAA BBB 1 AAABBB FROM trouble AAA CCC 1 AAACCC WHERE pkey = 'AAA' AAA DDD 1 AAADDD UNION ALL CCC EEE 2 AAACCCEEE SELECT C.pkey DDD EEE 2 AAADDDEEE ,C.ckey DDD FFF 2 AAADDDFFF ,P.lvl + 1 FFF GGG 3 AAADDDFFFGGG ,P.path || C.ckey FROM trouble C ,parent P WHERE P.ckey = C.pkey AND LOCATE_BLOCK(C.ckey,P.path) = 0 ) SELECT * FROM parent;
Figure 879, Use LOCATE_BLOCK function to stop recursion The next query is the same as the previous, except that instead of excluding all loops from the answer-set, it marks them as such, and gets the first item, but goes no further;
Recursive SQL
323
Graeme Birchall ©
WITH parent (pkey, ckey, lvl, path, loop) AS (SELECT DISTINCT pkey ,pkey ,0 ,VARCHAR(pkey,20) ANSWER ,0 =============================== FROM trouble PKEY CKEY LVL PATH LOOP WHERE pkey = 'AAA' ---- ---- --- ------------ ---UNION ALL AAA AAA 0 AAA 0 SELECT C.pkey AAA BBB 1 AAABBB 0 ,C.ckey AAA CCC 1 AAACCC 0 ,P.lvl + 1 AAA DDD 1 AAADDD 0 ,P.path || C.ckey CCC EEE 2 AAACCCEEE 0 ,LOCATE_BLOCK(C.ckey,P.path) DDD AAA 2 AAADDDAAA 1 FROM trouble C DDD EEE 2 AAADDDEEE 0 ,parent P DDD FFF 2 AAADDDFFF 0 WHERE P.ckey = C.pkey FFF GGG 3 AAADDDFFFGGG 0 AND P.loop = 0 ) SELECT * FROM parent;
Figure 880, Use LOCATE_BLOCK function to stop recursion The next query tosses in another predicate (in the final select) to only list those rows that point back to a previously processed parent: WITH parent (pkey, ckey, lvl, path, loop) (SELECT DISTINCT pkey ,pkey ,0 ,VARCHAR(pkey,20) ,0 FROM trouble WHERE pkey = 'AAA' UNION ALL SELECT C.pkey ,C.ckey ,P.lvl + 1 ,P.path || C.ckey ,LOCATE_BLOCK(C.ckey,P.path) FROM trouble C ,parent P WHERE P.ckey = C.pkey AND P.loop = 0 ) SELECT pkey ,ckey FROM parent WHERE loop > 0;
AS
ANSWER ========= PKEY CKEY ---- ---DDD AAA
This row ===> points back to the hierarchy parent.
TROUBLE +---------+ |PKEY|CKEY| |----|----| |AAA |BBB | |AAA |CCC | |AAA |DDD | |CCC |EEE | |DDD |AAA | |DDD |FFF | |DDD |EEE | |FFF |GGG | +---------+
Figure 881,List rows that point back to a parent To delete the offending rows from the table, all one has to do is insert the above values into a temporary table, then delete those rows in the TROUBLE table that match. However, before one does this, one has decide which rows are the ones that should not be there. In the above query, we started processing at AAA, and then said that any row that points back to AAA, or to some child or AAA, is causing a loop. We thus identified the row from DDD to AAA as being a problem. But if we had started at the value DDD, we would have said instead that the row from AAA to DDD was the problem. The point to remember her is that the row you decide to delete is a consequence of the row that you decided to define as your starting point.
324
Halting Recursive Processing
DB2 V9.7 Cookbook ©
DECLARE GLOBAL TEMPORARY TABLE SESSION.del_list (pkey CHAR(03) NOT NULL ,ckey CHAR(03) NOT NULL) ON COMMIT PRESERVE ROWS; INSERT INTO SESSION.del_list WITH parent (pkey, ckey, lvl, path, loop) (SELECT DISTINCT pkey ,pkey ,0 ,VARCHAR(pkey,20) ,0 FROM trouble WHERE pkey = 'AAA' UNION ALL SELECT C.pkey ,C.ckey ,P.lvl + 1 ,P.path || C.ckey ,LOCATE_BLOCK(C.ckey,P.path) FROM trouble C ,parent P WHERE P.ckey = C.pkey AND P.loop = 0 ) SELECT pkey ,ckey FROM parent WHERE loop > 0;
AS
This row ===> points back to the hierarchy parent.
DELETE FROM trouble WHERE (pkey,ckey) IN (SELECT pkey, ckey FROM SESSION.del_list);
TROUBLE +---------+ |PKEY|CKEY| |----|----| |AAA |BBB | |AAA |CCC | |AAA |DDD | |CCC |EEE | |DDD |AAA | |DDD |FFF | |DDD |EEE | |FFF |GGG | +---------+
AAA -+ | | +-+ +-+--+ | | | EEE FFF | | GGG
Figure 882, Delete rows that loop back to a parent Working with Other Key Types
The LOCATE_BLOCK solution shown above works fine, as long as the key in question is a fixed length character field. If it isn't, it can be converted to one, depending on what it is:
Cast VARCHAR columns as type CHAR.
Convert other field types to character using the HEX function.
Keeping the Hierarchy Clean
Rather that go searching for loops, one can toss in a couple of triggers that will prevent the table from every getting data loops in the first place. There will be one trigger for inserts, and another for updates. Both will have the same general logic:
For each row inserted/updated, retain the new PKEY value.
Recursively scan the existing rows, starting with the new CKEY value.
Compare each existing CKEY value retrieved to the new PKEY value. If it matches, the changed row will cause a loop, so flag an error.
If no match is found, allow the change.
Here is the insert trigger:
Recursive SQL
325
Graeme Birchall ©
CREATE TRIGGER TBL_INS NO CASCADE BEFORE INSERT ON trouble REFERENCING NEW AS NNN This trigger FOR EACH ROW MODE DB2SQL would reject WITH temp (pkey, ckey) AS insertion of (VALUES (NNN.pkey this row. ,NNN.ckey) | UNION ALL | SELECT TTT.pkey +---> ,CASE WHEN TTT.ckey = TBL.pkey THEN RAISE_ERROR('70001','LOOP FOUND') ELSE TBL.ckey END FROM trouble TBL ,temp TTT WHERE TTT.ckey = TBL.pkey ) SELECT * FROM temp;
TROUBLE +---------+ |PKEY|CKEY| |----|----| |AAA |BBB | |AAA |CCC | |AAA |DDD | |CCC |EEE | |DDD |AAA | |DDD |FFF | |DDD |EEE | |FFF |GGG | +---------+
Figure 883, INSERT trigger Here is the update trigger: CREATE TRIGGER TBL_UPD NO CASCADE BEFORE UPDATE OF pkey, ckey ON trouble REFERENCING NEW AS NNN FOR EACH ROW MODE DB2SQL WITH temp (pkey, ckey) AS (VALUES (NNN.pkey ,NNN.ckey) UNION ALL SELECT TTT.pkey ,CASE WHEN TTT.ckey = TBL.pkey THEN RAISE_ERROR('70001','LOOP FOUND') ELSE TBL.ckey END FROM trouble TBL ,temp TTT WHERE TTT.ckey = TBL.pkey ) SELECT * FROM temp;
Figure 884, UPDATE trigger Given the above preexisting TROUBLE data (absent the DDD to AAA row), the following statements would be rejected by the above triggers: INSERT INTO trouble VALUES('GGG','AAA'); UPDATE trouble SET ckey = 'AAA' WHERE pkey = 'FFF'; UPDATE trouble SET pkey = 'GGG' WHERE ckey = 'DDD';
Figure 885, Invalid DML statements Observe that neither of the above triggers use the LOCATE_BLOCK function to find a loop. This is because these triggers are written assuming that the table is currently loop free. If this is not the case, they may run forever. The LOCATE_BLOCK function enables one to check every row processed, to see if one has been to that row before. In the above triggers, only the start position is checked for loops. So if there was a loop that did not encompass the start position, the LOCATE_BLOCK check would find it, but the code used in the triggers would not.
326
Halting Recursive Processing
DB2 V9.7 Cookbook ©
Clean Hierarchies and Efficient Joins Introduction
One of the more difficult problems in any relational database system involves joining across multiple hierarchical data structures. The task is doubly difficult when one or more of the hierarchies involved is a data structure that has to be resolved using recursive processing. In this section, we will describe how one can use a mixture of tables and triggers to answer this kind of query very efficiently. A typical question might go as follows: Find all matching rows where the customer is in some geographic region, and the item sold is in some product category, and person who made the sale is in some company sub-structure. If each of these qualifications involves expanding a hierarchy of object relationships of indeterminate and/or nontrivial depth, then a simple join or standard data denormalization will not work. In DB2, one can answer this kind of question by using recursion to expand each of the data hierarchies. Then the query would join (sans indexes) the various temporary tables created by the recursive code to whatever other data tables needed to be accessed. Unfortunately, the performance will probably be lousy. Alternatively, one can often efficiently answer this general question using a set of suitably indexed summary tables that are an expanded representation of each data hierarchy. With these tables, the DB2 optimizer can much more efficiently join to other data tables, and so deliver suitable performance. In this section, we will show how to make these summary tables and, because it is a prerequisite, also show how to ensure that the related base tables do not have recursive data structures. Two solutions will be described: One that is simple and efficient, but which stops updates to key values. And another that imposes fewer constraints, but which is a bit more complicated. Limited Update Solution
Below on the left is a hierarchy of data items. This is a typical unbalanced, non-recursive data hierarchy. In the center is a normalized representation of this hierarchy. The only thing that is perhaps a little unusual here is that an item at the top of a hierarchy (e.g. AAA) is deemed to be a parent of itself. On the right is an exploded representation of the same hierarchy. AAA | BBB | +-----+ | | CCC EEE | DDD
HIERARCHY#1 +--------------------+ |KEYY|PKEY|DATA | |----|----|----------| |AAA |AAA |SOME DATA | |BBB |AAA |MORE DATA | |CCC |BBB |MORE JUNK | |DDD |CCC |MORE JUNK | |EEE |BBB |JUNK DATA | +--------------------+
EXPLODED#1 +-------------+ |PKEY|CKEY|LVL| |----|----|---| |AAA |AAA | 0| |AAA |BBB | 1| |AAA |CCC | 2| |AAA |DDD | 3| |AAA |EEE | 2| |BBB |BBB | 0| |BBB |CCC | 1| |BBB |DDD | 2| |BBB |EEE | 1| |CCC |CCC | 0| |CCC |DDD | 1| |DDD |DDD | 0| |EEE |EEE | 0| +-------------+
Figure 886, Data Hierarchy, with normalized and exploded representations
Recursive SQL
327
Graeme Birchall ©
Below is the CREATE code for the above normalized table and a dependent trigger: CREATE TABLE hierarchy#1 (keyy CHAR(3) NOT NULL ,pkey CHAR(3) NOT NULL ,data VARCHAR(10) ,CONSTRAINT hierarchy11 PRIMARY KEY(keyy) ,CONSTRAINT hierarchy12 FOREIGN KEY(pkey) REFERENCES hierarchy#1 (keyy) ON DELETE CASCADE); CREATE TRIGGER HIR#1_UPD NO CASCADE BEFORE UPDATE OF pkey ON hierarchy#1 REFERENCING NEW AS NNN OLD AS OOO FOR EACH ROW MODE DB2SQL WHEN (NNN.pkey OOO.pkey) SIGNAL SQLSTATE '70001' ('CAN NOT UPDATE pkey');
Figure 887, Hierarchy table that does not allow updates to PKEY Note the following:
The KEYY column is the primary key, which ensures that each value must be unique, and that this field can not be updated.
The PKEY column is a foreign key of the KEYY column. This means that this field must always refer to a valid KEYY value. This value can either be in another row (if the new row is being inserted at the bottom of an existing hierarchy), or in the new row itself (if a new independent data hierarchy is being established).
The ON DELETE CASCADE referential integrity rule ensures that when a row is deleted, all dependent rows are also deleted.
The TRIGGER prevents any updates to the PKEY column. This is a BEFORE trigger, which means that it stops the update before it is applied to the database.
All of the above rules and restrictions act to prevent either an insert or an update for ever acting on any row that is not at the bottom of a hierarchy. Consequently, it is not possible for a hierarchy to ever exist that contains a loop of multiple data items. Creating an Exploded Equivalent
Once we have ensured that the above table can never have recursive data structures, we can define a dependent table that holds an exploded version of the same hierarchy. Triggers will be used to keep the two tables in sync. Here is the CREATE code for the table: CREATE TABLE exploded#1 (pkey CHAR(4) NOT NULL ,ckey CHAR(4) NOT NULL ,lvl SMALLINT NOT NULL ,PRIMARY KEY(pkey,ckey));
Figure 888, Exploded table CREATE statement The following trigger deletes all dependent rows from the exploded table whenever a row is deleted from the hierarchy table: CREATE TRIGGER EXP#1_DEL AFTER DELETE ON hierarchy#1 REFERENCING OLD AS OOO FOR EACH ROW MODE DB2SQL DELETE FROM exploded#1 WHERE ckey = OOO.keyy;
Figure 889, Trigger to maintain exploded table after delete in hierarchy table
328
Clean Hierarchies and Efficient Joins
DB2 V9.7 Cookbook ©
The next trigger is run every time a row is inserted into the hierarchy table. It uses recursive code to scan the hierarchy table upwards, looking for all parents of the new row. The resultset is then inserted into the exploded table: CREATE TRIGGER EXP#1_INS AFTER INSERT ON hierarchy#1 REFERENCING NEW AS NNN FOR EACH ROW MODE DB2SQL INSERT INTO exploded#1 WITH temp(pkey, ckey, lvl) AS (VALUES (NNN.keyy ,NNN.keyy ,0) UNION ALL SELECT N.pkey ,NNN.keyy ,T.lvl +1 FROM temp T ,hierarchy#1 N WHERE N.keyy = T.pkey AND N.keyy N.pkey ) SELECT * FROM temp;
HIERARCHY#1 +--------------+ |KEYY|PKEY|DATA| |----|----|----| |AAA |AAA |S...| |BBB |AAA |M...| |CCC |BBB |M...| |DDD |CCC |M...| |EEE |BBB |J...| +--------------+
EXPLODED#1 +-------------+ |PKEY|CKEY|LVL| |----|----|---| |AAA |AAA | 0| |AAA |BBB | 1| |AAA |CCC | 2| |AAA |DDD | 3| |AAA |EEE | 2| |BBB |BBB | 0| |BBB |CCC | 1| |BBB |DDD | 2| |BBB |EEE | 1| |CCC |CCC | 0| |CCC |DDD | 1| |DDD |DDD | 0| |EEE |EEE | 0| +-------------+
Figure 890, Trigger to maintain exploded table after insert in hierarchy table There is no update trigger because updates are not allowed to the hierarchy table. Querying the Exploded Table
Once supplied with suitable indexes, the exploded table can be queried like any other table. It will always return the current state of the data in the related hierarchy table. SELECT * FROM exploded#1 WHERE pkey = :host-var ORDER BY pkey ,ckey ,lvl;
Figure 891, Querying the exploded table Full Update Solution
Not all applications want to limit updates to the data hierarchy as was done above. In particular, they may want the user to be able to move an object, and all its dependents, from one valid point (in a data hierarchy) to another. This means that we cannot prevent valid updates to the PKEY value. Below is the CREATE statement for a second hierarchy table. The only difference between this table and the previous one is that there is now an ON UPDATE RESTRICT clause. This prevents updates to PKEY that do not point to a valid KEYY value – either in another row, or in the row being updated: CREATE TABLE hierarchy#2 (keyy CHAR(3) NOT NULL ,pkey CHAR(3) NOT NULL ,data VARCHAR(10) ,CONSTRAINT NO_loopS21 PRIMARY KEY(keyy) ,CONSTRAINT NO_loopS22 FOREIGN KEY(pkey) REFERENCES hierarchy#2 (keyy) ON DELETE CASCADE ON UPDATE RESTRICT);
Figure 892, Hierarchy table that allows updates to PKEY
Recursive SQL
329
Graeme Birchall ©
The previous hierarchy table came with a trigger that prevented all updates to the PKEY field. This table comes instead with a trigger than checks to see that such updates do not result in a recursive data structure. It starts out at the changed row, then works upwards through the chain of PKEY values. If it ever comes back to the original row, it flags an error: CREATE TRIGGER HIR#2_UPD HIERARCHY#2 NO CASCADE BEFORE UPDATE OF pkey ON hierarchy#2 +--------------+ REFERENCING NEW AS NNN |KEYY|PKEY|DATA| OLD AS OOO |----|----|----| FOR EACH ROW MODE DB2SQL |AAA |AAA |S...| WHEN (NNN.pkey OOO.pkey |BBB |AAA |M...| AND NNN.pkey NNN.keyy) |CCC |BBB |M...| WITH temp (keyy, pkey) AS |DDD |CCC |M...| (VALUES (NNN.keyy |EEE |BBB |J...| ,NNN.pkey) +--------------+ UNION ALL SELECT LP2.keyy ,CASE WHEN LP2.keyy = NNN.keyy THEN RAISE_ERROR('70001','LOOP FOUND') ELSE LP2.pkey END FROM hierarchy#2 LP2 ,temp TMP WHERE TMP.pkey = LP2.keyy AND TMP.keyy TMP.pkey ) SELECT * FROM temp;
Figure 893, Trigger to check for recursive data structures before update of PKEY NOTE: The above is a BEFORE trigger, which means that it gets run before the change is applied to the database. By contrast, the triggers that maintain the exploded table are all AFTER triggers. In general, one uses before triggers check for data validity, while after triggers are used to propagate changes. Creating an Exploded Equivalent
The following exploded table is exactly the same as the previous. It will be maintained in sync with changes to the related hierarchy table: CREATE TABLE exploded#2 (pkey CHAR(4) NOT NULL ,ckey CHAR(4) NOT NULL ,lvl SMALLINT NOT NULL ,PRIMARY KEY(pkey,ckey));
Figure 894, Exploded table CREATE statement Three triggers are required to maintain the exploded table in sync with the related hierarchy table. The first two, which handle deletes and inserts, are the same as what were used previously. The last, which handles updates, is new (and quite tricky). The following trigger deletes all dependent rows from the exploded table whenever a row is deleted from the hierarchy table: CREATE TRIGGER EXP#2_DEL AFTER DELETE ON hierarchy#2 REFERENCING OLD AS OOO FOR EACH ROW MODE DB2SQL DELETE FROM exploded#2 WHERE ckey = OOO.keyy;
Figure 895, Trigger to maintain exploded table after delete in hierarchy table
330
Clean Hierarchies and Efficient Joins
DB2 V9.7 Cookbook ©
The next trigger is run every time a row is inserted into the hierarchy table. It uses recursive code to scan the hierarchy table upwards, looking for all parents of the new row. The resultset is then inserted into the exploded table: CREATE TRIGGER EXP#2_INS AFTER INSERT ON hierarchy#2 REFERENCING NEW AS NNN FOR EACH ROW MODE DB2SQL INSERT INTO exploded#2 WITH temp(pkey, ckey, lvl) AS (SELECT NNN.keyy ,NNN.keyy ,0 FROM hierarchy#2 WHERE keyy = NNN.keyy UNION ALL SELECT N.pkey ,NNN.keyy ,T.lvl +1 FROM temp T ,hierarchy#2 N WHERE N.keyy = T.pkey AND N.keyy N.pkey ) SELECT * FROM temp;
HIERARCHY#2 +--------------+ |KEYY|PKEY|DATA| |----|----|----| |AAA |AAA |S...| |BBB |AAA |M...| |CCC |BBB |M...| |DDD |CCC |M...| |EEE |BBB |J...| +--------------+
EXPLODED#2 +-------------+ |PKEY|CKEY|LVL| |----|----|---| |AAA |AAA | 0| |AAA |BBB | 1| |AAA |CCC | 2| |AAA |DDD | 3| |AAA |EEE | 2| |BBB |BBB | 0| |BBB |CCC | 1| |BBB |DDD | 2| |BBB |EEE | 1| |CCC |CCC | 0| |CCC |DDD | 1| |DDD |DDD | 0| |EEE |EEE | 0| +-------------+
Figure 896, Trigger to maintain exploded table after insert in hierarchy table The next trigger is run every time a PKEY value is updated in the hierarchy table. It deletes and then reinserts all rows pertaining to the updated object, and all it’s dependents. The code goes as follows: Delete all rows that point to children of the row being updated. The row being updated is also considered to be a child. In the following insert, first use recursion to get a list of all of the children of the row that has been updated. Then work out the relationships between all of these children and all of their parents. Insert this second result-set back into the exploded table. CREATE TRIGGER EXP#2_UPD AFTER UPDATE OF pkey ON hierarchy#2 REFERENCING OLD AS OOO NEW AS NNN FOR EACH ROW MODE DB2SQL BEGIN ATOMIC DELETE FROM exploded#2 WHERE ckey IN (SELECT ckey FROM exploded#2 WHERE pkey = OOO.keyy); INSERT INTO exploded#2 WITH temp1(ckey) AS (VALUES (NNN.keyy) UNION ALL SELECT N.keyy FROM temp1 T ,hierarchy#2 N WHERE N.pkey = T.ckey AND N.pkey N.keyy )
Figure 897, Trigger to run after update of PKEY in hierarchy table (part 1 of 2)
Recursive SQL
331
Graeme Birchall ©
,temp2(pkey, ckey, lvl) AS (SELECT ckey ,ckey ,0 FROM temp1 UNION ALL SELECT N.pkey ,T.ckey ,T.lvl +1 FROM temp2 T ,hierarchy#2 N WHERE N.keyy = T.pkey AND N.keyy N.pkey ) SELECT * FROM temp2; END
Figure 898, Trigger to run after update of PKEY in hierarchy table (part 2 of 2) NOTE: The above trigger lacks a statement terminator because it contains atomic SQL, which means that the semi-colon can not be used. Choose anything you like. Querying the Exploded Table
Once supplied with suitable indexes, the exploded table can be queried like any other table. It will always return the current state of the data in the related hierarchy table. SELECT * FROM exploded#2 WHERE pkey = :host-var ORDER BY pkey ,ckey ,lvl;
Figure 899, Querying the exploded table Below are some suggested indexes:
PKEY, CKEY (already defined as part of the primary key).
CKEY, PKEY (useful when joining to this table).
332
Clean Hierarchies and Efficient Joins
DB2 V9.7 Cookbook ©
Triggers A trigger initiates an action whenever a row, or set of rows, is changed. The change can be either an insert, update or delete. NOTE. The DB2 Application Development Guide: Programming Server Applications is an excellent source of information on using triggers. The SQL Reference has all the basics.
Trigger Syntax CREATE TRIGGER
trigger-name
NO CASCADE BEFORE AFTER INSTEAD OF
INSERT
ON
table-name view-name
DELETE UPDATE
, OF
column-name
REFERENCING
AS
OLD
AS
NEW OLD_TABLE
AS
correlation-name correlation-name identifier
AS NEW_TABLE
identifier
FOR EACH STATEMENT WHEN
FOR EACH ROW
(
search-condition
)
triggered-action label:
Figure 900, Create Trigger syntax Usage Notes Trigger Types
A BEFORE trigger is run before the row is changed. It is typically used to change the values being entered (e.g. set a field to the current date), or to flag an error. It cannot be used to initiate changes in other tables.
An AFTER trigger is run after the row is changed. It can do everything a before trigger can do, plus modify data in other tables or systems (e.g. it can insert a row into an audit table after an update).
An INSTEAD OF trigger is used in a view to do something instead of the action that the user intended (e.g. do an insert instead of an update). There can be only one instead of trigger per possible DML type on a given view.
Triggers
333
Graeme Birchall ©
NOTE: See the chapter titled "Retaining a Record" on page 351 for a sample application that uses INSTEAD OF triggers to record all changes to the data in a set of tables. Action Type
Each trigger applies to a single kind of DML action (i.e. insert, update, or delete). With the exception of instead of triggers, there can be as many triggers per action and per table as desired. An update trigger can be limited to changes to certain columns.
Object Type
A table can have both BEFORE and AFTER triggers. The former have to be defined FOR EACH ROW.
A view can have INSTEAD OF triggers (up to three - one per DML type).
Referencing
In the body of the trigger the object being changed can be referenced using a set of optional correlation names:
OLD refers to each individual row before the change (does not apply to an insert).
NEW refers to each individual row after the change (does not apply to a delete).
OLD_TABLE refers to the set of rows before the change (does not apply to an insert).
NEW_TABLE refers to the set of rows after the change (does to apply to a delete).
Application Scope
A trigger defined FOR EACH STATEMENT is invoked once per statement.
A trigger defined FOR EACH ROW is invoked once per individual row changed. NOTE: If one defines two FOR EACH ROW triggers, the first is applied for all rows before the second is run. To do two separate actions per row, one at a time, one has to define a single trigger that includes the two actions in a single compound SQL statement.
When Check
One can optionally include some predicates so that the body of the trigger is only invoked when certain conditions are true. Trigger Usage
A trigger can be invoked whenever one of the following occurs:
A row in a table is inserted, updated, or deleted.
An (implied) row in a view is inserted, updated, or deleted.
A referential integrity rule on a related table causes a cascading change (i.e. delete or set null) to the triggered table.
A trigger on an unrelated table or view is invoked - and that trigger changes rows in the triggered table.
If no rows are changed, a trigger defined FOR EACH ROW is not run, while a trigger defined FOR EACH STATEMENT is still run. To prevent the latter from doing anything when this happens, add a suitable WHEN check.
334
Trigger Syntax
DB2 V9.7 Cookbook ©
Trigger Examples This section uses a set of simple sample tables to illustrate general trigger usage. Sample Tables CREATE TABLE cust_balance (cust# INTEGER GENERATED ALWAYS ,status CHAR(2) ,balance DECIMAL(18,2) ,num_trans INTEGER ,cur_ts TIMESTAMP ,PRIMARY KEY (cust#));
NOT NULL AS IDENTITY NOT NULL NOT NULL NOT NULL NOT NULL
CREATE TABLE (cust# ,trans# ,balance ,bgn_ts ,end_ts ,PRIMARY KEY
cust_history INTEGER NOT INTEGER NOT DECIMAL(18,2) NOT TIMESTAMP NOT TIMESTAMP NOT (cust#, bgn_ts));
CREATE TABLE (min_cust# ,max_cust# ,rows_tot ,change_val ,change_type ,cur_ts ,PRIMARY KEY
cust_trans INTEGER INTEGER INTEGER DECIMAL(18,2) CHAR(1) TIMESTAMP (cur_ts));
NULL NULL NULL NULL NULL
NOT NULL NOT NULL NOT NULL
Every state of a row in the balance table will be recorded in the history table. Every valid change to the balance table will be recorded in the transaction table.
Figure 901, Sample Tables Before Row Triggers - Set Values
The first trigger below overrides whatever the user enters during the insert, and before the row is inserted, sets both the cur-ts and number-of-trans columns to their correct values: CREATE TRIGGER cust_bal_ins1 NO CASCADE BEFORE INSERT ON cust_balance REFERENCING NEW AS nnn FOR EACH ROW MODE DB2SQL SET nnn.cur_ts = CURRENT TIMESTAMP ,nnn.num_trans = 1;
Figure 902, Before insert trigger - set values The following trigger does the same before an update: CREATE TRIGGER cust_bal_upd1 NO CASCADE BEFORE UPDATE ON cust_balance REFERENCING NEW AS nnn OLD AS ooo FOR EACH ROW MODE DB2SQL SET nnn.cur_ts = CURRENT TIMESTAMP ,nnn.num_trans = ooo.num_trans + 1;
Figure 903, Before update trigger - set values
Triggers
335
Graeme Birchall ©
Before Row Trigger - Signal Error
The next trigger will flag an error (and thus fail the update) if the customer balance is reduced by too large a value: CREATE TRIGGER cust_bal_upd2 NO CASCADE BEFORE UPDATE OF balance ON cust_balance REFERENCING NEW AS nnn OLD AS ooo FOR EACH ROW MODE DB2SQL WHEN (ooo.balance - nnn.balance > 1000) SIGNAL SQLSTATE VALUE '71001' SET MESSAGE_TEXT = 'Cannot withdraw > 1000';
Figure 904, Before Trigger - flag error After Row Triggers - Record Data States
The three triggers in this section record the state of the data in the customer table. The first is invoked after each insert. It records the new data in the customer-history table: CREATE TRIGGER cust_his_ins1 AFTER INSERT ON cust_balance REFERENCING NEW AS nnn FOR EACH ROW MODE DB2SQL INSERT INTO cust_history VALUES (nnn.cust# ,nnn.num_trans ,nnn.balance ,nnn.cur_ts ,'9999-12-31-24.00.00');
Figure 905, After Trigger - record insert The next trigger is invoked after every update of a row in the customer table. It first runs an update (of the old history row), and then does an insert. Because this trigger uses a compound SQL statement, it cannot use the semi-colon as the statement delimiter: CREATE TRIGGER cust_his_upd1 AFTER UPDATE ON cust_balance REFERENCING OLD AS ooo NEW AS nnn FOR EACH ROW MODE DB2SQL BEGIN ATOMIC UPDATE cust_history SET end_ts = CURRENT TIMESTAMP WHERE cust# = ooo.cust# AND bgn_ts = ooo.cur_ts; INSERT INTO cust_history VALUES (nnn.cust# ,nnn.num_trans ,nnn.balance ,nnn.cur_ts ,'9999-12-31-24.00.00'); END
Figure 906, After Trigger - record update
336
Trigger Examples
DB2 V9.7 Cookbook ©
Notes
The above trigger relies on the fact that the customer-number cannot change (note: it is generated always) to link the two rows in the history table together. In other words, the old row will always have the same customer-number as the new row.
The above also trigger relies on the presence of the cust_bal_upd1 before trigger (see page 335) to set the nnn.cur_ts value to the current timestamp.
The final trigger records a delete by doing an update to the history table: CREATE TRIGGER cust_his_del1 AFTER DELETE ON cust_balance REFERENCING OLD AS ooo FOR EACH ROW MODE DB2SQL UPDATE cust_history SET end_ts = CURRENT TIMESTAMP WHERE cust# = ooo.cust# AND bgn_ts = ooo.cur_ts;
Figure 907, After Trigger - record delete After Statement Triggers - Record Changes
The following three triggers record every type of change (i.e. insert, update, or delete) to any row, or set of rows (including an empty set) in the customer table. They all run an insert that records the type and number of rows changed: CREATE TRIGGER trans_his_ins1 AFTER INSERT ON cust_balance REFERENCING NEW_TABLE AS newtab FOR EACH STATEMENT MODE DB2SQL INSERT INTO cust_trans SELECT MIN(cust#) ,MAX(cust#) ,COUNT(*) ,SUM(balance) ,'I' ,CURRENT TIMESTAMP FROM newtab;
Figure 908, After Trigger - record insert CREATE TRIGGER trans_his_upd1 AFTER UPDATE ON cust_balance REFERENCING OLD_TABLE AS oldtab NEW_TABLE AS newtab FOR EACH STATEMENT MODE DB2SQL INSERT INTO cust_trans SELECT MIN(nt.cust#) ,MAX(nt.cust#) ,COUNT(*) ,SUM(nt.balance - ot.balance) ,'U' ,CURRENT TIMESTAMP FROM oldtab ot ,newtab nt WHERE ot.cust# = nt.cust#;
Figure 909, After Trigger - record update
Triggers
337
Graeme Birchall ©
CREATE TRIGGER trans_his_del1 AFTER DELETE ON cust_balance REFERENCING OLD_TABLE AS oldtab FOR EACH STATEMENT MODE DB2SQL INSERT INTO cust_trans SELECT MIN(cust#) ,MAX(cust#) ,COUNT(*) ,SUM(balance) ,'D' ,CURRENT TIMESTAMP FROM oldtab;
Figure 910, After Trigger - record delete Notes
If the DML statement changes no rows, the OLD or NEW table referenced by the trigger will be empty, but still exist, and a SELECT COUNT(*) on the (empty) table will return a zero, which will then be inserted.
Any DML statements that failed (e.g. stopped by the before trigger), or that were subsequently rolled back, will not be recorded in the transaction table.
Examples of Usage
The following DML statements were run against the customer table: INSERT INTO cust_balance (status, balance) VALUES ('C',123.45); INSERT INTO cust_balance (status, balance) VALUES ('C',000.00); INSERT INTO cust_balance (status, balance) VALUES ('D', -1.00); UPDATE cust_balance SET balance = balance + 123 WHERE cust# us_dollars(0)) ,CONSTRAINT u2 FOREIGN KEY (cust_id) REFERENCES customer_balance ON DELETE RESTRICT); COMMIT; CREATE INDEX us_sales_cust ON us_sales (cust_id);
Figure 918, US-Sales table DDL The following business rules are enforced above:
The invoice# is defined as the primary key, which automatically generates a unique index on the field, and also prevents updates.
The sale-value uses the type us-dollars.
Constraint U1 checks that the sale-value is always greater than zero.
Constraint U2 checks that the customer-ID exists in the customer-balance table, and also prevents rows from being deleted from the latter if there is a related row in this table.
All of the columns are defined as NOT NULL, so a value must be provided for each.
A secondary non-unique index is defined on customer-ID, so that deletes to the customerbalance table (which require checking this table for related customer-ID rows) are as efficient as possible.
The CUST_UPDATE_TS column is generated always (by DB2) and gets a unique value that is the current timestamp.
Generated Always Timestamp Columns
A TIMESTAMP column that is defined as GENERATED ALWAYS will get a value that is unique for all rows in the table. This value will usually be the CURRENT TIMESTAMP of the last insert or update of the row. However, if more than row was inserted or updated in a single stmt, the secondary rows (updated) will get a value that is equal to the CURRENT TIMESTAMP special register, plus "n" microseconds, where "n" goes up in steps of 1. One consequence of the above logic is that some rows changed will get a timestamp value that is ahead of the CURRENT TIMESTAMP special register. This can cause problems if one is relying on this value to find all rows that were changed before the start of the query. To illus-
344
Sample Application
DB2 V9.7 Cookbook ©
trate, imagine that one inserted multiple rows (in a single insert) into the US_SALES table, and then immediately ran the following query: SELECT FROM WHERE
* us_sales sale_update_ts 1000 ANSWER ORDER BY #rows DESC ============================ FOR FETCH ONLY SCHEMA TABNAME #ROWS WITH UR; ------ --------------- ----SYSIBM SYSCOLUMNS 3518 SYSIBM SYSROUTINEPARMS 2035
Figure 963, List tables never had RUNSTATS Efficient Queries
The query shown above would typically process lots of rows, but this need not be the case. The next example lists all tables with a department column and at least one row for the 'A00'
370
DB2 SQL Functions
DB2 V9.7 Cookbook ©
department. Only a single matching row is fetched from each table, so as long as there is a suitable index on the department column, the query should fly: SELECT
CHAR(tab.tabname,15) AS tabname ,CHAR(col.colname,10) AS colname ,CHAR(COALESCE(return_VARCHAR( ' SELECT ''Y''' || ' FROM ' || tab.tabschema || '.' || tab.tabname || ' WHERE ' || col.colname || ' = ''A00''' || ' FETCH FIRST 1 ROWS ONLY ' || ' OPTIMIZE FOR 1 ROW ' || ' WITH UR' ),'N'),1) AS has_dept FROM syscat.columns col ,syscat.tables tab WHERE col.tabschema = USER AND col.colname IN ('DEPTNO','WORKDEPT') AND col.tabschema = tab.tabschema AND col.tabname = tab.tabname AND tab.type = 'T' FOR FETCH ONLY WITH UR; ANSWER ============================= TABNAME COLNAME HAS_DEPT ---------- --------- -------DEPARTMENT DEPTNO Y EMPLOYEE WORKDEPT Y PROJECT DEPTNO N
Figure 964, List tables with a row for A00 department The next query is the same as the previous, except that it only searches those matching tables that have a suitable index on the department field: SELECT
CHAR(tab.tabname,15) AS tabname ,CHAR(col.colname,10) AS colname ,CHAR(COALESCE(return_VARCHAR( ' SELECT ''Y''' || ' FROM ' || tab.tabschema || '.' || tab.tabname || ' WHERE ' || col.colname || ' = ''A00''' || ' FETCH FIRST 1 ROWS ONLY ' || ' OPTIMIZE FOR 1 ROW ' || ' WITH UR' ),'N'),1) AS has_dept FROM syscat.columns col ,syscat.tables tab WHERE col.tabschema = USER AND col.colname IN ('DEPTNO','WORKDEPT') AND col.tabschema = tab.tabschema AND col.tabname = tab.tabname AND tab.type = 'T' AND col.colname IN (SELECT SUBSTR(idx.colnames,2,LENGTH(col.colname)) FROM syscat.indexes idx WHERE tab.tabschema = idx.tabschema AND tab.tabname = idx.tabname) FOR FETCH ONLY WITH UR; ANSWER =========================== TABNAME COLNAME HAS_DEPT ---------- ------- -------DEPARTMENT DEPTNO Y
Figure 965, List suitably-indexed tables with a row for A00 department Using logic very similar to the above, one can efficiently ask questions like: "list all tables in the application that have references to customer-number 1234 in indexed fields". Even if the
Running SQL Within SQL
371
Graeme Birchall ©
query has to process hundreds of tables, each with billions of rows, it should return an answer in less than ten seconds. In the above examples we knew what columns we wanted to process, but not the tables. But for some questions we don't even need to know the column name. For example, we could scan all indexed DATE columns in an application - looking for date values that are more than five years old. Once again, such a query should run in seconds.
Java Functions We can do the same as the above by calling a user-defined-function that invokes a java program, but we can also do much more. This section will cover the basics. Scalar Functions
The following code creates a user-defined scalar function that sends a query to a java program, and gets back the first row/column fetched when the query is run: CREATE FUNCTION get_Integer(VARCHAR(4000)) RETURNS INTEGER LANGUAGE JAVA EXTERNAL NAME 'Graeme2!get_Integer' PARAMETER STYLE DB2GENERAL NO EXTERNAL ACTION NOT DETERMINISTIC READS SQL DATA FENCED;
Figure 966, CREATE FUNCTION code Below is the corresponding java code: import import import import import
java.lang.*; COM.ibm.db2.app.*; java.sql.*; java.math.*; java.io.*;
public class Graeme2 extends UDF { public void get_Integer(String inStmt, int outValue) throws Exception { try { Connection con = DriverManager.getConnection ("jdbc:default:connection"); PreparedStatement stmt = con.prepareStatement(inStmt); ResultSet rs = stmt.executeQuery(); if (rs.next() == true && rs.getString(1) != null) { set(2, rs.getInt(1)); } rs.close(); stmt.close(); con.close(); } catch (SQLException sqle) { setSQLstate("38999"); setSQLmessage("SQLCODE = " + sqle.getSQLState()); return; } } }
Figure 967, CREATE FUNCTION java code
372
Java Functions
DB2 V9.7 Cookbook ©
Java Logic
Establish connection.
Prepare the SQL statement (i.e. input string).
Execute the SQL statement (i.e. open cursor).
If a row is found, and the value (of the first column) is not null, return value.
Close cursor.
Return.
Usage Example
SELECT
ANSWER ========================== DEPT EMPNO SALARY #ROWS ---- ------ -------- ----E11 000290 35340.00 7 E21 200330 35370.00 6 E21 200340 31840.00 6
workdept AS dept ,empno ,salary ,get_Integer( ' SELECT count(*)' || ' FROM employee' || ' where workdept = ''' || workdept || ''' ') AS #rows FROM employee WHERE salary < 35500 ORDER BY workdept ,empno;
Figure 968, Java function usage example I have posted suitable examples (i.e. java code, plus related CREATE FUNCTION code) for the following data types on my personal website:
BIGINT
INTEGER
SMALLINT
DOUBLE
DECIMAL(31,6)
VARCHAR(254)
Tabular Functions
So far, all we have done in this chapter is get single values from tables. Now we will retrieve sets of rows from tables. To do this we need to define a tabular function: CREATE FUNCTION tab_Varchar (VARCHAR(4000)) RETURNS TABLE (row_number INTEGER ,row_value VARCHAR(254)) LANGUAGE JAVA EXTERNAL NAME 'Graeme2!tab_Varchar' PARAMETER STYLE DB2GENERAL NO EXTERNAL ACTION NOT DETERMINISTIC DISALLOW PARALLEL READS SQL DATA FINAL CALL FENCED;
Figure 969, CREATE FUNCTION code
Running SQL Within SQL
373
Graeme Birchall ©
Below is the corresponding java code. Observe that two columns are returned – a row-number and the value fetched: import import import import import
java.lang.*; COM.ibm.db2.app.*; java.sql.*; java.math.*; java.io.*;
public class Graeme2 extends UDF { Connection con; Statement stmt; ResultSet rs; int rowNum; public void tab_Varchar(String inStmt, int outNumber, String outValue) throws Exception { switch (getCallType()) { case SQLUDF_TF_FIRST: break; case SQLUDF_TF_OPEN: rowNum = 1; try { con = DriverManager.getConnection ("jdbc:default:connection"); stmt = con.createStatement(); rs = stmt.executeQuery(inStmt); } catch(SQLException sqle) { setSQLstate("38999"); setSQLmessage("SQLCODE = " + sqle.getSQLState()); return; } break; case SQLUDF_TF_FETCH: if (rs.next() == true) { set(2, rowNum); if (rs.getString(1) != null) { set(3, rs.getString(1)); } rowNum++; } else { setSQLstate ("02000"); } break; case SQLUDF_TF_CLOSE: rs.close(); stmt.close(); con.close(); break; case SQLUDF_TF_FINAL: break; } } }
Figure 970, CREATE FUNCTION java code Java Logic
Java programs that send data to DB2 table functions use a particular type of CASE logic to return the output data. In particular, a row is returned at the end of every FETCH process. OPEN:
374
Establish connection.
Java Functions
DB2 V9.7 Cookbook ©
Prepare the SQL statement (i.e. input string).
Execute the SQL statement (i.e. open cursor).
Set row-number variable to one.
FETCH:
If row exists, set row-number output value.
If value fetched is not null, set output value.
Increment row-number variable.
CLOSE:
Close cursor.
Return.
Usage Example
The following query lists all EMPNO values that exist in more than four tables: WITH make_queries AS (SELECT tab.tabschema ,tab.tabname ,' SELECT EMPNO ' || ' FROM ' || tab.tabschema || '.' || tab.tabname AS sql_text FROM syscat.tables tab ,syscat.columns col WHERE tab.tabschema = USER AND tab.type = 'T' AND col.tabschema = tab.tabschema AND col.tabname = tab.tabname AND col.colname = 'EMPNO' AND col.typename = 'CHARACTER' AND col.length = 6 ), run_queries AS (SELECT qqq.* ,ttt.* FROM make_queries qqq ,TABLE(tab_Varchar(sql_text)) AS ttt ) SELECT CHAR(row_value,10) AS empno ,COUNT(*) AS #rows ,COUNT(DISTINCT tabschema || tabname) AS #tabs ,CHAR(MIN(tabname),18) AS min_tab ,CHAR(MAX(tabname),18) AS max_tab FROM run_queries GROUP BY row_value HAVING COUNT(DISTINCT tabschema || tabname) > 3 ORDER BY row_value FOR FETCH ONLY WITH UR; ANSWER ====================================== EMPNO #ROWS#TABS MIN_TAB MAX_TAB ------ ---- ----- --------- ---------000130 7 4 EMP_PHOTO EMPPROJACT 000140 10 4 EMP_PHOTO EMPPROJACT 000150 7 4 EMP_PHOTO EMPPROJACT 000190 7 4 EMP_PHOTO EMPPROJACT
Figure 971, Use Tabular Function
Running SQL Within SQL
375
Graeme Birchall ©
Transpose Function
Below is some pseudo-code for a really cool query: SELECT FROM WHERE
all columns unknown tables any unknown columns = '%ABC%'
Figure 972, Cool query pseudo-code In the above query we want to retrieve an unknown number of unknown types of columns (i.e. all columns in each matching row) from an unknown set of tables where any unknown column in the row equals 'ABC'. Needless to say, the various (unknown) tables will have differing types and numbers of columns. The above query is remarkably easy to write in SQL (see page: 379) and reasonably efficient to run, if we invoke a cute little java program that transposes columns into rows. The act of transposition means that each row/column instance retrieved becomes a separate row. So the following result: SELECT FROM WHERE
* empprojact empno = '000150'; ANSWER ================================================= EMPNO PROJNO ACTNO EMPTIME EMSTDATE EMENDATE ------ ------ ----- ------- ---------- ---------000150 MA2112 60 1.00 01/01/2002 07/15/2002 000150 MA2112 180 1.00 07/15/2002 02/01/2003
Figure 973, Select rows Becomes this result: SELECT
SMALLINT(row_number) AS row# ,col_num AS col# ,CHAR(col_name,13) AS col_name ,CHAR(col_type,10) AS col_type ,col_length AS col_len ,SMALLINT(LENGTH(col_value)) AS val_len ,SUBSTR(col_value,1,20) AS col_value FROM TABLE(tab_Transpose( ' SELECT *' || ' FROM empprojact' || ' WHERE empno = ''000150''' )) AS ttt ORDER BY 1,2; ANSWER ====================================================== ROW# COL# COL_NAME COL_TYPE COL_LEN VAL_LEN COL_VALUE ---- ---- -------- -------- ------- ------- ---------1 1 EMPNO CHAR 6 6 000150 1 2 PROJNO CHAR 6 6 MA2112 1 3 ACTNO SMALLINT 6 2 60 1 4 EMPTIME DECIMAL 7 4 1.00 1 5 EMSTDATE DATE 10 10 2002-01-01 1 6 EMENDATE DATE 10 10 2002-07-15 2 1 EMPNO CHAR 6 6 000150 2 2 PROJNO CHAR 6 6 MA2112 2 3 ACTNO SMALLINT 6 3 180 2 4 EMPTIME DECIMAL 7 4 1.00 2 5 EMSTDATE DATE 10 10 2002-07-15 2 6 EMENDATE DATE 10 10 2003-02-01
Figure 974, Select rows – then transpose
376
Java Functions
DB2 V9.7 Cookbook ©
The user-defined transpose function invoked above accepts a query as input. It executes the query then returns the query result as one row per row/column instance found. The function output table has the following columns:
ROW_NUMBER: The number of the row fetched.
NUM_COLS: The number of columns fetched per row.
COL_NUM: The column-number for the current row. This value, in combination with the prior row-number value, identifies a unique output row.
COL_NAME: The name of the data column - as given in the query. If there is no name, the value is the column number.
COL_TYPE: The DB2 column-type for the value.
COL_LENGTH: The DB2 column-length (note: not data item length) for the value.
COL_VALUE: The row/column instance value itself. If the data column is too long, or of an unsupported type (e.g. CLOB, DBCLOB, or XML), null is returned.
The transpose function always returns the same set of columns, regardless of which table is being accessed. So we can use it to write a query where we don't know which tables we want to select from. In the next example, we select all columns from all rows in all tables where the EMPNO column has a certain value: WITH make_queries AS (SELECT tab.tabschema ,tab.tabname ,' SELECT *' || ' FROM ' || tab.tabname || ' WHERE empno = ''000150''' AS sql_text FROM syscat.tables tab ,syscat.columns col WHERE tab.tabschema = USER AND tab.type = 'T' AND col.tabschema = tab.tabschema AND col.tabname = tab.tabname AND col.colname = 'EMPNO' AND col.typename = 'CHARACTER' AND col.length = 6 ), run_queries AS (SELECT qqq.* ,ttt.* FROM make_queries qqq ,TABLE(tab_Transpose(sql_text)) AS ttt ) SELECT SUBSTR(tabname,1,11) AS tab_name ,SMALLINT(row_number) AS row# ,col_num AS col# ,CHAR(col_name,13) AS col_name ,CHAR(col_type,10) AS col_type ,col_length AS col_len ,SMALLINT(LENGTH(col_value)) AS val_len ,SUBSTR(col_value,1,20) AS col_value FROM run_queries ORDER BY 1,2,3;
Figure 975, Select rows in any table – then transpose When we run the above, we get the following answer:
Running SQL Within SQL
377
Graeme Birchall ©
TAB_NAME ROW# COL# COL_NAME ---------- ---- ---- ------------EMP_PHOTO 1 1 EMPNO EMP_PHOTO 1 2 PHOTO_FORMAT EMP_PHOTO 1 3 PICTURE EMP_PHOTO 1 4 EMP_ROWID EMP_PHOTO 2 1 EMPNO EMP_PHOTO 2 2 PHOTO_FORMAT EMP_PHOTO 2 3 PICTURE EMP_PHOTO 2 4 EMP_ROWID EMP_RESUME 1 1 EMPNO EMP_RESUME 1 2 RESUME_FORMAT EMP_RESUME 1 3 RESUME EMP_RESUME 1 4 EMP_ROWID EMP_RESUME 2 1 EMPNO EMP_RESUME 2 2 RESUME_FORMAT EMP_RESUME 2 3 RESUME EMP_RESUME 2 4 EMP_ROWID EMPLOYEE 1 1 EMPNO EMPLOYEE 1 2 FIRSTNME EMPLOYEE 1 3 MIDINIT EMPLOYEE 1 4 LASTNAME EMPLOYEE 1 5 WORKDEPT EMPLOYEE 1 6 PHONENO EMPLOYEE 1 7 HIREDATE EMPLOYEE 1 8 JOB EMPLOYEE 1 9 EDLEVEL EMPLOYEE 1 10 SEX EMPLOYEE 1 11 BIRTHDATE EMPLOYEE 1 12 SALARY EMPLOYEE 1 13 BONUS EMPLOYEE 1 14 COMM EMPPROJACT 1 1 EMPNO EMPPROJACT 1 2 PROJNO EMPPROJACT 1 3 ACTNO EMPPROJACT 1 4 EMPTIME EMPPROJACT 1 5 EMSTDATE EMPPROJACT 1 6 EMENDATE EMPPROJACT 2 1 EMPNO EMPPROJACT 2 2 PROJNO EMPPROJACT 2 3 ACTNO EMPPROJACT 2 4 EMPTIME EMPPROJACT 2 5 EMSTDATE EMPPROJACT 2 6 EMENDATE
COL_TYPE COL_LEN VAL_LEN COL_VALUE -------- ------- ------- --------CHAR 6 6 000150 VARCHAR 10 6 bitmap BLOB 204800 - CHAR 40 40 CHAR 6 6 000150 VARCHAR 10 3 gif BLOB 204800 - CHAR 40 40 CHAR 6 6 000150 VARCHAR 10 5 ascii CLOB 5120 - CHAR 40 40 CHAR 6 6 000150 VARCHAR 10 4 html CLOB 5120 - CHAR 40 40 CHAR 6 6 000150 VARCHAR 12 5 BRUCE CHAR 1 1 VARCHAR 15 7 ADAMSON CHAR 3 3 D11 CHAR 4 4 4510 DATE 10 10 2002-02-12 CHAR 8 8 DESIGNER SMALLINT 6 2 16 CHAR 1 1 M DATE 10 10 1977-05-17 DECIMAL 11 8 55280.00 DECIMAL 11 6 500.00 DECIMAL 11 7 2022.00 CHAR 6 6 000150 CHAR 6 6 MA2112 SMALLINT 6 2 60 DECIMAL 7 4 1.00 DATE 10 10 2002-01-01 DATE 10 10 2002-07-15 CHAR 6 6 000150 CHAR 6 6 MA2112 SMALLINT 6 3 180 DECIMAL 7 4 1.00 DATE 10 10 2002-07-15 DATE 10 10 2003-02-01
Figure 976, Select rows in any table – answer We are obviously on a roll, so now we will write the pseudo-query that we began this chapter with (see page: 376). We will fetch every row/column instance in all matching tables where any qualifying column in the row is a particular value. Query Logic
Define the search parameters.
Get the list of matching tables and columns to search.
Recursively work through the list of columns to search (for each table), building a search query with multiple EQUAL predicates – one per searchable column (see page: 381).
Run the generated queries (i.e. the final line of generated query for each table).
Select the output.
Now for the query:
378
Java Functions
DB2 V9.7 Cookbook ©
WITH search_values (search_type,search_length,search_value) AS (VALUES ('CHARACTER',6,'000150') ), list_columns AS (SELECT val.search_value ,tab.tabschema ,tab.tabname ,col.colname ,ROW_NUMBER() OVER(PARTITION BY val.search_value ,tab.tabschema ,tab.tabname ORDER BY col.colname ASC) AS col_a ,ROW_NUMBER() OVER(PARTITION BY val.search_value ,tab.tabschema ,tab.tabname ORDER BY col.colname DESC) AS col_d FROM search_values val ,syscat.tables tab ,syscat.columns col WHERE tab.tabschema = USER AND tab.type = 'T' AND tab.tabschema = col.tabschema AND tab.tabname = col.tabname AND col.typename = val.search_type AND col.length = val.search_length ), make_queries (search_value ,tabschema ,tabname ,colname ,col_a ,col_d ,sql_text) AS (SELECT tb1.* ,VARCHAR(' SELECT *' || ' FROM ' || tabname || ' WHERE ' || colname || ' = ''' || search_value || '''' ,4000) FROM list_columns tb1 WHERE col_a = 1 UNION ALL SELECT tb2.* ,mqy.sql_text || ' OR ' || tb2.colname || ' = ''' || tb2.search_value || '''' FROM list_columns tb2 ,make_queries mqy WHERE tb2.search_value = mqy.search_value AND tb2.tabschema = mqy.tabschema AND tb2.tabname = mqy.tabname AND tb2.col_a = mqy.col_a + 1 ), run_queries AS (SELECT qqq.* ,ttt.* FROM make_queries qqq ,TABLE(tab_Transpose_4K(sql_text)) AS ttt WHERE col_d = 1 )
Figure 977, Select rows in any table – then transpose (part 1 of 2)
Running SQL Within SQL
379
Graeme Birchall ©
SELECT
SUBSTR(tabname,1,11) ,SMALLINT(row_number) ,col_num ,CHAR(col_name,13) ,CHAR(col_type,10) ,col_length ,SMALLINT(LENGTH(col_value)) ,SUBSTR(col_value,1,20) FROM run_queries ORDER BY 1,2,3;
AS AS AS AS AS AS AS AS
tab_name row# col# col_name col_type col_len val_len col_value
Figure 978, Select rows in any table – then transpose (part 2 of 2) Below is the answer (with a few values truncated to fit): TAB_NAME ROW# COL# COL_NAME ---------- ---- ---- ------------EMP_PHOTO 1 1 EMPNO EMP_PHOTO 1 2 PHOTO_FORMAT EMP_PHOTO 1 3 PICTURE EMP_PHOTO 1 4 EMP_ROWID EMP_PHOTO 2 1 EMPNO EMP_PHOTO 2 2 PHOTO_FORMAT EMP_PHOTO 2 3 PICTURE EMP_PHOTO 2 4 EMP_ROWID EMP_RESUME 1 1 EMPNO EMP_RESUME 1 2 RESUME_FORMAT EMP_RESUME 1 3 RESUME EMP_RESUME 1 4 EMP_ROWID EMP_RESUME 2 1 EMPNO EMP_RESUME 2 2 RESUME_FORMAT EMP_RESUME 2 3 RESUME EMP_RESUME 2 4 EMP_ROWID EMPLOYEE 1 1 EMPNO EMPLOYEE 1 2 FIRSTNME EMPLOYEE 1 3 MIDINIT EMPLOYEE 1 4 LASTNAME EMPLOYEE 1 5 WORKDEPT EMPLOYEE 1 6 PHONENO EMPLOYEE 1 7 HIREDATE EMPLOYEE 1 8 JOB EMPLOYEE 1 9 EDLEVEL EMPLOYEE 1 10 SEX EMPLOYEE 1 11 BIRTHDATE EMPLOYEE 1 12 SALARY EMPLOYEE 1 13 BONUS EMPLOYEE 1 14 COMM EMPPROJACT 1 1 EMPNO EMPPROJACT 1 2 PROJNO EMPPROJACT 1 3 ACTNO EMPPROJACT 1 4 EMPTIME EMPPROJACT 1 5 EMSTDATE EMPPROJACT 1 6 EMENDATE EMPPROJACT 2 1 EMPNO EMPPROJACT 2 2 PROJNO EMPPROJACT 2 3 ACTNO EMPPROJACT 2 4 EMPTIME EMPPROJACT 2 5 EMSTDATE EMPPROJACT 2 6 EMENDATE PROJECT 1 1 PROJNO PROJECT 1 2 PROJNAME PROJECT 1 3 DEPTNO PROJECT 1 4 RESPEMP PROJECT 1 5 PRSTAFF PROJECT 1 6 PRSTDATE PROJECT 1 7 PRENDATE PROJECT 1 8 MAJPROJ
COL_TYPE COL_LEN VAL_LEN COL_VALUE -------- ------- ------- --------CHAR 6 6 000150 VARCHAR 10 6 bitmap BLOB 204800 - CHAR 40 40 CHAR 6 6 000150 VARCHAR 10 3 gif BLOB 204800 - CHAR 40 40 CHAR 6 6 000150 VARCHAR 10 5 ascii CLOB 5120 - CHAR 40 40 CHAR 6 6 000150 VARCHAR 10 4 html CLOB 5120 - CHAR 40 40 CHAR 6 6 000150 VARCHAR 12 5 BRUCE CHAR 1 1 VARCHAR 15 7 ADAMSON CHAR 3 3 D11 CHAR 4 4 4510 DATE 10 10 2002-02-12 CHAR 8 8 DESIGNER SMALLINT 6 2 16 CHAR 1 1 M DATE 10 10 1977-05-17 DECIMAL 11 8 55280.00 DECIMAL 11 6 500.00 DECIMAL 11 7 2022.00 CHAR 6 6 000150 CHAR 6 6 MA2112 SMALLINT 6 2 60 DECIMAL 7 4 1.00 DATE 10 10 2002-01-01 DATE 10 10 2002-07-15 CHAR 6 6 000150 CHAR 6 6 MA2112 SMALLINT 6 3 180 DECIMAL 7 4 1.00 DATE 10 10 2002-07-15 DATE 10 10 2003-02-01 CHAR 6 6 MA2112 VARCHAR 24 16 W L ROBOT CHAR 3 3 D11 CHAR 6 6 000150 DECIMAL 7 4 3.00 DATE 10 10 2002-01-01 DATE 10 10 1982-12-01 CHAR 6 6 MA2110
Figure 979, Select rows in any table – answer
380
Java Functions
DB2 V9.7 Cookbook ©
Below are the queries that were generated and run to get the above answer: SELECT * SELECT * SELECT * SELECT * SELECT * SELECT * SELECT * SELECT * SELECT * RESPEMP
FROM ACT WHERE ACTKWD = '000150' FROM DEPARTMENT WHERE MGRNO = '000150' FROM EMP_PHOTO WHERE EMPNO = '000150' FROM EMP_RESUME WHERE EMPNO = '000150' FROM EMPLOYEE WHERE EMPNO = '000150' FROM EXPLAIN_OPERATOR WHERE OPERATOR_TYPE = '000150' FROM PROJACT WHERE PROJNO = '000150' FROM EMPPROJACT WHERE EMPNO = '000150' OR PROJNO = '000150' FROM PROJECT WHERE MAJPROJ = '000150' OR PROJNO = '000150' OR = '000150'
Figure 980, Queries generated above Function Definition
The DB2 user-defined tabular function that does the transposing is defined thus: CREATE FUNCTION tab_Transpose (VARCHAR(4000)) RETURNS TABLE (row_number INTEGER ,num_cols SMALLINT ,col_num SMALLINT ,col_name VARCHAR(128) ,col_type VARCHAR(128) ,col_length INTEGER ,col_value VARCHAR(254)) LANGUAGE JAVA EXTERNAL NAME 'Graeme2!tab_Transpose' PARAMETER STYLE DB2GENERAL NO EXTERNAL ACTION NOT DETERMINISTIC DISALLOW PARALLEL READS SQL DATA FINAL CALL FENCED;
Figure 981, Create transpose function Java Code
import import import import import
java.lang.*; COM.ibm.db2.app.*; java.sql.*; java.math.*; java.io.*;
public class Graeme2 extends UDF { Connection con; Statement stmt; ResultSet rs; ResultSetMetaData rsmtadta; int rowNum; int i; int outLength; short colNum; int colCount; String[] colName = new String[1100]; String[] colType = new String[1100]; int[] colSize = new int[1100]; public void writeRow() throws Exception { set(2, rowNum); set(3, (short) colCount); set(4, colNum); set(5, colName[colNum]); set(6, colType[colNum]);
Figure 982, CREATE FUNCTION java code (part 1 of 3)
Running SQL Within SQL
381
Graeme Birchall ©
set(7, colSize[colNum]); if (colType[colNum].equals("XML") colType[colNum].equals("BLOB") colType[colNum].equals("CLOB") colType[colNum].equals("DBLOB") colType[colNum].equals("GRAPHIC") colType[colNum].equals("VARGRAPHIC") colSize[colNum] > outLength) { // DON'T DISPLAY THIS VALUE return; } else if (rs.getString(colNum) != null) { // DISPLAY THIS COLUMN VALUE set(8, rs.getString(colNum)); }
|| || || || || ||
} public void tab_Transpose(String inStmt ,int rowNumber ,short numColumns ,short outColNumber ,String outColName ,String outColtype ,int outColSize ,String outColValue) throws Exception { switch (getCallType()) { case SQLUDF_TF_FIRST: break; case SQLUDF_TF_OPEN: try { con = DriverManager.getConnection ("jdbc:default:connection"); stmt = con.createStatement(); rs = stmt.executeQuery(inStmt); // GET COLUMN NAMES rsmtadta = rs.getMetaData(); colCount = rsmtadta.getColumnCount(); for (i=1; i colCount) { colNum = 1; rowNum++; } }
Figure 983, CREATE FUNCTION java code (part 2 of 3)
382
Java Functions
DB2 V9.7 Cookbook ©
else if (colNum > 1 && colNum colCount) { colNum = 1; rowNum++; } } else { setSQLstate ("02000"); } break; case SQLUDF_TF_CLOSE: rs.close(); stmt.close(); con.close(); break; case SQLUDF_TF_FINAL: break; } }}
Figure 984, CREATE FUNCTION java code (part 3 of 3) Java Logic
OPEN (run once):
Establish connection.
Prepare the SQL statement (i.e. input string).
Execute the SQL statement (i.e. open cursor).
Get meta-data for each column returned by query.
Set row-number and column-number variables to one.
Set the maximum output length accepted to 254.
FETCH (run for each row/column instance):
If row exists and column-number is 1, fetch row.
For value is not null and of valid DB2 type, return row.
Increment row-number and column-number variables.
CLOSE (run once):
Close the cursor.
Return.
Update Real Data using Meta-Data DB2 does not allow one to do DML or DDL using a scalar function, but one can do something similar by calling a table function. Thus if the table function defined below is joined to in a query, the following happens:
User query joins to table function - sends DML or DDL statement to be executed.
Table function calls stored procedure - sends statement to be executed.
Running SQL Within SQL
383
Graeme Birchall ©
Stored procedure executes statement.
Stored procedure returns SQLCODE of statement to the table function.
Table function joins back to the user query a single-row table with two columns: The SQLCODE and the original input statement.
Now for the code: CREATE PROCEDURE execute_immediate (IN in_stmt VARCHAR(1000) ,OUT out_sqlcode INTEGER) LANGUAGE SQL MODIFIES SQL DATA BEGIN DECLARE sqlcode INTEGER; DECLARE EXIT HANDLER FOR sqlexception SET out_sqlcode = sqlcode; EXECUTE IMMEDIATE in_stmt; SET out_sqlcode = sqlcode; RETURN; END! CREATE FUNCTION execute_immediate (in_stmt VARCHAR(1000)) RETURNS TABLE (sqltext VARCHAR(1000) ,sqlcode INTEGER) LANGUAGE SQL MODIFIES SQL DATA BEGIN ATOMIC DECLARE out_sqlcode INTEGER; CALL execute_immediate(in_stmt, out_sqlcode); RETURN VALUES (in_stmt, out_sqlcode); END!
IMPORTANT ============ This example uses an "!" as the stmt delimiter.
Figure 985, Define function and stored-procedure WARNING: This code is extremely dangerous! Use with care. As we shall see, it is very easy for the above code to do some quite unexpected.
Usage Examples
The following query gets a list of materialized query tables for a given table-schema that need to be refreshed, and then refreshes the table: WITH temp1 AS (SELECT tabschema ,tabname FROM syscat.tables WHERE tabschema = 'FRED' AND type = 'S' AND status = 'C' AND tabname LIKE '%DEPT%' ) SELECT CHAR(tab.tabname,20) AS tabname ,stm.sqlcode AS sqlcode ,CHAR(stm.sqltext,100) AS sqltext FROM temp1 AS tab ,TABLE(execute_immediate( 'REFRESH TABLE ' || RTRIM(tab.tabschema) || '.' || tab.tabname ))AS stm ORDER BY tab.tabname WITH UR;
Figure 986, Refresh matching tables I had two matching tables that needed to be refreshed, so I got the following answer:
384
Update Real Data using Meta-Data
DB2 V9.7 Cookbook ©
TABNAME ----------STAFF_DEPT1 STAFF_DEPT2
SQLCODE ------0 0
SQLTEXT -----------------------------REFRESH TABLE FRED.STAFF_DEPT1 REFRESH TABLE FRED.STAFF_DEPT2
Figure 987, Refresh matching tables - answer Observe above that the set of matching tables to be refreshed was defined in a common-tableexpression, and then joined to the table function. It is very important that one always code thus, because in an ordinary join it is possible for the table function to be called before all of the predicates have been applied. To illustrate this concept, the next query is supposed to make a copy of two matching tables. The answer indicates that it did just this. But what it actually did was make copies of many more tables - because the table function was called before all of the predicates on SYSCAT.TABLES were applied. The other tables that were created don't show up in the query output, because they were filtered out later in the query processing: SELECT
CHAR(tab.tabname,20) AS tabname ,stm.sqlcode AS sqlcode ,CHAR(stm.sqltext,100) AS sqltext FROM syscat.tables AS tab ,TABLE(execute_immediate( ' CREATE TABLE ' || RTRIM(tab.tabschema) || '.' || tab.tabname || '_C1' || ' LIKE ' || RTRIM(tab.tabschema) || '.' || tab.tabname ))AS stm WHERE tab.tabschema = USER AND tab.tabname LIKE 'S%' ORDER BY tab.tabname FOR FETCH ONLY WITH UR; ANSWER ========================================================== TABNAME SQLCODE SQLTEXT ------- ------- -----------------------------------------SALES 0 CREATE TABLE FRED.SALES_C1 LIKE FRED.SALES STAFF 0 CREATE TABLE FRED.STAFF_C1 LIKE FRED.STAFF
Figure 988, Create copies of tables - wrong The above is bad enough, but I once managed to do much worse. In a variation of the above code, the query created a copy, of a copy, of a copy, etc. The table function kept finding the table just created, and making a copy of it - until the TABNAME reached the length limit. The correct way to create a copy of a set of tables is shown below. In this query, the list of tables to be copied is identified in a common table expression before the table function is called:
Running SQL Within SQL
385
Graeme Birchall ©
WITH temp1 AS (SELECT tabschema ,tabname FROM syscat.tables WHERE tabschema = USER AND tabname LIKE 'S%' ) SELECT CHAR(tab.tabname,20) AS tabname ,stm.sqlcode AS sqlcode ,CHAR(stm.sqltext,100) AS sqltext FROM temp1 tab ,TABLE(execute_immediate( ' CREATE TABLE ' || RTRIM(tab.tabschema) || '.' || tab.tabname || '_C1' || ' LIKE ' || RTRIM(tab.tabschema) || '.' || tab.tabname ))AS stm ORDER BY tab.tabname FOR FETCH ONLY WITH UR; ANSWER ========================================================== TABNAME SQLCODE SQLTEXT ------- ------- -----------------------------------------SALES 0 CREATE TABLE FRED.SALES_C1 LIKE FRED.SALES STAFF 0 CREATE TABLE FRED.STAFF_C1 LIKE FRED.STAFF
Figure 989, Create copies of tables - right The next example is similar to the previous, except that it creates a copy, and then populates the new table with the contents of the original table: WITH temp0 AS (SELECT FROM WHERE AND ), temp1 AS (SELECT FROM
), temp2 AS (SELECT FROM
RTRIM(tabschema) ,tabname ,tabname || '_C2' syscat.tables tabschema = tabname LIKE
AS schema AS old_tabname AS new_tabname USER 'S%'
tab.* ,stm.sqlcode AS sqlcode1 ,CHAR(stm.sqltext,200) AS sqltext1 temp0 AS tab ,TABLE(execute_immediate( ' CREATE TABLE ' || schema || '.' || new_tabname || ' LIKE ' || schema || '.' || old_tabname ))AS stm tab.* ,stm.sqlcode AS ,CHAR(stm.sqltext,200) AS temp1 AS tab ,TABLE(execute_immediate( ' INSERT INTO ' || ' SELECT * FROM ' || ))AS stm
sqlcode2 sqltext2 schema || '.' || new_tabname || schema || '.' || old_tabname
) SELECT
CHAR(old_tabname,20) AS tabname ,sqlcode1 ,sqlcode2 FROM temp2 ORDER BY old_tabname FOR FETCH ONLY WITH UR;
ANSWER ========================= TABNAME SQLCODE1 SQLCODE2 ------- -------- -------SALES 0 0 STAFF 0 0
Figure 990, Create copies of tables, then populate
386
Update Real Data using Meta-Data
DB2 V9.7 Cookbook ©
Query Processing Sequence
In order to explain the above, we need to understand in what sequence the various parts of a query are executed in order to avoid semantic ambiguity: FROM clause JOIN ON clause WHERE clause GROUP BY and aggregate HAVING clause SELECT list ORDER BY clause FETCH FIRST
Figure 991, Query Processing Sequence Observe above that the FROM clause is resolved before any WHERE predicates are applied. This is why the query in figure 988 did the wrong thing.
Running SQL Within SQL
387
Graeme Birchall ©
388
Update Real Data using Meta-Data
DB2 V9.7 Cookbook ©
Fun with SQL In this chapter will shall cover some of the fun things that one can and, perhaps, should not do, using DB2 SQL. Read on at your own risk.
Creating Sample Data If every application worked exactly as intended from the first, we would never have any need for test databases. Unfortunately, one often needs to builds test systems in order to both tune the application SQL, and to do capacity planning. In this section we shall illustrate how very large volumes of extremely complex test data can be created using relatively simple SQL statements. Good Sample Data is
Reproducible.
Easy to make.
Similar to Production:
Same data volumes (if needed).
Same data distribution characteristics.
Data Generation
Create the set of integers between zero and one hundred. In this statement we shall use recursive coding to expand a single value into many more. WITH temp1 (col1) AS (VALUES 0 UNION ALL SELECT col1 + 1 FROM temp1 WHERE col1 + 1 < 100 ) SELECT * FROM temp1;
ANSWER ====== COL1 ---0 1 2 3 etc
Figure 992, Use recursion to get list of 100 numbers Instead of coding a recursion join every time, we use the table function described on page 196 to create the required rows. Assuming that the function exists, one would write the following: SELECT FROM
* TABLE(NumList(100)) AS xxx;
Figure 993, Use user-defined-function to get list of 100 numbers Make Reproducible Random Data
So far, all we have done is create sets of fixed values. These are usually not suitable for testing purposes because they are too consistent. To mess things up a bit we need to use the RAND function, which generates random numbers in the range of zero to one inclusive. In the next example we will get a (reproducible) list of five random numeric values:
Fun with SQL
389
Graeme Birchall ©
WITH temp1 (s1, r1) AS (VALUES (0, RAND(1)) UNION ALL SELECT s1+1, RAND() FROM temp1 WHERE s1+1 < 5 ) SELECT SMALLINT(s1) AS seq# ,DECIMAL(r1,5,3) AS ran1 FROM temp1;
ANSWER ============ SEQ# RAN1 ---- ----0 0.001 1 0.563 2 0.193 3 0.808 4 0.585
Figure 994, Use RAND to create pseudo-random numbers The initial invocation of the RAND function above is seeded with the value 1. Subsequent invocations of the same function (in the recursive part of the statement) use the initial value to generate a reproducible set of pseudo-random numbers. Using the GENERATE_UNIQUE function
With a bit of data manipulation, the GENERATE_UNIQUE function can be used (instead of the RAND function) to make suitably random test data. The are advantages and disadvantages to using both functions:
The GENERATE_UNIQUE function makes data that is always unique. The RAND function only outputs one of 32,000 distinct values.
The RAND function can make reproducible random data, while the GENERATE_UNIQUE function can not.
See the description of the GENERATE_UNIQUE function (see page 147) for an example of how to use it to make random data. Make Random Data - Different Ranges
There are several ways to mess around with the output from the RAND function: We can use simple arithmetic to alter the range of numbers generated (e.g. convert from 0 to 10 to 0 to 10,000). We can alter the format (e.g. from FLOAT to DECIMAL). Lastly, we can make fewer, or more, distinct random values (e.g. from 32K distinct values down to just 10). All of this is done below: WITH temp1 (s1, r1) AS (VALUES (0, RAND(2)) UNION ALL SELECT s1+1, RAND() FROM temp1 WHERE s1+1 < 5 ) SELECT SMALLINT(s1) ,SMALLINT(r1*10000) ,DECIMAL(r1,6,4) ,SMALLINT(r1*10) FROM temp1;
AS AS AS AS
seq# ran2 ran1 ran3
ANSWER ======================== SEQ# RAN2 RAN1 RAN3 ---- ---- ------ ---0 13 0.0013 0 1 8916 0.8916 8 2 7384 0.7384 7 3 5430 0.5430 5 4 8998 0.8998 8
Figure 995, Make differing ranges of random numbers Make Random Data - Varying Distribution
In the real world, there is a tendency for certain data values to show up much more frequently than others. Likewise, separate fields in a table usually have independent semi-random data distribution patterns. In the next statement we create three independently random fields. The first has the usual 32K distinct values evenly distributed in the range of zero to one. The sec-
390
Creating Sample Data
DB2 V9.7 Cookbook ©
ond and third have random numbers that are skewed towards the low end of the range, and have many more distinct values: ANSWER ======================= S# RAN1 RAN2 RAN3 -- ------ ------ -----0 1251 365370 114753 1 350291 280730 88106 2 710501 149549 550422 3 147312 33311 2339 4 8911 556 73091
WITH temp1 (s1) AS (VALUES (0) UNION ALL SELECT s1 + 1 FROM temp1 WHERE s1 + 1 < 5 ) SELECT SMALLINT(s1) AS ,INTEGER((RAND(1)) * 1E6) AS ,INTEGER((RAND() * RAND()) * 1E6) AS ,INTEGER((RAND() * RAND()* RAND()) * 1E6) AS FROM temp1;
s# ran1 ran2 ran3
Figure 996, Create RAND data with different distributions Make Random Data - Different Flavours
The RAND function generates random numbers. To get random character data one has to convert the RAND output into a character. There are several ways to do this. The first method shown below uses the CHR function to convert a number in the range: 65 to 90 into the ASCII equivalent: "A" to "Z". The second method uses the CHAR function to translate a number into the character equivalent. WITH temp1 (s1, r1) AS (VALUES (0, RAND(2)) UNION ALL SELECT s1+1, RAND() FROM temp1 WHERE s1+1 < 5 ) SELECT SMALLINT(s1) ,SMALLINT(r1*26+65) ,CHR(SMALLINT(r1*26+65)) ,CHAR(SMALLINT(r1*26)+65) FROM temp1;
AS AS AS AS
seq# ran2 ran3 ran4
ANSWER =================== SEQ# RAN2 RAN3 RAN4 ---- ---- ---- ---0 65 A 65 1 88 X 88 2 84 T 84 3 79 O 79 4 88 X 88
Figure 997, Converting RAND output from number to character Make Test Table & Data
So far, all we have done in this chapter is use SQL to select sets of rows. Now we shall create a Production-like table for performance testing purposes. We will then insert 10,000 rows of suitably lifelike test data into the table. The DDL, with constraints and index definitions, follows. The important things to note are:
The EMP# and the SOCSEC# must both be unique.
The JOB_FTN, FST_NAME, and LST_NAME fields must all be non-blank.
The SOCSEC# must have a special format.
The DATE_BN must be greater than 1900.
Several other fields must be within certain numeric ranges.
Fun with SQL
391
Graeme Birchall ©
CREATE TABLE personnel (emp# INTEGER NOT NULL ,socsec# CHAR(11) NOT NULL ,job_ftn CHAR(4) NOT NULL ,dept SMALLINT NOT NULL ,salary DECIMAL(7,2) NOT NULL ,date_bn DATE NOT NULL WITH DEFAULT ,fst_name VARCHAR(20) ,lst_name VARCHAR(20) ,CONSTRAINT pex1 PRIMARY KEY (emp#) ,CONSTRAINT pe01 CHECK (emp# > 0) ,CONSTRAINT pe02 CHECK (LOCATE(' ',socsec#) = 0) ,CONSTRAINT pe03 CHECK (LOCATE('-',socsec#,1) = 4) ,CONSTRAINT pe04 CHECK (LOCATE('-',socsec#,5) = 7) ,CONSTRAINT pe05 CHECK (job_ftn '') ,CONSTRAINT pe06 CHECK (dept BETWEEN 1 AND 99) ,CONSTRAINT pe07 CHECK (salary BETWEEN 0 AND 99999) ,CONSTRAINT pe08 CHECK (fst_name '') ,CONSTRAINT pe09 CHECK (lst_name '') ,CONSTRAINT pe10 CHECK (date_bn >= '1900-01-01' )); CREATE UNIQUE INDEX PEX2 ON PERSONNEL (SOCSEC#); CREATE UNIQUE INDEX PEX3 ON PERSONNEL (DEPT, EMP#);
Figure 998, Production-like test table DDL Now we shall populate the table. The SQL shall be described in detail latter. For the moment, note the four RAND fields. These contain, independently generated, random numbers which are used to populate the other data fields. INSERT INTO personnel WITH temp1 (s1,r1,r2,r3,r4) AS (VALUES (0 ,RAND(2) ,RAND()+(RAND()/1E5) ,RAND()* RAND() ,RAND()* RAND()* RAND()) UNION ALL SELECT s1 + 1 ,RAND() ,RAND()+(RAND()/1E5) ,RAND()* RAND() ,RAND()* RAND()* RAND() FROM temp1 WHERE s1 < 10000) SELECT 100000 + s1 ,SUBSTR(DIGITS(INT(r2*988+10)),8) || '-' || SUBSTR(DIGITS(INT(r1*88+10)),9) || '-' || TRANSLATE(SUBSTR(DIGITS(s1),7),'9873450126','0123456789') ,CASE WHEN INT(r4*9) > 7 THEN 'MGR' WHEN INT(r4*9) > 5 THEN 'SUPR' WHEN INT(r4*9) > 3 THEN 'PGMR' WHEN INT(R4*9) > 1 THEN 'SEC' ELSE 'WKR' END ,INT(r3*98+1) ,DECIMAL(r4*99999,7,2) ,DATE('1930-01-01') + INT(50-(r4*50)) YEARS + INT(r4*11) MONTHS + INT(r4*27) DAYS ,CHR(INT(r1*26+65))|| CHR(INT(r2*26+97))|| CHR(INT(r3*26+97))|| CHR(INT(r4*26+97))|| CHR(INT(r3*10+97))|| CHR(INT(r3*11+97)) ,CHR(INT(r2*26+65))|| TRANSLATE(CHAR(INT(r2*1E7)),'aaeeiibmty','0123456789') FROM temp1;
Figure 999, Production-like test table INSERT
392
Creating Sample Data
DB2 V9.7 Cookbook ©
Some sample data follows: EMP# -----100000 100001 100002 100003 100004 100005 100006 100007 100008
SOCSEC# ----------484-10-9999 449-38-9998 979-90-9997 580-50-9993 264-87-9994 661-84-9995 554-53-9990 482-23-9991 536-41-9992
JOB_ DEPT SALARY DATE_BN F_NME ---- ---- --------- ---------- --------WKR 47 13.63 1979-01-01 Ammaef SEC 53 35758.87 1962-04-10 Ilojff WKR 1 8155.23 1975-01-03 Xzacaa WKR 31 16643.50 1971-02-05 Lpiedd WKR 21 962.87 1979-01-01 Wgfacc WKR 19 4648.38 1977-01-02 Wrebbc WKR 8 375.42 1979-01-01 Mobaaa SEC 36 23170.09 1968-03-07 Emjgdd WKR 6 10514.11 1974-02-03 Jnbcaa
L_NME --------Mimytmbi Liiiemea Zytaebma Pimmeeat Geimteei Rbiybeet Oiiaiaia Mimtmamb Nieebayt
Figure 1000, Production-like test table, Sample Output In order to illustrate some of the tricks that one can use when creating such data, each field above was calculated using a different schema:
The EMP# is a simple ascending number.
The SOCSEC# field presented three problems: It had to be unique, it had to be random with respect to the current employee number, and it is a character field with special layout constraints (see the DDL on page 392).
To make it random, the first five digits were defined using two of the temporary random number fields. To try and ensure that it was unique, the last four digits contain part of the employee number with some digit-flipping done to hide things. Also, the first random number used is the one with lots of unique values. The special formatting that this field required is addressed by making everything in pieces and then concatenating.
The JOB FUNCTION is determined using the fourth (highly skewed) random number. This ensures that we get many more workers than managers.
The DEPT is derived from another, somewhat skewed, random number with a range of values from one to ninety nine.
The SALARY is derived using the same, highly skewed, random number that was used for the job function calculation. This ensures that theses two fields have related values.
The BIRTH DATE is a random date value somewhere between 1930 and 1981.
The FIRST NAME is derived using seven independent invocation of the CHR function, each of which is going to give a somewhat different result.
The LAST NAME is (mostly) made by using the TRANSLATE function to convert a large random number into a corresponding character value. The output is skewed towards some of the vowels and the lower-range characters during the translation.
Time-Series Processing The following table holds data for a typical time-series application. Observe is that each row has both a beginning and ending date, and that there are three cases where there is a gap between the end-date of one row and the begin-date of the next (with the same key).
Fun with SQL
393
Graeme Birchall ©
CREATE TABLE time_series (KYY CHAR(03) NOT NULL ,bgn_dt DATE NOT NULL ,end_dt DATE NOT NULL ,CONSTRAINT tsc1 CHECK (kyy '') ,CONSTRAINT tsc2 CHECK (bgn_dt a.bgn_dt AND z.bgn_dt < b.bgn_dt) ORDER BY 1,2;
TIME_SERIES +-------------------------+ |KYY|BGN_DT |END_DT | |---|----------|----------| |AAA|1995-10-01|1995-10-04| |AAA|1995-10-06|1995-10-06| |AAA|1995-10-07|1995-10-07| |AAA|1995-10-15|1995-10-19| |BBB|1995-10-01|1995-10-01| |BBB|1995-10-03|1995-10-03| +-------------------------+
Figure 1004, Find gap in Time-Series, SQL KEYCOL -----AAA AAA BBB
BGN_DT ---------1995-10-01 1995-10-07 1995-10-01
END_DT ---------1995-10-04 1995-10-07 1995-10-01
BGN_DT ---------1995-10-06 1995-10-15 1995-10-03
END_DT ---------1995-10-06 1995-10-19 1995-10-03
DIFF ---2 8 2
Figure 1005, Find gap in Time-Series, Answer WARNING: If there are many rows per key value, the above SQL will be very inefficient. This is because the join (done first) does a form of Cartesian Product (by key value) making an internal result table that can be very large. The sub-query then cuts this temporary table down to size by removing results-rows that have other intermediate rows.
Instead of looking at those rows that encompass a gap in the data, we may want to look at the actual gap itself. To this end, the following SQL differs from the prior in that the SELECT list has been modified to get the start, end, and duration, of each gap.
Fun with SQL
395
Graeme Birchall ©
SELECT a.kyy AS kyy ,a.end_dt + 1 DAY AS bgn_gap ,b.bgn_dt - 1 DAY AS end_gap ,(DAYS(b.bgn_dt) DAYS(a.end_dt) - 1) AS sz FROM time_series a ,time_series b WHERE a.kyy = b.kyy AND a.end_dt < b.bgn_dt - 1 DAY AND NOT EXISTS (SELECT * FROM time_series z WHERE z.kyy = a.kyy AND z.kyy = b.kyy AND z.bgn_dt > a.bgn_dt AND z.bgn_dt < b.bgn_dt) ORDER BY 1,2;
TIME_SERIES +-------------------------+ |KYY|BGN_DT |END_DT | |---|----------|----------| |AAA|1995-10-01|1995-10-04| |AAA|1995-10-06|1995-10-06| |AAA|1995-10-07|1995-10-07| |AAA|1995-10-15|1995-10-19| |BBB|1995-10-01|1995-10-01| |BBB|1995-10-03|1995-10-03| +-------------------------+ ANSWER ============================ KYY BGN_GAP END_GAP SZ --- ---------- ---------- -AAA 1995-10-05 1995-10-05 1 AAA 1995-10-08 1995-10-14 7 BBB 1995-10-02 1995-10-02 1
Figure 1006, Find gap in Time-Series Show Each Day in Gap
Imagine that we wanted to see each individual day in a gap. The following statement does this by taking the result obtained above and passing it into a recursive SQL statement which then generates additional rows - one for each day in the gap after the first. WITH temp (kyy, gap_dt, gsize) AS (SELECT a.kyy ,a.end_dt + 1 DAY ,(DAYS(b.bgn_dt) DAYS(a.end_dt) - 1) FROM time_series a ,time_series b WHERE a.kyy = b.kyy AND a.end_dt < b.bgn_dt - 1 DAY AND NOT EXISTS (SELECT * FROM time_series z WHERE z.kyy = a.kyy AND z.kyy = b.kyy AND z.bgn_dt > a.bgn_dt AND z.bgn_dt < b.bgn_dt) UNION ALL SELECT kyy ,gap_dt + 1 DAY ,gsize - 1 FROM temp WHERE gsize > 1 ) SELECT * FROM temp ORDER BY 1,2;
TIME_SERIES +-------------------------+ |KYY|BGN_DT |END_DT | |---|----------|----------| |AAA|1995-10-01|1995-10-04| |AAA|1995-10-06|1995-10-06| |AAA|1995-10-07|1995-10-07| |AAA|1995-10-15|1995-10-19| |BBB|1995-10-01|1995-10-01| |BBB|1995-10-03|1995-10-03| +-------------------------+
ANSWER ======================= KEYCOL GAP_DT GSIZE ------ ---------- ----AAA 1995-10-05 1 AAA 1995-10-08 7 AAA 1995-10-09 6 AAA 1995-10-10 5 AAA 1995-10-11 4 AAA 1995-10-12 3 AAA 1995-10-13 2 AAA 1995-10-14 1 BBB 1995-10-02 1
Figure 1007, Show each day in Time-Series gap
Other Fun Things Randomly Sample Data
One can use the TABLESAMPLE schema to randomly sample rows for subsequent analysis.
396
Other Fun Things
DB2 V9.7 Cookbook ©
SELECT ... FROM
table name correrelation name
TABLESAMPLE
BERNOULLI
(percent)
SYSTEM
REPEATABLE
(num)
Figure 1008, TABLESAMPLE Syntax Notes
The table-name must refer to a real table. This can include a declared global temporary table, or a materialized query table. It cannot be a nested table expression.
The sampling is an addition to any predicates specified in the where clause. Under the covers, sampling occurs before any other query processing, such as applying predicates or doing a join.
The BERNOUL option checks each row individually.
The SYSTEM option lets DB2 find the most efficient way to sample the data. This may mean that all rows on each page that qualifies are included. For small tables, this method often results in an misleading percentage of rows selected.
The "percent" number must be equal to or less than 100, and greater than zero. It determines what percentage of the rows processed are returns.
The REPEATABLE option and number is used if one wants to get the same result every time the query is run (assuming no data changes). Without this option, each run will be both random and different.
Examples
Sample 5% of the rows in the staff table. Get the same result each time: SELECT * FROM staff TABLESAMPLE BERNOULLI(5) REPEATABLE(1234) ORDER BY id;
Figure 1009, Sample rows in STAFF table Sample 18% of the rows in the employee table and 25% of the rows in the employee-activity table, then join the two tables together. Because each table is sampled independently, the fraction of rows that join will be much less either sampling rate: SELECT FROM
* employee ee TABLESAMPLE BERNOULLI(18) ,emp_act ea TABLESAMPLE BERNOULLI(25) WHERE ee.empno = ea.empno ORDER BY ee.empno;
Figure 1010, Sample rows in two tables Sample a declared global temporary table, and also apply other predicates: DECLARE GLOBAL TEMPORARY TABLE session.nyc_staff LIKE staff; SELECT FROM WHERE AND ORDER BY
* session.nyc_staff TABLESAMPLE SYSTEM(34.55) id < 100 salary > 100 id;
Figure 1011, Sample Views used in Join Examples
Fun with SQL
397
Graeme Birchall ©
Convert Character to Numeric
The DOUBLE, DECIMAL, INTEGER, SMALLINT, and BIGINT functions call all be used to convert a character field into its numeric equivalent: WITH temp1 (c1) AS (VALUES '123 ',' 345 ',' 567') SELECT c1 ,DOUBLE(c1) AS dbl ,DECIMAL(c1,3) AS dec ,SMALLINT(c1) AS sml ,INTEGER(c1) AS int FROM temp1;
ANSWER (numbers shortened) ================================= C1 DBL DEC SML INT ----- ----------- ----- ---- ---123 +1.2300E+2 123. 123 123 345 +3.4500E+2 345. 345 345 567 +5.6700E+2 567. 567 567
Figure 1012, Convert Character to Numeric - SQL Not all numeric functions support all character representations of a number. The following table illustrates what's allowed and what's not: INPUT STRING ============ " 1234" " 12.4" " 12E4"
COMPATIBLE FUNCTIONS ========================================== DOUBLE, DECIMAL, INTEGER, SMALLINT, BIGINT DOUBLE, DECIMAL DOUBLE
Figure 1013, Acceptable conversion values Checking the Input
There are several ways to check that the input character string is a valid representation of a number - before doing the conversion. One simple solution involves converting all digits to blank, then removing the blanks. If the result is not a zero length string, then the input must have had a character other than a digit: WITH temp1 (c1) AS (VALUES ' 123','456 ',' 1 2',' 33%',NULL) SELECT c1 ,TRANSLATE(c1,' ','1234567890') AS c2 ,LENGTH(LTRIM(TRANSLATE(c1,' ','1234567890'))) AS c3 FROM temp1; ANSWER ============ C1 C2 C3 ---- ---- -123 0 456 0 1 2 0 33% % 1 -
Figure 1014, Checking for non-digits One can also write a user-defined scalar function to check for non-numeric input, which is what is done below. This function returns "Y" if the following is true:
The input is not null.
There are no non-numeric characters in the input.
The only blanks in the input are to the left of the digits.
There is only one "+" or "-" sign, and it is next to the left-side blanks, if any.
There is at least one digit in the input.
Now for the code:
398
Other Fun Things
DB2 V9.7 Cookbook ©
--#SET DELIMITER !
IMPORTANT ============ This example uses an "!" as the stmt delimiter.
CREATE FUNCTION isnumeric(instr VARCHAR(40)) RETURNS CHAR(1) BEGIN ATOMIC DECLARE is_number CHAR(1) DEFAULT 'Y'; DECLARE bgn_blank CHAR(1) DEFAULT 'Y'; DECLARE found_num CHAR(1) DEFAULT 'N'; DECLARE found_pos CHAR(1) DEFAULT 'N'; DECLARE found_neg CHAR(1) DEFAULT 'N'; DECLARE found_dot CHAR(1) DEFAULT 'N'; DECLARE ctr SMALLINT DEFAULT 1; IF instr IS NULL THEN RETURN NULL; END IF; wloop: WHILE ctr 10000 AND salary < 12200 )AS xxx ANSWER ORDER BY d_sal; ========================================= D_SAL D_CHR D_DGT I_SAL I_CHR I_DGT ------- -------- ------ ----- ----- -----494.10 -0494.10 049410 -494 -494 00494 -12.00 -0012.00 001200 -12 -12 00012 508.60 0508.60 050860 508 508 00508 1009.75 1009.75 100975 1009 1009 01009
Figure 1017, CHAR and DIGITS function usage The DIGITS function discards both the sign indicator and the decimal point, while the CHAR function output is (annoyingly) left-justified, and (for decimal data) has leading zeros. We can do better.
400
Other Fun Things
DB2 V9.7 Cookbook ©
Below are three user-defined functions that convert integer data from numeric to character, displaying the output right-justified, and with a sign indicator if negative. There is one function for each flavor of integer that is supported in DB2: CREATE FUNCTION char_right(inval SMALLINT) RETURNS CHAR(06) RETURN RIGHT(CHAR('',06) CONCAT RTRIM(CHAR(inval)),06); CREATE FUNCTION char_right(inval INTEGER) RETURNS CHAR(11) RETURN RIGHT(CHAR('',11) CONCAT RTRIM(CHAR(inval)),11); CREATE FUNCTION char_right(inval BIGINT) RETURNS CHAR(20) RETURN RIGHT(CHAR('',20) CONCAT RTRIM(CHAR(inval)),20);
Figure 1018, User-defined functions - convert integer to character Each of the above functions works the same way (working from right to left):
First, convert the input number to character using the CHAR function.
Next, use the RTRIM function to remove the right-most blanks.
Then, concatenate a set number of blanks to the left of the value. The number of blanks appended depends upon the input type, which is why there are three separate functions.
Finally, use the RIGHT function to get the right-most "n" characters, where "n" is the maximum number of digits (plus the sign indicator) supported by the input type.
The next example uses the first of the above functions: SELECT
i_sal ,char_right(i_sal) AS i_chr FROM (SELECT SMALLINT(salary - 11000) AS i_sal FROM staff WHERE salary > 10000 AND salary < 12200 )AS xxx ORDER BY i_sal;
ANSWER =========== I_SAL I_CHR ----- -----494 -494 -12 -12 508 508 1009 1009
Figure 1019, Convert SMALLINT to CHAR Decimal Input
Creating a similar function to handle decimal input is a little more tricky. One problem is that the CHAR function adds leading zeros to decimal data, which we don't want. A more serious problem is that there are many sizes and scales of decimal data, but we can only create one function (with a given name) for a particular input data type. Decimal values can range in both length and scale from 1 to 31 digits. This makes it impossible to define a single function to convert any possible decimal value to character with possibly running out of digits, or losing some precision. NOTE: The fact that one can only have one user-defined function, with a given name, per DB2 data type, presents a problem for all variable-length data types - notably character, varchar, and decimal. For character and varchar data, one can address the problem, to some extent, by using maximum length input and output fields. But decimal data has both a scale and a length, so there is no way to make an all-purpose decimal function.
Despite the above, below is a function that converts decimal data to character. It compromises by assuming an input of type decimal(22,2), which should handle most monetary values:
Fun with SQL
401
Graeme Birchall ©
CREATE FUNCTION char_right(inval DECIMAL(20,2)) RETURNS CHAR(22) RETURN RIGHT(CHAR('',19) REPLACE(SUBSTR(CHAR(inval*1),1,1),'0','') STRIP(CHAR(ABS(BIGINT(inval)))) '.' SUBSTR(DIGITS(inval),19,2),22);
CONCAT CONCAT CONCAT CONCAT
Figure 1020, User-defined function - convert decimal to character The function works as follows:
The input value is converted to CHAR and the first byte obtained. This will be a minus sign if the number is negative, else blank.
The non-fractional part of the number is converted to BIGINT then to CHAR.
A period (dot) is included.
The fractional digits (converted to character using the DIGITS function) are appended to the back of the output.
All of the above is concatenation together, along with some leading blanks. Finally, the 22 right-most characters are returned.
Below is the function in action: WITH temp1 (num, tst) AS (VALUES (1 ,DEC(0.01 ,20,2)) UNION ALL SELECT num + 1 ,tst * -3.21 FROM temp1 WHERE num < 8) select num ,tst ,char_right(tst) AS tchar FROM temp1;
ANSWER ================= NUM TST TCHAR --- ------ -----1 0.01 0.01 2 -0.03 -0.03 3 0.09 0.09 4 -0.28 -0.28 5 0.89 0.89 6 -2.85 -2.85 7 9.14 9.14 8 -29.33 -29.33
Figure 1021, Convert DECIMAL to CHAR Floating point data can be processed using the above function, as long as it is first converted to decimal using the standard DECIMAL function. Adding Commas
The next function converts decimal input to character, with embedded comas. It first coverts the value to character - as per the above function. It then steps though the output string, three bytes at a time, from right to left, checking to see if the next-left character is a number. If it is, it insert a comma, else it adds a blank byte to the front of the string:
402
Other Fun Things
DB2 V9.7 Cookbook ©
CREATE FUNCTION comma_right(inval DECIMAL(20,2)) RETURNS CHAR(27) LANGUAGE SQL DETERMINISTIC NO EXTERNAL ACTION BEGIN ATOMIC DECLARE i INTEGER DEFAULT 17; DECLARE abs_inval BIGINT; DECLARE out_value CHAR(27); SET abs_inval = ABS(BIGINT(inval)); SET out_value = RIGHT(CHAR('',19) CONCAT RTRIM(CHAR(BIGINT(inval))),19) CONCAT '.' CONCAT SUBSTR(DIGITS(inval),19,2); WHILE i > 2 DO IF SUBSTR(out_value,i-1,1) BETWEEN '0' AND '9' THEN SET out_value = SUBSTR(out_value,1,i-1) CONCAT ',' CONCAT SUBSTR(out_value,i); ELSE SET out_value = ' ' CONCAT out_value; END IF; SET i = i - 3; END WHILE; RETURN out_value; END
Figure 1022, User-defined function - convert decimal to character - with commas Below is an example of the above function in use: WITH ANSWER temp1 (num) AS ==================================== (VALUES (DEC(+1,20,2)) INPUT OUTPUT ,(DEC(-1,20,2)) ----------------- -----------------UNION ALL -975460660753.97 -975,460,660,753.97 SELECT num * 987654.12 -987655.12 -987,655.12 FROM temp1 -2.00 -2.00 WHERE ABS(num) < 1E10), 0.00 0.00 temp2 (num) AS 987653.12 987,653.12 (SELECT num - 1 975460660751.97 975,460,660,751.97 FROM temp1) SELECT num AS input ,comma_right(num) AS output FROM temp2 ORDER BY num;
Figure 1023, Convert DECIMAL to CHAR with commas Convert Timestamp to Numeric
There is absolutely no sane reason why anyone would want to convert a date, time, or timestamp value directly to a number. The only correct way to manipulate such data is to use the provided date/time functions. But having said that, here is how one does it: WITH tab1(ts1) AS (VALUES CAST('1998-11-22-03.44.55.123456' AS TIMESTAMP)) SELECT
FROM
ts1 , HEX(ts1) , DEC(HEX(ts1),20) ,FLOAT(DEC(HEX(ts1),20)) ,REAL (DEC(HEX(ts1),20)) tab1;
=> => => => =>
1998-11-22-03.44.55.123456 19981122034455123456 19981122034455123456. 1.99811220344551e+019 1.998112e+019
Figure 1024, Convert Timestamp to number
Fun with SQL
403
Graeme Birchall ©
Selective Column Output
There is no way in static SQL to vary the number of columns returned by a select statement. In order to change the number of columns you have to write a new SQL statement and then rebind. But one can use CASE logic to control whether or not a column returns any data. Imagine that you are forced to use static SQL. Furthermore, imagine that you do not always want to retrieve the data from all columns, and that you also do not want to transmit data over the network that you do not need. For character columns, we can address this problem by retrieving the data only if it is wanted, and otherwise returning to a zero-length string. To illustrate, here is an ordinary SQL statement: SELECT
empno ,firstnme ,lastname ,job FROM employee WHERE empno < '000100' ORDER BY empno;
Figure 1025, Sample query with no column control Here is the same SQL statement with each character column being checked against a hostvariable. If the host-variable is 1, the data is returned, otherwise a zero-length string: SELECT
empno ,CASE :host-var-1 WHEN 1 THEN firstnme ELSE '' END AS firstnme ,CASE :host-var-2 WHEN 1 THEN lastname ELSE '' END AS lastname ,CASE :host-var-3 WHEN 1 THEN VARCHAR(job) ELSE '' END AS job FROM employee WHERE empno < '000100' ORDER BY empno;
Figure 1026, Sample query with column control Making Charts Using SQL
Imagine that one had a string of numeric values that one wants to display as a line-bar chart. With a little coding, this is easy to do in SQL: SELECT
id ,salary ,INT(salary / 1500) AS len ,REPEAT('*',INT(salary / 1500)) AS salary_chart FROM staff WHERE id > 120 ANSWER AND id < 190 =================================== ORDER BY id; ID SALARY LEN SALARY_CHART --- -------- --- --------------130 10505.90 7 ******* 140 21150.00 14 ************** 150 19456.50 12 ************ 160 22959.20 15 *************** 170 12258.50 8 ******** 180 12009.75 8 ********
Figure 1027, Make chart using SQL
404
Other Fun Things
DB2 V9.7 Cookbook ©
To create the above graph we first converted the column of interest to an integer field of a manageable length, and then used this value to repeat a single "*" character a set number of times. One problem with the above query is that we won't know how long the chart will be until we run the statement. This may cause problems if we guess wrongly and we are tight for space. The next query addresses this issue by creating a chart of known length. It does it by dividing the row value by the maximum value for the selected rows (all divided by 20). The result is used to repeat the "*" character "n" times:
SELECT
ANSWER ======================================= DEPT ID SALARY CHART ---- --- --------- -------------------10 160 82959.20 ****************** 10 210 90010.00 ******************** 10 240 79260.25 ***************** 10 260 81234.00 ****************** 15 110 42508.20 ********* 15 170 42258.50 *********
dept ,id ,salary ,VARCHAR(REPEAT('*' ,INT(salary / (MAX(salary) OVER() / 20))),20) AS chart FROM staff WHERE dept = 100 ORDER BY 1,2;
Figure 1028, Make chart of fixed length The above code can be enhanced to have two charts in the same column. To illustrate, the next query expresses the salary as a chart, but separately by department. This can be useful to do when the two departments have very different values and one wants to analyze the data in each department independently:
SELECT
ANSWER ======================================= DEPT ID SALARY CHART ---- --- --------- -------------------10 160 82959.20 ****************** 10 210 90010.00 ******************** 10 240 79260.25 ***************** 10 260 81234.00 ****************** 15 110 42508.20 ******************** 15 170 42258.50 *******************
dept ,id ,salary ,VARCHAR(REPEAT('*',INT(salary / (MAX(salary) OVER(PARTITION BY dept) / 20))),20) AS chart FROM staff WHERE dept = 100 ORDER BY 1,2;
Figure 1029, Make two fixed length charts in the same column Multiple Counts in One Pass
The STATS table that is defined on page 116 has a SEX field with just two values, 'F' (for female) and 'M' (for male). To get a count of the rows by sex we can write the following:
Fun with SQL
405
Graeme Birchall ©
SELECT
sex ,COUNT(*) AS num FROM stats GROUP BY sex ORDER BY sex;
ANSWER >>
SEX --F M
NUM --595 405
Figure 1030, Use GROUP BY to get counts Imagine now that we wanted to get a count of the different sexes on the same line of output. One, not very efficient, way to get this answer is shown below. It involves scanning the data table twice (once for males, and once for females) then joining the result. WITH f (f) AS (SELECT COUNT(*) FROM stats WHERE sex = 'F') ,m (m) AS (SELECT COUNT(*) FROM stats WHERE sex = 'M') SELECT f, m FROM f, m;
Figure 1031, Use Common Table Expression to get counts It would be more efficient if we answered the question with a single scan of the data table. This we can do using a CASE statement and a SUM function: SELECT FROM
SUM(CASE sex WHEN 'F' THEN 1 ELSE 0 END) AS female ,SUM(CASE sex WHEN 'M' THEN 1 ELSE 0 END) AS male stats;
Figure 1032, Use CASE and SUM to get counts We can now go one step further and also count something else as we pass down the data. In the following example we get the count of all the rows at the same time as we get the individual sex counts. SELECT FROM
COUNT(*) AS total ,SUM(CASE sex WHEN 'F' THEN 1 ELSE 0 END) AS female ,SUM(CASE sex WHEN 'M' THEN 1 ELSE 0 END) AS male stats;
Figure 1033, Use CASE and SUM to get counts Find Missing Rows in Series / Count all Values
One often has a sequence of values (e.g. invoice numbers) from which one needs both found and not-found rows. This cannot be done using a simple SELECT statement because some of rows being selected may not actually exist. For example, the following query lists the number of staff that have worked for the firm for "n" years, but it misses those years during which no staff joined: SELECT
years ,COUNT(*) AS #staff FROM staff WHERE UCASE(name) LIKE '%E%' AND years 20000) OR (cat.subcat = 'NAME LIKE ABC%' AND emp.firstnme LIKE 'ABC%') OR (cat.dept '' AND cat.dept = emp.workdept) )AS xxx GROUP BY xxx.cat ,xxx.subcat ORDER BY 1,2;
Figure 1038, Multiple counts in one pass, SQL In the above query, a temporary table is defined and then populated with all of the summation types. This table is then joined (using a left outer join) to the EMPLOYEE table. Any matches (i.e. where EMPNO is not null) are given a FOUND value of 1. The output of the join is then feed into a GROUP BY to get the required counts.
408
Other Fun Things
DB2 V9.7 Cookbook ©
CATEGORY -------1ST 2ND 3RD 4TH 5TH 5TH 5TH 5TH 5TH 5TH 5TH 5TH 5TH
SUBCATEGORY/DEPT ----------------------------ROWS IN TABLE SALARY > $20K NAME LIKE ABC% NUMBER MALES ADMINISTRATION SYSTEMS DEVELOPMENT CENTER INFORMATION CENTER MANUFACTURING SYSTEMS OPERATIONS PLANNING SOFTWARE SUPPORT SPIFFY COMPUTER SERVICE DIV. SUPPORT SERVICES
#ROWS ----32 25 0 19 6 0 3 9 5 1 4 3 1
Figure 1039, Multiple counts in one pass, Answer Normalize Denormalized Data
Imagine that one has a string of text that one wants to break up into individual words. As long as the word delimiter is fairly basic (e.g. a blank space), one can use recursive SQL to do this task. One recursively divides the text into two parts (working from left to right). The first part is the word found, and the second part is the remainder of the text: WITH temp1 (id, data) AS (VALUES (01,'SOME TEXT TO PARSE.') ,(02,'MORE SAMPLE TEXT.') ,(03,'ONE-WORD.') ,(04,'') ), temp2 (id, word#, word, data_left) AS (SELECT id ,SMALLINT(1) ,SUBSTR(data,1, CASE LOCATE(' ',data) WHEN 0 THEN LENGTH(data) ELSE LOCATE(' ',data) END) ,LTRIM(SUBSTR(data, CASE LOCATE(' ',data) WHEN 0 THEN LENGTH(data) + 1 ELSE LOCATE(' ',data) END)) FROM temp1 WHERE data '' UNION ALL SELECT id ,word# + 1 ,SUBSTR(data_left,1, CASE LOCATE(' ',data_left) WHEN 0 THEN LENGTH(data_left) ELSE LOCATE(' ',data_left) END) ,LTRIM(SUBSTR(data_left, CASE LOCATE(' ',data_left) WHEN 0 THEN LENGTH(data_left) + 1 ELSE LOCATE(' ',data_left) END)) FROM temp2 WHERE data_left '' ) SELECT * FROM temp2 ORDER BY 1,2;
Figure 1040, Break text into words - SQL
Fun with SQL
409
Graeme Birchall ©
The SUBSTR function is used above to extract both the next word in the string, and the remainder of the text. If there is a blank byte in the string, the SUBSTR stops (or begins, when getting the remainder) at it. If not, it goes to (or begins at) the end of the string. CASE logic is used to decide what to do. ID -1 1 1 1 2 2 2 3
WORD# ----1 2 3 4 1 2 3 1
WORD --------SOME TEXT TO PARSE. MORE SAMPLE TEXT. ONE-WORD.
DATA_LEFT -------------TEXT TO PARSE. TO PARSE. PARSE. SAMPLE TEXT. TEXT.
Figure 1041, Break text into words - Answer Denormalize Normalized Data
The SUM function can be used to accumulate numeric values. To accumulate character values (i.e. to string the individual values from multiple lines into a single long value) is a little harder, but it can also be done. The following example uses the XMLAGG column function to aggregate multiple values into one. The processing goes as follows:
The XMLTEXT scalar function converts each character value into XML. A space is put at the end of the each name, so there is a gap before the next.
The XMLAGG column function aggregates the individual XML values in name order.
The XMLSERIALIZE scalar function converts the aggregated XML value into a CLOB.
The SUBSTR scalar function converts the CLOB to a CHAR.
Now for the code: SELECT
dept ,SMALLINT(COUNT(*)) AS ,MAX(name) AS ,SUBSTR( XMLSERIALIZE( XMLAGG( XMLTEXT(name ORDER BY name) AS CLOB(1M)) ,1,50) AS FROM staff GROUP BY dept ORDER BY dept;
#w max_name
|| ' ') all_names
Figure 1042, Denormalize Normalized Data - SQL Here is the answer:
410
Other Fun Things
DB2 V9.7 Cookbook ©
DEPT W# MAX_NAME ALL_NAMES ---- -- --------- ------------------------------------------10 4 Molinare Daniels Jones Lu Molinare 15 4 Rothman Hanes Kermisch Ngan Rothman 20 4 Sneider James Pernal Sanders Sneider 38 5 Quigley Abrahams Marenghi Naughton O'Brien Quigley 42 4 Yamaguchi Koonitz Plotz Scoutten Yamaguchi 51 5 Williams Fraye Lundquist Smith Wheeler Williams 66 5 Wilson Burke Gonzales Graham Lea Wilson 84 4 Quill Davis Edwards Gafney Quill
Figure 1043, Denormalize Normalized Data - Answer The next example uses recursion to do exactly the same thing. It begins by getting the minimum name in each department. It then recursively gets the next to lowest name, then the next, and so on. As the query progresses, it maintains a count of names added, stores the current name in the temporary NAME field, and appends the same to the end of the ALL_NAMES field. Once all of the names have been processed, the final SELECT eliminates from the answer-set all rows, except the last for each department: WITH temp1 (dept,w#,name,all_names) AS (SELECT dept ,SMALLINT(1) ,MIN(name) ,VARCHAR(MIN(name),50) FROM staff a GROUP BY dept UNION ALL SELECT a.dept ,SMALLINT(b.w#+1) ,a.name ,b.all_names || ' ' || a.name FROM staff a ,temp1 b WHERE a.dept = b.dept AND a.name > b.name AND a.name = (SELECT MIN(c.name) FROM staff c WHERE c.dept = b.dept AND c.name > b.name) ) SELECT dept ,w# ,name AS max_name ,all_names FROM temp1 d WHERE w# = (SELECT MAX(w#) FROM temp1 e WHERE d.dept = e.dept) ORDER BY dept;
Figure 1044, Denormalize Normalized Data - SQL If there are no suitable indexes, the above query may be horribly inefficient. If this is the case, one can create a user-defined function to string together the names in a department:
Fun with SQL
411
Graeme Birchall ©
CREATE FUNCTION list_names(indept SMALLINT) RETURNS VARCHAR(50) BEGIN ATOMIC DECLARE outstr VARCHAR(50) DEFAULT ''; FOR list_names AS SELECT name FROM staff WHERE dept = indept ORDER BY name DO SET outstr = outstr || name || ' '; END FOR; SET outstr = rtrim(outstr); RETURN outstr; END!
IMPORTANT ============ This example uses an "!" as the stmt delimiter.
SELECT
dept AS DEPT ,SMALLINT(cnt) AS W# ,mxx AS MAX_NAME ,list_names(dept) AS ALL_NAMES FROM (SELECT dept ,COUNT(*) as cnt ,MAX(name) AS mxx FROM staff GROUP BY dept )as ddd ORDER BY dept!
Figure 1045, Creating a function to denormalize names Even the above might have unsatisfactory performance - if there is no index on department. If adding an index to the STAFF table is not an option, it might be faster to insert all of the rows into a declared temporary table, and then add an index to that. Transpose Numeric Data
In this section we will turn rows of numeric data into columns. This cannot be done directly in SQL because the language does not support queries where the output columns are unknown at query start. We will get around this limitation by sending the transposed output to a suitably long VARCHAR field. Imagine that we want to group the data in the STAFF sample table by DEPT and JOB to get the SUM salary for each instance, but not in the usual sense with one output row per DEPT and JOB value. Instead, we want to generate one row per DEPT, with a set of "columns" (in a VARCHAR field) that hold the SUM salary values for each JOB in the department. We will also put column titles on the first line of output. To make the following query simpler, three simple scalar functions will be used to convert data from one type to another:
Convert decimal data to character - similar to the one on page 401.
Convert smallint data to character - same as the one page 401.
Right justify and add leading blanks to character data.
Now for the functions:
412
Other Fun Things
DB2 V9.7 Cookbook ©
CREATE FUNCTION num_to_char(inval SMALLINT) RETURNS CHAR(06) RETURN RIGHT(CHAR('',06) CONCAT RTRIM(CHAR(inval)),06); CREATE FUNCTION num_to_char(inval DECIMAL(9,2)) RETURNS CHAR(10) RETURN RIGHT(CHAR('',7) CONCAT RTRIM(CHAR(BIGINT(inval))),7) CONCAT '.' CONCAT SUBSTR(DIGITS(inval),8,2); CREATE FUNCTION right_justify(inval CHAR(5)) RETURNS CHAR(10) RETURN RIGHT(CHAR('',10) || RTRIM(inval),10);
Figure 1046, Data Transformation Functions The query consists of lots of little steps that are best explained by describing each temporary table built:
DATA_INPUT: This table holds the set of matching rows in the STAFF table, grouped by DEPT and JOB as per a typical query (see page 415 for the contents). This is the only time that we touch the original STAFF table. All subsequent queries directly or indirectly reference this table.
JOBS_LIST: The list of distinct jobs in all matching rows. Each job is assigned two rownumbers, one ascending, and one descending.
DEPT_LIST: The list of distinct departments in all matching rows.
DEPT_JOB_LIST: The list of all matching department/job combinations. We need this table because not all departments have all jobs.
DATA_ALL_JOBS: The DEPT_JOB_LIST table joined to the original DATA_INPUT table using a left outer join, so we now have one row with a sum-salary value for every JOB and DEPT instance.
DATA_TRANSFORM: Recursively go through the DATA_ALL_JOBS table (for each department), adding the a character representation of the current sum-salary value to the back of a VARCHAR column.
DATA_LAST_ROW: For each department, get the row with the highest ascending JOB# value. This row has the concatenated string of sum-salary values.
At this point we are done, except that we don't have any column headings in our output. The rest of the query gets these.
JOBS_TRANSFORM: Recursively go through the list of distinct jobs, building a VARCHAR string of JOB names. The job names are right justified - to match the sumsalary values, and have the same output length.
JOBS_LAST_ROW: Get the one row with the lowest descending job number. This row has the complete string of concatenated job names.
DATA_AND_JOBS: Use a UNION ALL to vertically combine the JOBS_LAST_ROW and DATA_LAST_ROW tables. The result is a new table with both column titles and sum-salary values.
Finally, we select the list of column names and sum-salary values. The output is ordered so that the column names are on the first line fetched. Now for the query:
Fun with SQL
413
Graeme Birchall ©
WITH data_input AS (SELECT dept ,job ,SUM(salary) AS sum_sal FROM staff WHERE id < 200 AND name 'Sue' AND salary > 10000 GROUP BY dept ,job), jobs_list AS (SELECT job ,ROW_NUMBER() OVER(ORDER BY job ASC) AS job#A ,ROW_NUMBER() OVER(ORDER BY job DESC) AS job#D FROM data_input GROUP BY job), dept_list AS (SELECT dept FROM data_input GROUP BY dept), dept_jobs_list AS (SELECT dpt.dept ,job.job ,job.job#A ,job.job#D FROM jobs_list job FULL OUTER JOIN dept_list dpt ON 1 = 1), data_all_jobs AS (SELECT djb.dept ,djb.job ,djb.job#A ,djb.job#D ,COALESCE(dat.sum_sal,0) AS sum_sal FROM dept_jobs_list djb LEFT OUTER JOIN data_input dat ON djb.dept = dat.dept AND djb.job = dat.job), data_transform (dept, job#A, job#D, outvalue) AS (SELECT dept ,job#A ,job#D ,VARCHAR(num_to_char(sum_sal),250) FROM data_all_jobs WHERE job#A = 1 UNION ALL SELECT dat.dept ,dat.job#A ,dat.job#D ,trn.outvalue || ',' || num_to_char(dat.sum_sal) FROM data_transform trn ,data_all_jobs dat WHERE trn.dept = dat.dept AND trn.job#A = dat.job#A - 1), data_last_row AS (SELECT dept ,num_to_char(dept) AS dept_char ,outvalue FROM data_transform WHERE job#D = 1),
Figure 1047, Transform numeric data - part 1 of 2
414
Other Fun Things
DB2 V9.7 Cookbook ©
jobs_transform (job#A, job#D, outvalue) AS (SELECT job#A ,job#D ,VARCHAR(right_justify(job),250) FROM jobs_list WHERE job#A = 1 UNION ALL SELECT job.job#A ,job.job#D ,trn.outvalue || ',' || right_justify(job.job) FROM jobs_transform trn ,jobs_list job WHERE trn.job#A = job.job#A - 1), jobs_last_row AS (SELECT 0 AS dept ,' DEPT' AS dept_char ,outvalue FROM jobs_transform WHERE job#D = 1), data_and_jobs AS (SELECT dept ,dept_char ,outvalue FROM jobs_last_row UNION ALL SELECT dept ,dept_char ,outvalue FROM data_last_row) SELECT dept_char || ',' || outvalue AS output FROM data_and_jobs ORDER BY dept;
Figure 1048, Transform numeric data - part 2 of 2 For comparison, below are the contents of the first temporary table, and the final output: DATA_INPUT =================== DEPT JOB SUM_SAL ---- ----- -------10 Mgr 22959.20 15 Clerk 24766.70 15 Mgr 20659.80 15 Sales 16502.83 20 Clerk 27757.35 20 Mgr 18357.50 20 Sales 78171.25 38 Clerk 24964.50 38 Mgr 77506.75 38 Sales 34814.30 42 Clerk 10505.90 42 Mgr 18352.80 42 Sales 18001.75 51 Mgr 21150.00 51 Sales 19456.50
OUTPUT ===================================== DEPT, Clerk, Mgr, Sales 10, 0.00, 22959.20, 0.00 15, 24766.70, 20659.80, 16502.83 20, 27757.35, 18357.50, 78171.25 38, 24964.50, 77506.75, 34814.30 42, 10505.90, 18352.80, 18001.75 51, 0.00, 21150.00, 19456.50
Figure 1049, Contents of first temporary table and final output Reversing Field Contents
DB2 lacks a simple function for reversing the contents of a data field. Fortunately, we can create a function to do it ourselves.
Fun with SQL
415
Graeme Birchall ©
Input vs. Output
Before we do any data reversing, we have to define what the reversed output should look like relative to a given input value. For example, if we have a four-digit numeric field, the reverse of the number 123 could be 321, or it could be 3210. The latter value implies that the input has a leading zero. It also assumes that we really are working with a four digit field. Likewise, the reverse of the number 123.45 might be 54.321, or 543.21. Another interesting problem involves reversing negative numbers. If the value "-123" is a string, then the reverse is probably "321-". If it is a number, then the desired reverse is more likely to be "-321". Trailing blanks in character strings are a similar problem. Obviously, the reverse of "ABC" is "CBA", but what is the reverse of "ABC "? There is no general technical answer to any of these questions. The correct answer depends upon the business needs of the application. Below is a user defined function that can reverse the contents of a character field: --#SET DELIMITER !
IMPORTANT ============ This example uses an "!" as the stmt delimiter.
CREATE FUNCTION reverse(instr VARCHAR(50)) RETURNS VARCHAR(50) BEGIN ATOMIC DECLARE outstr VARCHAR(50) DEFAULT ''; DECLARE curbyte SMALLINT DEFAULT 0; SET curbyte = LENGTH(RTRIM(instr)); WHILE curbyte >= 1 DO SET outstr = outstr || SUBSTR(instr,curbyte,1); SET curbyte = curbyte - 1; END WHILE; RETURN outstr; END! ANSWER SELECT id AS ID ==================== ,name AS NAME1 ID NAME1 NAME2 ,reverse(name) AS NAME2 -- -------- ------FROM staff 10 Sanders srednaS WHERE id < 40 20 Pernal lanreP ORDER BY id! 30 Marenghi ihgneraM
Figure 1050, Reversing character field The same function can be used to reverse numeric values, as long as they are positive: SELECT
id AS ID ,salary AS SALARY1 ,DEC(reverse(CHAR(salary)),7,4) AS SALARY2 FROM staff ANSWER WHERE id < 40 =================== ORDER BY id; ID SALARY1 SALARY2 -- -------- ------10 18357.50 5.7538 20 78171.25 52.1718 30 77506.75 57.6057
Figure 1051, Reversing numeric field Simple CASE logic can be used to deal with negative values (i.e. to move the sign to the front of the string, before converting back to numeric), if they exist. Fibonacci Series
A Fibonacci Series is a series of numbers where each value is the sum of the previous two. Regardless of the two initial (seed) values, if run for long enough, the division of any two adjacent numbers will give the value 0.618 or inversely 1.618.
416
Other Fun Things
DB2 V9.7 Cookbook ©
The following user defined function generates a Fibonacci series using three input values:
First seed value.
Second seed value.
Number values to generate in series.
Observe that that the function code contains a check to stop series generation if there is not enough space in the output field for more numbers: --#SET DELIMITER !
IMPORTANT ============ This example uses an "!" as the stmt delimiter.
CREATE FUNCTION Fibonacci (inval1 INTEGER ,inval2 INTEGER ,loopno INTEGER) RETURNS VARCHAR(500) BEGIN ATOMIC DECLARE loopctr INTEGER DEFAULT 0; DECLARE tempval1 BIGINT; DECLARE tempval2 BIGINT; DECLARE tempval3 BIGINT; DECLARE outvalue VARCHAR(500); SET tempval1 = inval1; SET tempval2 = inval2; SET outvalue = RTRIM(LTRIM(CHAR(tempval1))) || ', ' || RTRIM(LTRIM(CHAR(tempval2))); calc: WHILE loopctr < loopno DO SET tempval3 = tempval1 + tempval2; SET tempval1 = tempval2; SET tempval2 = tempval3; SET outvalue = outvalue || ', ' || RTRIM(LTRIM(CHAR(tempval3))); SET loopctr = loopctr + 1; IF LENGTH(outvalue) > 480 THEN SET outvalue = outvalue || ' etc...'; LEAVE calc; END IF; END WHILE; RETURN outvalue; END!
Figure 1052, Fibonacci Series function The following query references the function: WITH temp1 (v1,v2,lp) AS (VALUES (00,01,11) ,(12,61,10) ,(02,05,09) ,(01,-1,08)) SELECT t1.* ,Fibonacci(v1,v2,lp) AS sequence FROM temp1 t1; ANSWER ===================================================================== V1 V2 LP SEQUENCE -- -- -- ----------------------------------------------------------0 1 11 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144 12 61 10 12, 61, 73, 134, 207, 341, 548, 889, 1437, 2326, 3763, 6089 2 5 9 2, 5, 7, 12, 19, 31, 50, 81, 131, 212, 343 1 -1 8 1, -1, 0, -1, -1, -2, -3, -5, -8, -13
Figure 1053, Fibonacci Series generation The above example generates the complete series of values. If needed, the code could easily be simplified to simply return only the last value in the series. Likewise, a recursive join can be used to create a set of rows that are a Fibonacci series.
Fun with SQL
417
Graeme Birchall ©
Business Day Calculation
The following function will calculate the number of business days (i.e. Monday to Friday) between to two dates: CREATE FUNCTION business_days (lo_date DATE, hi_date DATE) RETURNS INTEGER BEGIN ATOMIC DECLARE bus_days INTEGER DEFAULT 0; DECLARE cur_date DATE; SET cur_date = lo_date; WHILE cur_date < hi_date DO IF DAYOFWEEK(cur_date) IN (2,3,4,5,6) THEN SET bus_days = bus_days + 1; IMPORTANT END IF; ============ SET cur_date = cur_date + 1 DAY; This example END WHILE; uses an "!" RETURN bus_days; as the stmt END! delimiter.
Figure 1054, Calculate number of business days between two dates Below is an example of the function in use: WITH temp1 (ld, hd) AS (VALUES (DATE('2006-01-10'),DATE('2007-01-01')) ,(DATE('2007-01-01'),DATE('2007-01-01')) ,(DATE('2007-02-10'),DATE('2007-01-01'))) SELECT t1.* ,DAYS(hd) - DAYS(ld) AS diff ,business_days(ld,hd) AS bdays FROM temp1 t1; ANSWER ================================ LD HD DIFF BDAYS ---------- ---------- ---- ----2006-01-10 2007-01-01 356 254 2007-01-01 2007-01-01 0 0 2007-02-10 2007-01-01 -40 0
Figure 1055, Use business-day function Query Runs for "n" Seconds
Imagine that one wanted some query to take exactly four seconds to run. The following query does just this - by looping (using recursion) until such time as the current system timestamp is four seconds greater than the system timestamp obtained at the beginning of the query: WITH temp1 (num,ts1,ts2) AS (VALUES (INT(1) ,TIMESTAMP(GENERATE_UNIQUE()) ,TIMESTAMP(GENERATE_UNIQUE())) UNION ALL SELECT num + 1 ,ts1 ,TIMESTAMP(GENERATE_UNIQUE()) FROM temp1 WHERE TIMESTAMPDIFF(2,CHAR(ts2-ts1)) < 4 ) SELECT MAX(num) AS #loops ,MIN(ts2) AS bgn_timestamp ,MAX(ts2) AS end_timestamp FROM temp1; ANSWER ============================================================ #LOOPS BGN_TIMESTAMP END_TIMESTAMP ------ -------------------------- -------------------------58327 2001-08-09-22.58.12.754579 2001-08-09-22.58.16.754634
Figure 1056, Run query for four seconds
418
Other Fun Things
DB2 V9.7 Cookbook ©
Observe that the CURRENT TIMESTAMP special register is not used above. It is not appropriate for this situation, because it always returns the same value for each invocation within a single query. Function to Pause for "n" Seconds
We can take the above query and convert it into a user-defined function that will loop for "n" seconds, where "n" is the value passed to the function. However, there are several caveats:
Looping in SQL is a "really stupid" way to hang around for a couple of seconds. A far better solution would be to call a stored procedure written in an external language that has a true pause command.
The number of times that the function is invoked may differ, depending on the access path used to run the query.
The recursive looping is going to result in the calling query getting a warning message.
Now for the code: CREATE FUNCTION pause(inval INT) RETURNS INTEGER NOT DETERMINISTIC EXTERNAL ACTION RETURN WITH ttt (num, strt, stop) AS (VALUES (1 ,TIMESTAMP(GENERATE_UNIQUE()) ,TIMESTAMP(GENERATE_UNIQUE())) UNION ALL SELECT num + 1 ,strt ,TIMESTAMP(GENERATE_UNIQUE()) FROM ttt WHERE TIMESTAMPDIFF(2,CHAR(stop - strt)) < inval ) SELECT MAX(num) FROM ttt;
Figure 1057, Function that pauses for "n" seconds Below is a query that calls the above function: SELECT
FROM WHERE
id ,SUBSTR(CHAR(TIMESTAMP(GENERATE_UNIQUE())),18) AS ss_mmmmmm ,pause(id / 10) AS #loops ,SUBSTR(CHAR(TIMESTAMP(GENERATE_UNIQUE())),18) AS ss_mmmmmm staff id < 31; ANSWER ============================= ID SS_MMMMMM #LOOPS SS_MMMMMM -- --------- ------ --------10 50.068593 76386 50.068587 20 52.068744 144089 52.068737 30 55.068930 206101 55.068923
Figure 1058, Query that uses pause function Sort Character Field Contents
The following user-defined scalar function will sort the contents of a character field in either ascending or descending order. There are two input parameters:
The input string: As written, the input can be up to 20 bytes long. To sort longer fields, change the input, output, and OUT-VAL (variable) lengths as desired.
Fun with SQL
419
Graeme Birchall ©
The sort order (i.e. 'A' or 'D').
The function uses a very simple, and not very efficient, bubble-sort. In other words, the input string is scanned from left to right, comparing two adjacent characters at a time. If they are not in sequence, they are swapped - and flag indicating this is set on. The scans are repeated until all of the characters in the string are in order: --#SET DELIMITER ! CREATE FUNCTION sort_char(in_val VARCHAR(20),sort_dir VARCHAR(1)) RETURNS VARCHAR(20) BEGIN ATOMIC DECLARE cur_pos SMALLINT; DECLARE do_sort CHAR(1); DECLARE out_val VARCHAR(20); IF UCASE(sort_dir) NOT IN ('A','D') THEN SIGNAL SQLSTATE '75001' SET MESSAGE_TEXT = 'Sort order not ''A'' or ''D'''; END IF; SET out_val = in_val; SET do_sort = 'Y'; WHILE do_sort = 'Y' DO SET do_sort = 'N'; IMPORTANT SET cur_pos = 1; ============ WHILE cur_pos < length(in_val) DO This example IF (UCASE(sort_dir) = 'A' uses an "!" AND SUBSTR(out_val,cur_pos+1,1) < as the stmt SUBSTR(out_val,cur_pos,1)) delimiter. OR (UCASE(sort_dir) = 'D' AND SUBSTR(out_val,cur_pos+1,1) > SUBSTR(out_val,cur_pos,1)) THEN SET do_sort = 'Y'; SET out_val = CASE WHEN cur_pos = 1 THEN '' ELSE SUBSTR(out_val,1,cur_pos-1) END CONCAT SUBSTR(out_val,cur_pos+1,1) CONCAT SUBSTR(out_val,cur_pos ,1) CONCAT CASE WHEN cur_pos = length(in_val) - 1 THEN '' ELSE SUBSTR(out_val,cur_pos+2) END; END IF; SET cur_pos = cur_pos + 1; END WHILE; END WHILE; RETURN out_val; END!
Figure 1059, Define sort-char function Here is the function in action:
420
Other Fun Things
DB2 V9.7 Cookbook ©
WITH word1 (w#, word_val) AS (VALUES(1,'12345678') ,(2,'ABCDEFG') ,(3,'AaBbCc') ,(4,'abccb') ,(5,'''%#.') ,(6,'bB') ,(7,'a') ,(8,'')) SELECT w# ,word_val ,sort_char(word_val,'a') sa ,sort_char(word_val,'D') sd FROM word1 ORDER BY w#;
ANSWER ============================= W# WORD_VAL SA SD -- --------- ------- -------1 12345678 12345678 87654321 2 ABCDEFG ABCDEFG GFEDCBA 3 AaBbCc aAbBcC CcBbAa 4 abccb abbcc ccbba 5 '%#. .'#% %#'. 6 bB bB Bb 7 a a a 8
Figure 1060, Use sort-char function Calculating the Median
The median is defined at that value in a series of values where half of the values are higher to it and the other half are lower. The median is a useful number to get when the data has a few very extreme values that skew the average. If there are an odd number of values in the list, then the median value is the one in the middle (e.g. if 7 values, the median value is #4). If there is an even number of matching values, there are two formulas that one can use:
The most commonly used definition is that the median equals the sum of the two middle values, divided by two.
A less often used definition is that the median is the smaller of the two middle values.
DB2 does not come with a function for calculating the median, but it can be obtained using the ROW_NUMBER function. This function is used to assign a row number to every matching row, and then one searches for the row with the middle row number. Using Formula #1
Below is some sample code that gets the median SALARY, by JOB, for some set of rows in the STAFF table. Two JOB values are referenced - one with seven matching rows, and one with four. The query logic goes as follows:
Get the matching set of rows from the STAFF table, and give each row a row-number, within each JOB value.
Using the set of rows retrieved above, get the maximum row-number, per JOB value, then add 1.0, then divide by 2, then add or subtract 0.6. This will give one two values that encompass a single row-number, if an odd number of rows match, and two row-numbers, if an even number of rows match.
Finally, join the one row per JOB obtained in step 2 above to the set of rows retrieved in step 1 - by common JOB value, and where the row-number is within the high/low range. The average salary of whatever is retrieved is the median.
Now for the code:
Fun with SQL
421
Graeme Birchall ©
WITH numbered_rows AS (SELECT s.* ,ROW_NUMBER() OVER(PARTITION BY job ORDER BY salary, id) AS row# FROM staff s WHERE comm > 0 AND name LIKE '%e%'), median_row_num AS (SELECT job ,(MAX(row# + 1.0) / 2) - 0.5 AS med_lo ,(MAX(row# + 1.0) / 2) + 0.5 AS med_hi FROM numbered_rows GROUP BY job) SELECT nn.job ,DEC(AVG(nn.salary),7,2) AS med_sal FROM numbered_rows nn ANSWER ,median_row_num mr ============== WHERE nn.job = mr.job JOB MED_SAL AND nn.row# BETWEEN mr.med_lo AND mr.med_hi ----- -------GROUP BY nn.job Clerk 13030.50 ORDER BY nn.job; Sales 17432.10
Figure 1061, Calculating the median IMPORTANT: To get consistent results when using the ROW_NUMBER function, one must ensure that the ORDER BY column list encompasses the unique key of the table. Otherwise the row-number values will be assigned randomly - if there are multiple rows with the same value. In this particular case, the ID has been included in the ORDER BY list, to address duplicate SALARY values.
The next example is the essentially the same as the prior, but there is additional code that gets the average SALARY, and a count of the number of matching rows per JOB value. Observe that all this extra code went in the second step: WITH numbered_rows AS (SELECT s.* ,ROW_NUMBER() OVER(PARTITION BY job ORDER BY salary, id) AS row# FROM staff s WHERE comm > 0 AND name LIKE '%e%'), median_row_num AS (SELECT job ,(MAX(row# + 1.0) / 2) - 0.5 AS med_lo ,(MAX(row# + 1.0) / 2) + 0.5 AS med_hi ,DEC(AVG(salary),7,2) AS avg_sal ,COUNT(*) AS #rows FROM numbered_rows GROUP BY job) SELECT nn.job ,DEC(AVG(nn.salary),7,2) AS med_sal ,MAX(mr.avg_sal) AS avg_sal ,MAX(mr.#rows) AS #r FROM numbered_rows nn ,median_row_num mr ANSWER WHERE nn.job = mr.job ========================== AND nn.row# BETWEEN mr.med_lo JOB MED_SAL AVG_SAL #R AND mr.med_hi ----- -------- -------- -GROUP BY nn.job Clerk 13030.50 12857.56 7 ORDER BY nn.job; Sales 17432.10 17460.93 4
Figure 1062, Get median plus average Using Formula #2
Once again, the following sample code gets the median SALARY, by JOB, for some set of rows in the STAFF table. Two JOB values are referenced - one with seven matching rows, the
422
Other Fun Things
DB2 V9.7 Cookbook ©
other with four. In this case, when there is an even number of matching rows, the smaller of the two middle values is chosen. The logic goes as follows:
Get the matching set of rows from the STAFF table, and give each row a row-number, within each JOB value.
Using the set of rows retrieved above, get the maximum row-number per JOB, then add 1, then divide by 2. This will get the row-number for the row with the median value.
Finally, join the one row per JOB obtained in step 2 above to the set of rows retrieved in step 1 - by common JOB and row-number value. WITH numbered_rows AS (SELECT s.* ,ROW_NUMBER() OVER(PARTITION BY job ORDER BY salary, id) AS row# FROM staff s WHERE comm > 0 AND name LIKE '%e%'), median_row_num AS (SELECT job ,MAX(row# + 1) / 2 AS med_row# FROM numbered_rows GROUP BY job) SELECT nn.job ,nn.salary AS med_sal ANSWER FROM numbered_rows nn ============== ,median_row_num mr JOB MED_SAL WHERE nn.job = mr.job ----- -------AND nn.row# = mr.med_row# Clerk 13030.50 ORDER BY nn.job; Sales 16858.20
Figure 1063, Calculating the median The next query is the same as the prior, but it uses a sub-query, instead of creating and then joining to a second temporary table: WITH numbered_rows AS (SELECT s.* ,ROW_NUMBER() OVER(PARTITION BY job ORDER BY salary, id) AS row# FROM staff s WHERE comm > 0 AND name LIKE '%e%') SELECT job ,salary AS med_sal FROM numbered_rows WHERE (job,row#) IN ANSWER (SELECT job ============== ,MAX(row# + 1) / 2 JOB MED_SAL FROM numbered_rows ----- -------GROUP BY job) Clerk 13030.50 ORDER BY job; Sales 16858.20
Figure 1064, Calculating the median The next query lists every matching row in the STAFF table (per JOB), and on each line of output, shows the median salary:
Fun with SQL
423
Graeme Birchall ©
WITH numbered_rows AS (SELECT s.* ,ROW_NUMBER() OVER(PARTITION BY job ORDER BY salary, id) AS row# FROM staff s WHERE comm > 0 AND name LIKE '%e%') SELECT r1.* ,(SELECT r2.salary FROM numbered_rows r2 WHERE r2.job = r1.job AND r2.row# = (SELECT MAX(r3.row# + 1) / 2 FROM numbered_rows r3 WHERE r2.job = r3.job)) AS med_sal FROM numbered_rows r1 ORDER BY job ,salary;
Figure 1065, List matching rows and median Converting HEX Data to Number
The following trigger accepts as input a hexadecimal representation of an integer value, and returns a BIGINT number. It works for any integer type:
424
Other Fun Things
DB2 V9.7 Cookbook ©
CREATE FUNCTION hex_to_int(input_val VARCHAR(16)) RETURNS BIGINT BEGIN ATOMIC DECLARE parse_val VARCHAR(16) DEFAULT ''; DECLARE sign_val BIGINT DEFAULT 1; DECLARE out_val BIGINT DEFAULT 0; DECLARE cur_exp BIGINT DEFAULT 1; DECLARE input_len SMALLINT DEFAULT 0; DECLARE cur_byte SMALLINT DEFAULT 1; IF LENGTH(input_val) NOT IN (4,8,16) THEN SIGNAL SQLSTATE VALUE '70001' SET MESSAGE_TEXT = 'Length wrong'; END IF; SET input_len = LENGTH(input_val); WHILE cur_byte
