Top Three SQL Scripts Oracle DBA

 

Top 3 Oracle DBA SQL Scripts HASHJOIN Data Labs  Prepared By: Vitaliy Vitaliy Mogilevskiy # Fri: December 18 2015 DOCID: top_three_sql_scripts_oracle_dba

 

top_three_sql_scripts_oracle_dba

12-18-2015

Table Of Contents I. Why? 1. ASH 1.  ASH Monitoring 2. LOCK Monitoring 3. SPACE Monitoring II. Conclusion III.  Additional Resources Resources

Page 2 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

Why? Currently I have 68 SQL scripts that were battle tested on a very busy database serving a popular internet site. However if you asked me what are the TOP 3 sql scripts that I consider most useful - I would say it’s the scripts allow to Monitor ASH, LOCKS and SPACE. Because if you have three areas covered to thethat point thatme it only takes you a few seconds to answer questions such as:all What is eating up all our space in the last 15 minutes? What happened to the 800GB of free space we added last month? What was causing all the TX locks 3 hours ago? What SQL caused race condition and where did it come from? How many sessions were blocked on row contention this morning? Why did the analytics report take 4 times longer to complete last night? What was the database doing at 3am this morning that consumed almost all I/O bandwidth on the storage? !

Where did the flood of tra c come from? Was it front-end cluster of nodes or back-end DW  jobs? then your life as an Oracle DBA is much easier than a DBA next door! Do I have your attention? If so, lets dive in and I’ll show you the three scripts I use that do exactly that!

Page 3 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

 ASH Monitoring h1.sql is a little gem that I reach for when I am asked to troubleshoot a performance problem that h1.sql is was reported hours ago. For example lets say it’s 9:30am and I get a call from a dev saying that her  APEX web application is hanging - she suspects a locking issue. All I need to know at this point is when the problem was first reported - armed with the start time (lets say 7:00am) I simply do this:  wget https://s3.amazonaws.com/mve-shared/sql/h1.sql https://s3.amazonaws.com/mve-shared/sql/h1.sql sqlplus / as sysdba @h1 0700 0700   0930 0930   0  -1 1  -1 1

What’s happening here? There are 5 parameters: 1: start HHMI

[0700 0700 =  = 7:00 00am] am]

2: end HHMI

[0930 0930 =  = 9:30 30am] am]

3: days back 4: instance

[0 = today; or 7 = seven days back] [1 = INST_ID=1 INST_ID=1, give -1 -1 for for all]  all]

5: service_hash

[1225919510 1225919510 =  = dba_services.name_hash, give -1 - 1 for for all]  all]

Page 4 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

and here’s what I get back:

 h1.sql output sample

Can you spot the problem? It’s SQL*Net message from dblink, there are no locks - it’s a simple problem of a badly written distributed query that is waiting on remote DB. That was easy! One other hidden benefit of this script is that it saves it’s output in a table under a RUN_ID (in this case RUN_ID=81) allowing you to compare the output of two RUN_IDs and clearly spot the di "erences in values grouped by WAIT EVENT. This is extremely valuable especially when someone says - “this used to work last week!”, you simply do this: sqlplus / as sysdba @h1 0700 0700   0930 0930   7  -1 1  -1 1

The “7” in the third parameter instructs the script to look 7 days back for the same time frame (7– 9:30am). The output of the above report will have it’s own RUN_ID=82 (next in sequence) and you can now compare the two using a special script h1d.sql h1d.sql like  like so: @h1d 81 81   82

Page 5 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

We didn’t need OEM or any GUI apps to get to the bottom of the problem - all because the diagnostics data is already in AWR tables and is available to us directly from command line /  sqlplus. Years ago - we’d have to sample v$session_wait to get similar diagnostics, in fact, I wrote a complete monitoring system that utilized such technique. But now, Oracle built this into the core engine in a form of  Active Session History  (ASH)  (ASH) that automatically samples this data every second with practically no overhead to the database! That is an incredible level of instrumentation available to us and it would be a shame not to utilize it beyond what the OEM reports are capable of.

Page 6 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

LOCK Monitoring My #2 most used script is one that can detect and drill down into Oracle’s Blocking LOCKs across an entire Oracle RAC Cluster - locks.sql locks.sql.. Here’s Here’s how to use it:  wget https: https://s3.amazonaws.com/mve-shared/locks.sql //s3.amazonaws.com/mve-shared/locks.sql sqlplus / as as sysdba  sysdba @locks.sql

NOTE: if you don’t have wget on your system try curl instead: curl -o locks.sql https://s3.amazonaws.com/mve-shared/locks.sql https://s3.amazonaws.com/mve-shared/locks.sql

 And here’s here’s an example of locks fully decoded across multiple instances:   1 -- In In this  this example (see lines 45 45-49 49) ) we have a case case of:  of: 2 --

SID-3084 SID3084   on on INST_ID=  INST_ID=4 4 is blocking two SIDs on on INST_ID=  INST_ID=6 6 

3 -4 -- NOTE: lines 12 12-38 38 is  is just an FYI list list of  of all current locks 6 -7   8 17 17: :31 31: :45 45 VMOGILEVSKIY@LPROD3=>  VMOGILEVSKIY@LPROD3=> @locks. @locks.sql 9  10 10 blocked  blocked objects from GV$LOCK GV$LOCK   and and SYS  SYS. .OBJ$ 11  11   12  12  INST_ID SID LMODE MIN_BLOCKED BLOCKED_OBJ 13 ---------13  ---------- ---------- ---------- ----------- ---------------------------14  14  

3 

3961  3961  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

15  15  

3 

3866  3866  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

16  16  

5 

3887  3887  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

17  17  

3 

3484  3484  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

18  18  

3 

3161  3161  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

19  19  

3 

2998  2998  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

20  20  

3 

2979  2979  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

21  21  

3 

2752  2752  

3 

1 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

22  22  

3 

2618  2618  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

23  23  24  24  

3  3 

2610  2610  2456  2456  

3  3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS 0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

Page 7 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

 25 25   

3 

236  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

26  26  

3 

2243  2243  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

27  27  

3 

2134  2134  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

28  28  

3 

2132  2132  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

29  29  

6 

3854  3854  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

30  30  

6 

3507  3507  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

31  31  

6 

3417  3417  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

32  32  33  33  

6  6 

3303  3303  3222  3222  

3  3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS 1 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

34  34  

6 

3135  3135  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

35  35  

6 

2804  2804  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

36  36  

6 

2786  2786  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

37  37  

4 

3818  3818  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

38  38  

4 

2869  2869  

3 

0 APPUSER_OWNER  APPUSER_OWNER. .DBJOBREQUESTS

39  39   40  40  25 25   rows rows selected  selected. . 41  41   42 Elapsed: 42  Elapsed: 00 00: :00 00: :00.03 00.03    43 blocked 43  blocked sessions from GV$LOCK GV$LOCK    44  44   45  45  

INST_ID BLOCKER_SID

INST_ID BLOCKED_SID MIN_BLOCKED

REQUEST

46 ---------46  ---------- ----------- ---------- ----------- ----------- ---------47  47  

4 

3084  3084  

6 

3135  3135  

0 

6 

48  48  

4 

3084  3084  

6 

3485  3485  

0 

6 

49  49   50  50  2 rows rows selected  selected. . 51  51   52 Elapsed: 52  Elapsed: 00 00: :00 00: :00.02 00.02    53 blocked 53  blocked session details from GV$SESSION GV$SESSION   and and GV  GV$SQLTEXT $SQLTEXT    54  54   55 Instance 55  Instance...... ........ .. :

6 

56 Sid 56  Sid ............ ............ :  :

3135  3135  

57 Serial 57  Serial ......... ......... :  :

30604  30604  

58 Username 58  Username ...... ....... . : APP1USER_NAME 59 SQL 59  SQL Id ......... ......... :  : null null    60 Prev 60  Prev SQL Id ... .... . : gm424t8fyx3w6 61 Displayed 61  Displayed SQL Id : gm424t8fyx3w6 62 Client 62  Client Info ... .... . : null null    63 Machine 63  Machine ...... ........ .. : dbt4. dbt4.dc1 dc1.  mydomain.com .  mydomain. 64 OSuser 64  OSuser ......... ......... :  : dbt 65 Process 65  Process ...... ........ .. : 1234 1234   

Page 8 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

 

12-18-2015

.........  .........  

67 SQL_TEXT 67  SQL_TEXT 68 ---------------------------------------------------------------------68  ---------------------------------------------------------------------69  69  select select this_   this_  .WorkRequestId as WorkRequ1_1_0_, this_  .CreateTime a 70 s 70  s CreateTime1_0_, this_  .Event_Type as Event3_1_0_, this_  .Status 71 as 71  as Status1_0_, this_  .UserId as UserId1_0_ from DBJOBREQUESTS thi 72 s_ 72  s_ where  where this_   this_  .WorkRequestId = :1 :1  and and this_   this_  .Status=: Status=:2 2  for or upda  upda 73 te 73 te 74  74   75 Instance 75  Instance...... ........ .. :

6 

76 Sid 76  Sid ............ ............ :  :

3485  3485  

77 Serial 77  Serial ......... ......... :  :

45149  45149  

78 Username 78  Username ...... ....... . : APP1USER_NAME 79 SQL 79  SQL Id ......... ......... :  : null null    80 Prev 80  Prev SQL Id ... .... . : gm424t8fyx3w6 81 Displayed 81  Displayed SQL Id : gm424t8fyx3w6 82 Client 82  Client Info ... .... . : null null    83 Machine 83  Machine ...... ........ .. : dbt5. dbt5.dc1 dc1.  mydomain.com .  mydomain. 84 OSuser ......... 84 OSuser ......... :  : dbt 85 Process 85  Process ...... ........ .. : 1234 1234    86 Action 86  Action ......... ......... :  : JDBC Thin Client 87 SQL_TEXT 87  SQL_TEXT 88 ---------------------------------------------------------------------88  ---------------------------------------------------------------------89  89  select select this_   this_  .WorkRequestId as WorkRequ1_1_0_, this_  .CreateTime a 90 s 90  s CreateTime1_0_, this_  .Event_Type as Event3_1_0_, this_  .Status 91 as 91  as Status1_0_, this_  .UserId as UserId1_0_ from DBJOBREQUESTS thi 92 s_ 92  s_ where  where this_   this_  .WorkRequestId = :1 :1  and and this_   this_  .Status=: Status=:2 2  for or upda  upda 93 te 93  te 94  94   95  95  10 10   rows rows selected  selected. . 96  96   97 Elapsed: 97  Elapsed: 00 00: :00 00: :09.33 09.33    98 blocker 98  blocker session details from GV$SESSION GV$SESSION   and and GV  GV$SQLTEXT $SQLTEXT    99  99  

(current or or previous  previous SQL)

100 Instance 100  Instance...... ........ .. :

4 

101 Sid 101  Sid ............ ............ :  :

3084  3084  

102 Serial 102  Serial ......... ......... :  :

8911  8911  

103 Username 103  Username ...... ....... . : APP1USER_NAME 104 SQL 104  SQL Id ......... ......... :  : null null    105 Prev 105  Prev SQL Id ... .... . : 629 629vx81ykvhpp vx81ykvhpp 106 Displayed 106  Displayed SQL Id : 629 629vx81ykvhpp vx81ykvhpp

Page 9 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

 

12-18-2015

.... ... .

108 Machine 108  Machine ...... ........ .. : dbt1. dbt1.dc1 dc1.  mydomain.com .  mydomain. 109 OSuser 109  OSuser ......... ......... :  : dbt 110 Process 110  Process ...... ........ .. : 1234 1234    111 Action 111  Action ......... ......... :  : JDBC Thin Client 112 SQL_TEXT 112  SQL_TEXT 113 ---------------------------------------------------------------------113  ---------------------------------------------------------------------114 update DBT_LOCK set 114 update set FINISHED=:  FINISHED=:1 1 , VERSION=:2 VERSION=:2  where  where USER_ID=:  USER_ID=:3 3  115  115  and and VERSION=:  VERSION=:4 4  116  116   117  117  2 rows rows selected  selected. . 118  118   119 Elapsed: 119  Elapsed: 00 00: :00 00: :10.13

This script presented few performance challenges because a self-join query against gv$lock  joined with sys.obj$ to get a list of blocked objects is very expensive in a cluster environment, in fact it’s expensive even in a single instance environment. We also have to join gv$session with a gv$lock in order to get the SQL_TEXT of the sessions doing result of self-join query against blocking and being blocked - that’s extremely slow as well.

To solve the above performance challenges I created two tables and indexed them appropriately:

 gv table mapping mapping

Once that was done it was a simple matter of replacing GV$ Table name with COPY Table name on the key joins and performance improved dramatically. In fact, the script ran so fast that I created a custom event in my monitoring system and started to trap occurrences of these locks for historical purposes so that when a developer came to our team and asked if there were any DB locks/blocks 3 hours ago we could simply review our alerts and answer that question with authority providing exact details on the race condition that caused a block. This was much more helpful than the generic alert email we’d get from OEM stating that session 123 is blocking many sessions on instances 1,4 and 5 for example.

Page 10 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

SPACE Monitoring I don’t buy into the “space is cheap” mantra because when we are dealing with SAN/FC storage it’s actually quite expensive especially if it’s FLASH based. If you can e"ectively find the root cause of the vanishing space in an Oracle database you can save a good chunk of money for your organization and make your own life much easier by having fewer requests for new LUNs from the storage team. Back in 2014 I was working for a company that operated a very large internet site and our space consumption was at the rate of 800GB every month, we were adding 4x400GB LUNs to our RAC cluster every two months. Then, one month we almost doubled our space consumption rate for no apparent reason. I decided that it was time to find the root cause of the issue and developed a series of scripts that helped us find the root cause in no time. I’ll show you how it works so you can adopt it in your workflow and DB monitoring. The two scripts are: 1. segs2: Fast Extending Segments in the last X minutes for given TS 2. segs3: Fast Extending Segments Since Last Datafile Creation for given TS These scripts use Oracle AWR, specifically dba_hist_seg_stat and dba_hist_seg_stat_obj so it’s important that you have a recent AWR snapshot to work with you can easily create one using the following command: exec dbms_workload_repository.create_snapshot dbms_workload_repository.create_snapshot(); ();

Be careful don’trun add calland to dbms_workload_repository.create_snapshot  into the actual scripts and- then it a over over again - doing so can put a strain on a busy database.

Page 11 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

Once you have a fresh AWR Snapshot created it’s very easy to use the scripts - simply login to the Oracle DB in question via sqlplus and call the script: For example to find which segments grew in the last 60 minutes in a tablespace DATA   run  run segs2.sql as follows:  wget https: https://s3.amazonaws.com/mve-shared/segs2.sql //s3.amazonaws.com/mve-shared/segs2.sql sqlplus / as as sysdba  sysdba @segs2.sql 60 60 DATA   DATA 

or if you are interested in what segments grew in a tablespace DATA   since the last datafile was  since added run segs3.sql as follows:  wget https: https://s3.amazonaws.com/mve-shared/segs3.sql //s3.amazonaws.com/mve-shared/segs3.sql sqlplus / as as sysdba  sysdba @segs3.sql DATA 

Page 12 of 13

HASHJOIN Corporation © 2015

 

top_three_sql_scripts_oracle_dba

12-18-2015

Conclusion I am hopeful that the TOP 3 Oracle DBA scripts I described above will make it into your own toolkit or will give you an inspiration to write your own versions that tackle these three key areas of an Oracle Database Monitoring: ASH, LOCKs and SPACE.

 Additional Resources Resources If you liked this article and would like more like it, then please subscribe to my newsletter  where I share tips, scripts and tricks I’ve learned during almost two Confessions of an Oracle DBA  where decades in the tech field as an Oracle DBA: http://www.dbatoolz.com/subscribe End.

Page 13 of 13

HASHJOIN Corporation © 2015