Oracle Query.docx

How to find out Expected Time of Completion for an Oracle Query Too often we become impatient when Oracle Query executed by us does not seem to return any result. But Oracle (10g onwards) gives us an option to check how long a query will run, that is, to find out expected time of completion for a query. The option is using v$session_longops. Below is a sample query that will give you percentage of completion of a running Oracle query and Expected Time to Complete in minutes,

Script

SELECT opname, target, ROUND((sofar/totalwork),4)*100 Percentage_Complete, start_time, CEIL(time_remaining/60) Max_Time_Remaining_In_Min, FLOOR(elapsed_seconds/60) Time_Spent_In_Min FROM v$session_longops WHERE sofar != totalwork; If you have access to v$sqlarea table, then you can use another version of the above query that will also show you the exact SQL running. Here is how to get it, SELECT opname target, ROUND((sofar/totalwork),4)*100 Percentage_Complete, start_time, CEIL(TIME_REMAINING

/60) MAX_TIME_REMAINING_IN_MIN,

FLOOR(ELAPSED_SECONDS/60) TIME_SPENT_IN_MIN, AR.SQL_FULLTEXT, AR.PARSING_SCHEMA_NAME, AR.MODULE client_tool

FROM V$SESSION_LONGOPS L, V$SQLAREA AR WHERE L.SQL_ID = AR.SQL_ID AND TOTALWORK > 0 AND ar.users_executing > 0 AND sofar != totalwork;

NOTE This query will give you correct result only if a FULL Table Scan or INDEX FAST FULL SCAN are being performed by the database for your query. In case, there is no full table/index fast full scan, you can force Oracle to perform a full table scan by specifying /*+ FULL() */ hint.

Understanding Oracle QUERY PLAN - A 10 minutes guide Confused about how to understand Oracle Query Execution Plan? This 10 minutes step by step primer is the first of a two part article that will teach you exactly the things you must know about Query Plan.

What is Query Execution Plan? When you fire an SQL query to Oracle, Oracle database internally creates a query execution plan in order to fetch the desired data from the physical tables. The query execution plan is nothing but a set of methods on how the database will access the data from the tables. This query execution plan is crucial as different execution plans will need different cost and time for the query execution. How the Execution Plan is created actually depends on what type of query optimizer is being used in your Oracle database. There are two different optimizer options – Rule based (RBO) and Cost based (CBO) Optimizer. For Oracle 10g, CBO is the default optimizer. Cost Based optimizer enforces Oracle to generate the optimization plan by taking all the related table statistics into consideration. On the other hand, RBO uses a fixed set of pre-defined rules to generate the query plan. Obviously such fixed set of rules may not always be able to create the plan that is most efficient in nature. This is because an efficient plan will depend heavily on the nature and volume of tables’ data. Because of this reason, CBO is preferred over RBO.

Understanding Oracle Query Execution Plan But this article is not for comparing RBO and CBO (In fact, there is not much point in comparing these two). This article will briefly help you understand, 1.

How can we see Query Execution Plan

2.

How do we understand (or rather interpret) the execution plan.

So let’s begin. I will be using Oracle 10g server and SQL *Plus client to demonstrate all the details.

Oracle Full Table Scan (FTS) Let’s start by creating a simple product table with the following structure, ID number(10) NAME varchar2(100) DESCRIPTION varchar2(255) SERVICE varchar2(30) PART_NUM varchar2(50) LOAD_DATE date Next I will insert 15,000 records into this newly created table (data taken from one of my existing product table from one of my client’s production environment). Remember, currently there is no index on the table. So we start our journey by writing a simple select statement on this table as below,

SQL> explain plan for select * from product; Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT ----------------------------------------------------------

Plan hash value: 3917577207 ------------------------------------| Id

| Operation

| Name

|

------------------------------------|

0 | SELECT STATEMENT

|

1 |

|

|

TABLE ACCESS FULL | PRODUCT|

-------------------------------------

Note ----- rule based optimizer used (consider using cbo)

Notice that optimizer has decided to use RBO instead of CBO as Oracle does not have any statistics for this table. Let’s now build some statistics for this table by issuing the following command, SQL> Analyze table product compute statistics; Now let’s do the same experiment once again,

SQL> explain plan for select * from product; Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT ----------------------------------------------------Plan hash value: 3917577207 ----------------------------------------------------| Id

| Operation

| Name

| Rows

| Bytes |

----------------------------------------------------|

0 | SELECT STATEMENT

|

1 |

|

| 15856 |

1254K|

TABLE ACCESS FULL | PRODUCT | 15856 |

1254K|

-----------------------------------------------------

You can easily see that this time optimizer has used Cost Based Optimizer (CBO) and has also detailed some additional information (e.g. Rows etc.) The point to note here is, Oracle is reading the whole table (denoted by TABLE ACCESS FULL) which is very obvious because the select * statement that is being fired is trying to read everything. So, there’s nothing interesting up to this point.

Index Unique Scan Now let’s add a WHERE clause in the query and also create some additional indexes on the table. SQL> create unique index idx_prod_id on product (id) compute statistics;

Index created.

SQL>

explain plan for select id from product where id = 100;

Explained.

SQL>

select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT --------------------------------------------------------Plan hash value: 2424962071

--------------------------------------------------------| Id

| Operation

| Name

| Rows

| Bytes |

--------------------------------------------------------|

0 | SELECT STATEMENT

|*

1 |

|

|

1 |

4 |

INDEX UNIQUE SCAN | IDX_PROD_ID |

1 |

4 |

---------------------------------------------------------

So the above statement indicates that CBO is performing Index Unique Scan. This means, in order to fetch the id value as requested, Oracle is actually reading the index only and not the whole table. Of course this will be faster than FULL TABLE ACCESS operation shown earlier.

Table Access by Index RowID Searching the index is a fast and an efficient operation for Oracle and when Oracle finds the desired value it is looking for (in this case id=100), it can also find out the rowid of the record in product table that has id=100. Oracle can then use this rowid to fetch further information if requested in query. See below, SQL> explain plan for select * from product where id = 100;

Explained.

SQL>

select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT ---------------------------------------------------------Plan hash value: 3995597785

---------------------------------------------------------| Id

| Operation

| Name

|Rows | Bytes|

---------------------------------------------------------|

0 | SELECT STATEMENT

|

1 |

|*

2 |

|

1 |

81 |

TABLE ACCESS BY INDEX ROWID| PRODUCT|

1 |

81 |

INDEX UNIQUE SCAN

|

| IDX_PROD_ID | 1 |

|

----------------------------------------------------------

TABLE ACCESS BY INDEX ROWID is the interesting part to check here. Since now we have specified select * for id=100, so Oracle first use the index to obtain the rowid of the record. And then it selects all the columns by the rowid.

Index Range Scan But what if we specify a >, or between criteria in the WERE clause instead of equality condition? Like below,

SQL> explain plan for select id from product where id select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT --------------------------------------------Plan hash value: 1288034875

------------------------------------------------------| Id

| Operation

| Name

| Rows

| Bytes |

------------------------------------------------------|

0 | SELECT STATEMENT |

|

7 |

28 |

|*

1 |

INDEX RANGE SCAN| IDX_PROD_ID |

7 |

28 |

-------------------------------------------------------

So this time CBO goes for an Index Range Scan instead of INDEX UNIQUE SCAN. The same thing will normally happen if we use a between clause also.

Index Fast Full Scan Now, let’s see another interesting aspect of INDEX scan here by just altering the “ 10”. Before we see the outcome, just remind yourself that there are 15000 over products with their ids starting from 1 to 15000+. So if we write “10” we are likely to get almost 14990+ records in return. So does Oracle go for an INDEX RANGE SCAN in this case? Let’s see, SQL> explain plan for select id from product where id>10;

Explained.

SQL>

select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT -----------------------------------------------Plan hash value: 2179322443

-------------------------------------------------------| Id

| Operation

| Name

| Rows |Bytes |

-------------------------------------------------------|

0 | SELECT STATEMENT

|*

1 |

|

| 15849|63396 |

INDEX FAST FULL SCAN| IDX_PROD_ID| 15849|63396 |

---------------------------------------------------------

So, Oracle is actually using a INDEX FAST FULL SCAN to “quickly” scan through the index and return the records from table. This scan is "quick" because unlike index full scan or index unique scan, INDEX FAST FULL SCAN utilizes multiple-block input-output (I/O) whereas the formers utilizes single block I/O.

Note on QUERY PLAN FTS or Full Table Scan



Whole table is read upto high water mark



Uses multiblock input/output



Buffer from FTS operation is stored in LRU end of buffer cache

Index Unique Scan



Single block input/output

Index Fast Full Scan



Multi block i/o possible



Returned rows may not be in sorted order

Index Full Scan



Single block i/o



Returned rows generally will be in sorted order

Understanding Oracle QUERY PLAN - Part2 (Exploring SQL Joins)

This is the second part of the article Understanding Oracle Query Plan. In this part we will deal with SQL Joins. For the first part of this article, click here This time we will explore and try to understand query plan for joins. Let’s take on joining of two tables and let’s find out how Oracle query plan changes. We will start with two tables as following, Product Table - Stores 15000 products. Each product has unique numeric id. Buyer Table - Stores 15,000,00 buyers who buy the above products. This table has unique id field as well as a prodid (product id) field that links back to the product table. Before we start, please note, we do not have any index or table statistics present for these tables.

SORT MERGE JOIN

SQL> explain plan for SELECT * 2

FROM PRODUCT, BUYER

3

WHERE PRODUCT.ID = BUYER.PRODID;

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT ---------------------------------------------

--------------------------------------| Id

| Operation

| Name

|

--------------------------------------|

0 | SELECT STATEMENT

|

|

|

1 |

MERGE JOIN

|

|

|

2 |

SORT JOIN

|

|

|

3 |

|*

4 |

|

5 |

TABLE ACCESS FULL| BUYER SORT JOIN

|

| |

TABLE ACCESS FULL| PRODUCT |

---------------------------------------

Above plan tells us that CBO is opting for a Sort Merge join. In this type of joins, both tables are read individually and then sorted based on the join predicate and after that sorted results are merged together (joined).

Read Product ---> Sort by product id ------| |---> join Read Buyer

---> Sort by product id ------|

Joins are always a serial operation even though individual table access can be parallel. Now let’s create some statistics for these tables and let’s check if CBO does something else than SORT MERGE join.

HASH JOIN SQL> analyze table product compute statistics;

Table analyzed.

SQL> analyze table buyer compute statistics;

Table analyzed.

SQL> explain plan for SELECT * 2

FROM PRODUCT, BUYER

3

WHERE PRODUCT.ID = BUYER.PRODID;

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT -----------------------------------------------------Plan hash value: 2830850455

-----------------------------------------------------| Id

| Operation

| Name

| Rows

| Bytes |

-----------------------------------------------------|

0 | SELECT STATEMENT

|

| 25369 |

2279K|

|*

1 |

|

| 25369 |

2279K|

|

2 |

TABLE ACCESS FULL| PRODUCT | 15856 |

1254K|

|

3 |

TABLE ACCESS FULL| BUYER

1718K|

HASH JOIN

|

159K|

------------------------------------------------------

CBO chooses to use Hash join instead of SMJ once the tables are analyzed and CBO has enough statistics. Hash join is a comparatively new join algorithm which is theoretically more efficient than other types of joins. In hash join, Oracle chooses the smaller table to create an intermediate hash table and a bitmap. Then the second row source is hashed and checked against the intermediate hash table for matching joins. The bitmap is used to quickly check if the rows are present in hash table. The bitmap is especially handy if the hash table is too huge. Remember only cost based optimizer uses hash join. Also notice the FTS operation in the above example. This may be avoided if we create some index on both the tables. Watch this, SQL> create index idx_prod_id on product (id);

Index created.

SQL> create index idx_buyer_prodid on buyer (prodid);

Index created.

SQL>

explain plan for select product.id

2

FROM PRODUCT, BUYER

3

WHERE PRODUCT.ID = BUYER.PRODID;

Explained.

SQL>

select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT ------------------------------------------------------------------

-----------------------------------------------------------------| Id

| Operation

| Name

| Rows

| Bytes |

-----------------------------------------------------------------|

0 | SELECT STATEMENT

|

| 25369 |

198K|

|*

1 |

|

| 25369 |

198K|

|

2 |

INDEX FAST FULL SCAN| IDX_PROD_ID

|

3 |

INDEX FAST FULL SCAN| IDX_BUYER_PRODID |

HASH JOIN

| 15856 | 63424 | 159K|

624K|

------------------------------------------------------------------

NESTED LOOP JOIN There is yet another kind of joins called Nested Loop Join. In this kind of joins, each record from one source is probed against all the records of the other source. The performance of nested loop join depends heavily on the number of records returned from the first source. If the first source returns more record, that means there will be more probing on the second table. If the first source returns less record, that means, there will be less probing on the second table. To show a nested loop, let’s introduce one more table. We will just copy the product table into a new table, product_new. All these tables will have index. Now I write a simple query below,

select

*

from buyer, product, product_new where buyer.prodid=product.id and buyer.prodid = product_new.id;

And then I checked the plan. But the plan shows a HASH JOIN condition and not a NESTED LOOP. This is, in fact, expected because as discussed earlier hash-join is more efficient compared to other joins. But remember hash join is only used for cost based optimizer. So if I force Oracle to use rule based optimizer, I might be able to see nested joins. I can do that by using a query hint. Watch this, SQL> explain plan for 2

select /*+ RULE */ *

3

from buyer, product, product_new

4

where buyer.prodid=product.id

5

and buyer.prodid = product_new.id;

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT ----------------------------------------------------------Plan hash value: 3711554028

----------------------------------------------------------| Id

| Operation

| Name

|

----------------------------------------------------------|

0 | SELECT STATEMENT

|

|

|

1 |

| PRODUCT

|

|

2 |

|

|

|

3 |

|

|

|

4 |

TABLE ACCESS FULL

| PRODUCT_NEW

|

|

5 |

TABLE ACCESS BY INDEX ROWID| BUYER

|*

6 |

|*

7 |

TABLE ACCESS BY INDEX ROWID NESTED LOOPS NESTED LOOPS

INDEX RANGE SCAN INDEX RANGE SCAN

|

| IDX_BUYER_PRODID | | IDX_PROD_ID

|

----------------------------------------------------------Voila! I got nested loops! As you see, this time I have forced Oracle to use rule based optimizer by providing /*+ RULE */ hint. So Oracle has now no option but to use nested loops. As apparent from the plan, Oracle performs a full scan of product_new and index scans for other tables. First it joins buyer with product_new by feeding each row of buyer to product_new and then it sends the result set to probe against product.

Ok, with this I will conclude this article. The main purpose of this article and the earlier one was to make you familiar on Oracle query execution plans. Please keep all these ideas in mind because in my next article I will show how we can use this knowledge to better tune our SQL Queries. Stay tuned.

Oracle Analytic Functions Oracle Analytic Functions compute an aggregate value based on a group of rows. It opens up a whole new way of looking at the data. This article explains how we can unleash the full potential of this. Analytic functions differ from aggregate functions in the sense that they return multiple rows for each group. The group of rows is called a window and is defined by the analytic clause. For each row, a sliding window of rows is defined. The window determines the range of rows used to perform the calculations for the current row. Oracle provides many Analytic Functions such as AVG, CORR, COVAR_POP, COVAR_SAMP, COUNT, CUME_DIST, DENSE_RANK, FIRST, FIRST_VALUE, LAG, LAST, LAST_VALUE, LEAD, MAX, MIN, NTILE, PERCENT_RANK, PERCENTILE_CONT, PERCENTILE_DISC, RANK, RATIO_TO_REPORT, STDDEV, STDDEV_POP, STDDEV_SAMP, SUM, VAR_POP, VAR_SAMP, VARIANCE. The Syntax of analytic functions: Analytic-Function(Column1,Column2,...) OVER ( [Query-Partition-Clause] [Order-By-Clause] [Windowing-Clause] ) Analytic functions take 0 to 3 arguments. An Example: SELECT ename, deptno, sal, SUM(sal)

OVER (ORDER BY deptno, ename) AS Running_Total, SUM(sal) OVER ( PARTITION BY ORDER BY

deptno

ename) AS Dept_Total,

ROW_NUMBER() OVER (PARTITION BY deptno ORDER BY ename) As Sequence_No FROM emp ORDER BY deptno, ename;

The partition clause makes the SUM(sal) be computed within each department, independent of the other groups. The SUM(sal) is 'reset' as the department changes. The ORDER BY ENAME clause sorts the data within each department by ENAME; 1.

Query-Partition-Clause The PARTITION BY clause logically breaks a single result set into N groups, according to the criteria set by the partition expressions. The analytic functions are applied to each group independently, they are reset for each group.

2.

Order-By-Clause The ORDER BY clause specifies how the data is sorted within each group (partition). This will definitely affect the output of the analytic function.

3.

Windowing-Clause

The windowing clause gives us a way to define a sliding or anchored window of data, on which the analytic function will operate, within a group. This clause can be used to have the analytic function compute its value based on any arbitrary sliding or anchored window within a group. The default window is an anchored window that simply starts at the first row of a group an continues to the current row. Let's look an example with a sliding window within a group and compute the sum of the current row's salary column plus the previous 2 rows in that group. i.e ROW Window clause: SELECT deptno, ename, sal, SUM(sal) OVER

( PARTITION BY

deptno

ORDER BY ename ROWS 2 PRECEDING

) AS Sliding_Total

FROM emp ORDER BY deptno, ename;

Now if we look at the Sliding Total value of SMITH it is simply SMITH's salary plus the salary of two preceding rows in the window. [800+3000+2975 = 6775] We can set up windows based on two criteria: RANGES of data values or ROWS offset from the current row . It can be said, that the existance of an ORDER BY in an analytic function will add a default window clause

of RANGE UNBOUNDED PRECEDING. That says to get all rows in our partition that came before us as specified by the ORDER BY clause.

** Solving Top-N Queries ** Suppose we want to find out the top 3 salaried employee of each department: SELECT deptno, ename, sal, ROW_NUMBER() OVER ( PARTITION BY deptno ORDER BY sal DESC ) Rnk FROM emp; This will give us the employee name and salary with ranks based on descending order of salary for each department or the partition/group . Now to get the top 3 highest paid employees for each dept. SELECT * FROM ( SELECT deptno, ename, sal, ROW_NUMBER() OVER ( PARTITION BY deptno ORDER BY sal DESC ) Rnk FROM emp ) WHERE Rnk CONN SCOTT/TIGER@ORCL Connected. SQL> SET FEEDBACK OFF HEADING OFF ECHO OFF SQL> SPOOL C:\External_Tables\emp.csv

SQL> @C:\External_Tables\emp_query.sql SQL> SPOOL OFF To view the content of the file from SQLPLUS, type.. SQL> host type C:\External_Tables\emp.csv

Oracle to XML file Now let us see how can we output the table data in the form of xml file format. The following code will display the xml format of the table data. Now the easy way to create an xml file is to use the Oracle SPOOL sqlplus command to spool the output in the client machine. Example 1: SELECT DBMS_XMLGEN.GETXML( 'SELECT empno, ename, deptno FROM emp WHERE deptno = 10' , 0 ) FROM DUAL; Example 2: SELECT DBMS_XMLGEN.GETXML( 'SELECT dept.*' ||'

,CURSOR('

||'

SELECT emp.*'

||'

FROM emp'

||'

WHERE emp.deptno = dept.deptno'

||'

) emp_list'

||' FROM dept') xmldata FROM dual;

Database Performance Tuning This article tries to comprehensively list down many things one needs to know for Oracle Database Performance Tuning. The ultimate goal of this document is to provide a generic and comprehensive guideline to Tune Oracle Databases from both programmer and administrator's standpoint.

Oracle terms and Ideas you need to know before beginning Just to refresh your Oracle skills, here is a short go-through as a starter.

Oracle Parser It performs syntax analysis as well as semantic analysis of SQL statements for execution, expands views referenced in the query into separate query blocks, optimizing it and building (or locating) an executable form of that statement.

Hard Parse A hard parse occurs when a SQL statement is executed, and the SQL statement is either not in the shared pool, or it is in the shared pool but it cannot be shared. A SQL statement is not shared if the metadata for the two SQL statements is different i.e. a SQL statement textually identical to a preexisting SQL statement, but the tables referenced in the two statements are different, or if the optimizer environment is different.

Soft Parse A soft parse occurs when a session attempts to execute a SQL statement, and the statement is already in the shared pool, and it can be used (that is, shared). For a statement to be shared, all data, (including metadata, such as the optimizer execution plan) of the existing SQL statement must be equal to the current statement being issued.

Cost Based Optimizer It generates a set of potential execution plans for SQL statements, estimates the cost of each plan, calls the plan generator to generate the plan, compares the costs, and then chooses the plan with the lowest cost. This approach is used when the data dictionary has statistics for at least one of the tables accessed

by the SQL statements. The CBO is made up of the query transformer, the estimator and the plan generator.

EXPLAIN PLAN A SQL statement that enables examination of the execution plan chosen by the optimizer for DML statements. EXPLAIN PLAN makes the optimizer to choose an execution plan and then to put data describing the plan into a database table. The combination of the steps Oracle uses to execute a DML statement is called an execution plan. An execution plan includes an access path for each table that the statement accesses and an ordering of the tables i.e. the join order with the appropriate join method.

Oracle Trace Oracle utility used by Oracle Server to collect performance and resource utilization data, such as SQL parse, execute, fetch statistics, and wait statistics. Oracle Trace provides several SQL scripts that can be used to access server event tables, collects server event data and stores it in memory, and allows data to be formatted while a collection is occurring.

SQL Trace It is a basic performance diagnostic tool to monitor and tune applications running against the Oracle server. SQL Trace helps to understand the efficiency of the SQL statements an application runs and generates statistics for each statement. The trace files produced by this tool are used as input for TKPROF.

TKPROF It is also a diagnostic tool to monitor and tune applications running against the Oracle Server. TKPROF primarily processes SQL trace output files and translates them into readable output files, providing a summary of user-level statements and recursive SQL calls for the trace files. It also shows the efficiency of SQL statements, generate execution plans, and create SQL scripts to store statistics in the database.

Oracle External Tables The Oracle external tables feature allows us to access data in external sources as if it is a table in the database. This is a very convenient and fast method to retrieve data from flat files outside Oracle database.

What is an Oracle External Tables? The Oracle external tables feature allows us to access data in external sources as if it is a table in the database. External tables are read-only. No data manipulation language (DML) operations is allowed on an external table. An external table does not describe any data that is stored in the database.

So, how do I create an external table? To create an external table in Oracle we use the same CREATE TABLE DDL, but we specify the type of the table as external by an additional clause - ORGANIZATION EXTERNAL. Also we need to define a set of other parameters called ACCESS PARAMETERS in order to tell Oracle the location and structure of the source data. To understand the syntax of all these, let's start by creating an external table right away. First we will connect to the database and create a directory for the extrnal table. CONN SYS/SYS_PWORD AS SYSDBA CREATE

OR REPLACE DIRECTORY ext_tab_dir AS

'C:\External_Tables';

GRANT READ,WRITE ON DIRECTORY ext_tab_dir TO scott;

Flat File Structure We will start by trying to load a flat file as an external table. Suppose the flat file is named employee1.dat with the content as: empno,first_name,last_name,dob 1234,John,Lee,"31/12/1978" 7777,Sam,vichi,"19/03/1975" So our CREATE TABLE syntax will be something like below

CREATE TABLE Example for External Table CREATE TABLE emp_ext( empno NUMBER(4), first_name CHAR(20), last_name CHAR(20), dob CHAR(10)

ORGANIZATION EXTERNAL( TYPE ORACLE_LOADER DEFAULT DIRECTORY ext_tab_dir ACCESS PARAMETERS ( RECORDS DELIMITED BY NEWLINE SKIP 1 BADFILE 'bad_%a_%p.bad' LOGFILE 'log_%a_%p.log' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'

LRTRIM

MISSING FIELD VALUES ARE NULL REJECT ROWS WITH ALL NULL FIELDS (empno INTEGER EXTERNAL (4), first_name CHAR(20), last_name CHAR(20), dob CHAR(10) DATE_FORMAT DATE MASK "dd/mm/yyyy") ) LOCATION ('employee1.dat','employee2.dat') ) PARALLEL REJECT LIMIT 0;

SELECT * FROM emp_ext; Now we can insert this temporary read only data to our oracle table say employee. INSERT INTO employee (empno, first_name, last_name, dob) (SELECT empno, first_name, last_name, dob FROM emp_ext);

Explanation of the above External Table Syntax 

The SKIP no_rows clause allows you to eliminate the header of the file by skipping the first row.



The LRTRIM clause is used to trim leading and trailing blanks from fields.



The SKIP clause skips the specified number of records in the datafile before loading. SKIP can be specified only when nonparallel access is being made to the data.



The READSIZE parameter specifies the size of the read buffer. The size of the read buffer is a limit on the size of the largest record the access driver can handle. The size is specified with an integer

indicating the number of bytes. The default value is 512KB (524288 bytes). You must specify a larger value if any of the records in the datafile are larger than 512KB.



The LOGFILE clause names the file that contains messages generated by the external tables utility while it was accessing data in the datafile. If a log file already exists by the same name, the access driver reopens that log file and appends new log information to the end. This is different from bad files and discard files, which overwrite any existing file. NOLOGFILE is used to prevent creation of a log file. If you specify LOGFILE, you must specify a filename or you will receive an error. If neither LOGFILE nor NOLOGFILE is specified, the default is to create a log file. The name of the file will be the table name followed by _%p.



The BADFILE clause names the file to which records are written when they cannot be loaded because of errors. For example, a record was written to the bad file because a field in the datafile could not be converted to the datatype of a column in the external table. Records that fail the LOAD WHEN clause are not written to the bad file but are written to the discard file instead. The purpose of the bad file is to have one file where all rejected data can be examined and fixed so that it can be loaded. If you do not intend to fix the data, then you can use the NOBADFILE option to prevent creation of a bad file, even if there are bad records. If you specify BADFILE, you must specify a filename or you will receive an error. If neither BADFILE nor NOBADFILE is specified, the default is to create a bad file if at least one record is rejected. The name of the file will be the table name followed by _%p.



With external tables, if the SEQUENCE parameter is used, rejected rows do not update the sequence number value. For example, suppose we have to load 5 rows with sequence numbers beginning with 1 and incrementing by 1. If rows 2 and 4 are rejected, the successfully loaded rows are assigned the sequence numbers 1, 2, and 3.

External Table Access Driver An external table describes how the external table layer must present the data to the server. The access driver and the external table layer transform the data in the datafile to match the external table definition. The access driver runs inside of the database server hence the server must have access to any files to be loaded by the access driver. The server will write the log file, bad file, and discard file created by the access driver. The access driver does not allow to specify random names for a file. Instead, we have to specify directory objects as the locations from where it will read the datafiles and write logfiles. A directory object maps a name with the directory name on the file system.

Directory objects can be created by DBAs or by any user with the CREATE ANY DIRECTORY privilege. After a directory is created, the user creating the directory object needs to grant READ or WRITE permission on the directory to other users.

Notes 1.

If we do not specify the type for the external table, then the ORACLE_LOADER type is used as a default.

2.

Using the PARALLEL clause while creating the external table enables parallel processing on the datafiles. The access driver then attempts to divide large datafiles into chunks that can be processed separately and parallely. With external table loads, there is only one bad file and one discard file for all input datafiles. If parallel access drivers are used for the external table load, each access driver has its own bad file and discard file.

3.

We can change the target file name with the alter ddl command as: ALTER TABLE emp_tab LOCATION ('newempfile.dat');

4.

The SYS tables for Oracle External Tabbles are dba_external_tables, all_external_tables and user_external_tables.

Learn Oracle Server Architecture in 10 minutes Here is an easy to understand primer on Oracle architecture. Read this first to give yourself a head-start before you read more advanced articles on Oracle Server Architecture. We need to touch two major things here- first server architecture where we will know memory and process structure and then we will learn the Oracle storage structure.

Database and Instance Let’s first understand the difference between Oracle database and Oracle Instance.

Oracle database is a group of files that reside on disk and store the data. Whereas an Oracle instance is a piece of shared memory and a number of processes that allow information in the database to be accessed quickly and by multiple concurrent users. The following picture shows the parts of database and instance.

Database

Instance



Control File



Online Redo Log



Shared Memory (SGA)



Data File



Processes



Temp File

Now let's learn some details of both Database and Oracle Instance.

Oracle Database The database is comprised of different files as follows

Control

Control File contains information that defines the rest of the database like

File

names, location and types of other files etc.

Redo Log

Redo Log file keeps track of the changes made to the database. All user and

file

meta data are stored in data files

Temp file

Temp file stores the temporary information that are often generated when sorts are performed.

Each file has a header block that contains metadata about the file like SCN or system change number that says when data stored in buffer cache was flushed down to disk. This SCN information is important for Oracle to determine if the database is consistent.

Oracle Instance

This is comprised of a shared memory segment (SGA) and a few processes. The following picture shows the Oracle structure.

Shared Memory Segment

Shared Pool Shared SQL Area

Contains various structure for running SQL and dependency tracking

Database Buffer

Contains various data blocks that are read from database for some

Cache

transaction

Redo Log Buffer

It stores the redo information until the information is flushed out to disk

Details of the Processes are shown below

- Cleans up abnormally terminated connection PMON (Process Monitor)

- Rolls back uncommited transactions - Releases locks held by a terminated process - Frees SGA resources allocated to the failed processes - Database maintenance

SMON (System

- Performs automatic instance recovery

Monitor)

- Reclaims space used by temporary segments no longer in use - Merges contiguous area of free space in the datafile

DBWR

- write all dirty buffers to datafiles

(Database

- Use a LRU algorithm to keep most recently used blocks in memory

Writer)

- Defers write for I/O optimization

LGWR (Log Writer)

- writes redo log entries to disk - If enabled (by setting the parameter CHECKPOINT_PROCESS=TRUE),

CKPT (Check

take over LGWR’s task of updating files at a checkpoint

Point)

- Updates header of datafiles and control files at the end of checkpoint - More frequent checkpoint reduce recovery time from instance failure

Other Processes

LCKn (Lock), Dnnn (Dispatcher), Snnn (Server), RECO (Recover), Pnnn(Parallel), SNPn(Job Queue), QMNn(Queue Monitor) etc.

Oracle Storage Structure Here we will learn about both physical and logical storage structure. Physical storage is how Oracle stores the data physically in the system. Whereas logical storage talks about how an end user actually accesses that data. Physically Oracle stores everything in file, called data files. Whereas an end user accesses that data in terms of accessing the RDBMS tables, which is the logical part. Let's see the details of these structures. Physical storage space is comprised of different datafiles which contains data segments. Each segment can contain multiple extents and each extent contains the blocks which are the most granular storage structure. Relationship among Segments, extents and blocks are shown below.

Data Files | ^ Segments (size: 96k) |

^ Extents (Size: 24k) | ^ Blocks (size: 2k)