Database Operations

Database Operations

SAP AG 1999

1 System Operations

5 System Management

2 Deliveries - Tools Requirements

6 High Availability

3 Performance Management

7 Printing

4 Database Operations

8 Change Management

Database Operations

Purpose Verify the backup/recovery procedure Verify the disaster recovery procedure Verify database performance management Verify database monitoring procedure Verify growing management procedures

Key Issues Backup/recovery concept Disaster recovery concept File distribution Database performance Database monitoring Growing management

SAP AG 1999

Key Performance Indicators

Indicator Avg. DB Request Time in Dialog Task Avg. DB Request Time in Update Task Reads / User call SQL Statements with inefficient search strategy SQL Statements with high data transfer volume

Value

Target

DB Size Last Month DB Growth Top growing tables

Values depend on the avg. number of users

SAP AG 1999

Database Layer: Key Questions

Is backup strategy appropriate ? WHAT?

Are emergency procedures defined ? Are administrative tasks for regular operation defined ?

Database administrator Operating system administrator WHO?

System administrator Technical team lead/Technical consultant Network administrator

SAP AG 1999

Considerations for Setting Up a Backup Strategy

Backups: How often? When ?

!

24-hour availability? Maximum downtime?

Volume of data backed up? Backup duration? Backup cycle length? Costs? Administrator skills

SAP AG 1999

Emergency Procedures for Worst Case Scenario Obtain new hardware

Reinstall operating system

Apply database backup

Recover database

SAP AG 1999

Monitoring Tasks

Free space

Missing indexes

Database logs

Database performance Last successful backup

Expensive SQL statements

SAP AG 1999

Maintaining Your Database

Apply database updates and patches provided by your database manufacturer

Manage the growth of your database

Adapt your backup strategy to database growth

SAP AG 1999

Unit Content

Subunit 1

Backup Strategy and Tape Management

Subunit 2

Scheduling, Performing, and Monitoring Backups

Subunit 3

Restore and Recovery

Subunit 4

Advanced Backup Techniques

Subunit 5

Storage Management

SAP AG 1999

The Importance of Database Backups Physical errors External factors

(Such as hardware failure)

(Such as fire or water damage)

Logical errors (Such as a deleted table) DROP MARA

Data loss

A good database backup strategy prevents data loss and minimizes system downtime Procedure and Escalation Plan

SAP AG 1999

Preventing and Handling Errors

Logical data check: Verify database consistency

Physical data check: Verify backup on tape

ORA-1578: Oracle data block corrupted

Oracle data files

Database backup

SAP AG 1999

Possible Causes of Data Loss (1) Logical error recovery Logical Logical errors errors (Such as a deleted table) DROP MARA

Lost information 12:50 - 4:00

12:50 Time when logical error occurred

4:00 Time when database is stopped for recovery

You must recover the complete database to a point in time before the error

SAP AG 1999

Possible Causes of Data Loss (2) Forward recovery A database backup is restored and you want to recover data from offline redo log files

Sequence of offline redo log files:

Lost offline redo log file

Lost information

Point in time of the database error

Intact but unusable offline redo log files

If one offline redo log file is lost, none of the files that follow it can be used SAP AG 1999

Time

Possible Causes of Data Loss (3) Disaster recovery Physical errors

External factors

(Such as hardware failure)

(Such as fire or water damage)

Only data saved on tape can be recovered

SAP AG 1999

Only tapes stored in a safe location can be recovered

Backup Cycle Recommendations

Verify the database Verify the backup Offline

Verify the backup Online Offline redo log file backup (x2)

28 days Additional

SAP AG 1999

Online Offline redo log file backup (x2)

SAP Database Backup Tools Oracle database Control file

Detail log

Data files

Offline redo log files

Detail log

BRBACKUP BRRESTORE BRARCHIVE cpio/dd parallel

Media

SAP AG 1999

Online redo log files

cpio/dd

Media

Backup Objects Objects that need to be backed up Database objects Data files

BRBACKUP Database Database backup backup Online redo log files

BRRESTORE Control file Profiles Offline redo log files

BRARCHIVE Offline Offline redo redo log log file backup file backup

Computing center data Operating system Database executables

SAP AG 1999

R/3 data R/3 interfaces SAP executables

Tape Pools

Number of parallel backup devices

Length of backup cycle

?

Database size Tape pools

BRBACKUP + 30% Reserve BRARCHIVE

Frequency of backups SAP AG 1999

Number and size of redo log files in a backup cycle

Tape Initialization Profile init.sap contains the tape names: ... volume_backup = (B01,B02... volume_archive = (A01,A02... ...

Write the label to the tape that also contains the tape name

.tape.hdr0

Initialize new tapes, non-SAP tapes, or locked tapes: brbackup -i force or brarchive -i force Rename non-locked tapes: brbackup -i -v or brarchive -i -v SAP AG 1999

Tape Label Contents and Tape Checks

Tape label contents:

Tape name Database name Timestamp of last backup Number of backups performed

Tape checks:

Profile init.sap:

Tape name

Error

Lock period Use count

Warning

Show tape label contents: brbackup -i show or brarchive -i show SAP AG 1999

... expir_period tape_use_count ...

= 28 = 100

Tape Locking Physical tape locking

C11B01

C11B02

expir_period expir_period==30 30

28 days

.tape.hdr0

Days 1

2

28

29

30

31

32

33

Logical tape locking List of tapes used in expir_period

BRBACKUP BRARCHIVE

SAP AG 1999

Tables SDBAH and SDBAD

... C11B05, C11B06, ...

Profile: init.sap ... volume_backup = (C11... volume_archive = (C11... ...

Scenario 1: Automatic Tape Selection

Offline

Online Archives (2x)

28 days

Online Archives (2x)

Additional

BRBACKUP BRARCHIVE Tables SDBAH C11B01 and SDBAD

SAP AG 1999

C11A01

Tape management active, BRBACKUP finds tape names + Tape label check active = Only accepts tapes with requested names and with expired lock period

Scenario 2: Manual Tape Selection

Offline

Online

28 days

Tape lock expiration will be checked + Tape management is not active = To select any tape manually

Additional

BRBACKUP BRARCHIVE C11B01

SAP AG 1999

Archives (2x) brbackup -v SCRATCH or brarchive -v SCRATCH

C11A01

Tape with label name SCRATCH Tape lock expiration will be checked + Tape name is changed to the currently required tape name = You can use this option to replace tapes in your tape pool

Scenario 3: Tape Selection by an External Tool

Offline

Online Archives (2x)

28 days brbackup -v C11M01,C11M02

Additional

BRBACKUP BRARCHIVE C11B01 Search mechanism SAP AG 1999

C11A01

Tape management is deactivated but tape name is checked + Tape label check active = Only accepts tapes - With the specified names - With expired lock period

Tape Layout

BRBACKUP: .tape.hdr0

init.ora init.sap DB file 1 init.dba

DB file n

Control file

reorg.log struct.log

Detail log

Summary log

BRARCHIVE: .tape.hdr0

init.ora Offline init.sap init.dba redo log 1

Offline reorg.log Detail redo log n struct.log log

SAP AG 1999

Summary log

How SAP Backup Tools Work Together CCMS Planning Calendar Planning Goto Listing Help System MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

BRBACKUP

Data files

Log files .... .... .... .... .... .... .... .... ....

init.sap .... .... .... ....

sapr3.SDBAH sapr3.SDBAD

Log files - Command prompt x BRBACKUP -t ...

BRARCHIVE

or BRARCHIVE ...

SAP AG 1999

Offline redo log files

.... .... .... .... .... .... .... .... ....

Backup Profile Parameters

init.sap .... .... .... .... ?

? ??? ?

SAP AG 1999

compress compress_cmd compress_dir tape_copy_cmd exec_parallel tape_address tape_address_rew tape_address_arch tape_address_rew_arch

= hardware = “compress -b 12 -c $ > $” = /oracle//sapreorg = dd =0 = /dev/rmt/0mn = /dev/rmt/0m = /dev/rmt/1mn = /dev/rmt/1m

backup_ mode backup_ type volume_backup tape_ size tape_use_count expir_period backup_dev_type archive_function volume_archive tape_size_arch

= all = online [offline] = (B01, B02, ...) = 32G = 100 = 28 = tape = copy_delete_save = (A01, A02, ...) = 6000M

Profile init.sap Parameter tape_size

Incorrect

Correct BRBACKUP/BRARCHIVE

... cpio/dd cpio/dd

Data

... Data

Physical tape size Parameter tape_size SAP AG 1999

BRBACKUP/BRARCHIVE BRBACKUP: SAP follow-up tape

cpio/dd

Data

dd: Error

...

cpio: Error or cpio continuation volume

cpio/dd cpio/dd cpio/dd

Data

... Data

Data

Physical tape size Parameter tape_size

Hardware Compression Tape station without hardware compression

400 MB 400 MB 400 MB 400 MB 400 MB 400 MB 400 MB 400 MB

400 MB

400 MB

400 MB

BRBACKUP

400 MB

init.sap

compress = no tape_size = 1800M

Tape station with hardware compression

400 MB 400 MB 400 MB 400 MB 400 MB 400 MB 400 MB 400 MB

BRBACKUP

400 MB

400 MB

400 MB

200 MB 200 200 200 200 200 200 200 200 MB MB MB MB MB MB MB MB init.sap compress = hardware .... .... .... compress_cmd = “compress -b 12 -c $ > $” $” ....

exec_parallel tape_size

= 0 = 1600M

Once per cycle: Determine compression rate SAP AG 1999

Scheduling and Performing a Normal Database Backup Schedule an Action for Tue 05

CCMS Planning Calendar Planning Goto Listing Help System MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

18:00:00

Start time 1

Period (weeks):

Calendar

Full database offline + redo log backup Full database offline backup Full database online + redo log backup Full database online backup Redo log backup Partial database offline backup Partial database online backup Check optimizer statistics Adapt next extents

SAPDBA Check database

-

x

h - Backup database -

-

Command prompt

x

BRBACKUP -t online -d tape

-m all -c -u system

SAPDBA c - Backup device type d - Objects for backup e - Backup type g - Query only

tape all online no

S - Start BRBACKUP init.sap .... .... .... ....

SAP AG 1999

x

tape_copy_cmd = dd or tape_copy_cmd = cpio

Phases of a Whole Database Backup

Online

Save Save control control file file to to disk disk

*Start *Start database database

Retrieve Retrieve file file names names of of data data and and online online redo redo log log files files from from database database and and retrieve retrieve names names of of control control files files from from init.ora init.ora Shut Shut down down database database Back Back up up tape tape header, header, init.sap, init.sap, init.dba, init.dba, and and init.ora init.ora For For all all tablespaces tablespaces to to be be backed backed up: up: Begin tablespace backup mode Begin tablespace backup mode Back Back up up tablespace tablespace data data files files End End tablespace tablespace backup backup mode mode

Back Back up up data data files files *Back *Back up up online online redo redo log log files files

Back Back up up saved saved control control file file

Back Back up up control control file file

Log Log file file switch switch

Start Start database database

Back Back up up reorg.log, reorg.log, struct.log, struct.log, detail detail log, log, and and summary summary log log init.sap .... .... .... ....

SAP AG 1999

tape_copy_cmd = dd or tape_copy_cmd = cpio

*Shut *Shut down down database database

Offline

Logical Verification of a Database Backup BRBACKUP -c -w use_dbv

? =

Data files

A B

..

N

compress_dir

Corruption

A

File length

Oracle 8-KB Block

Log files .... .... .... .... .... .... .... .... ....

SAP AG 1999

Tape readable?

Once per week. Minimum: once per cycle

?

Physical Verification of Offline Database Backups BRBACKUP -t offline -c -w Data files

A B

..

N

? =

Offline (binary)

Check database backup SAP AG 1999

compress_dir

A

...

If possible: Once each cycle

Monitoring a Database Backup CCMS Planning Calendar Planning Goto Listing Help System MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

Database Operations Monitor (DB24)

CCMS

MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

sapr3.SDBAH sapr3.SDBAD

CCMS Calendar (DB13)

SAPDBA

-

x

L - Show/Cleanup SAPDBA

x

Log files

a - Show log files / profiles -

SAPDBA e - BRBACKUP log files

-

Command prompt

x

cd /oracle//sapbackup cat back.log | more cat b.and | more

SAP AG 1999

x

.... .... .... .... .... .... .... .... ....

Offline Redo Log Files: Status and Option -cds Status of an offline redo log file

BRARCHIVE option -cds (copy, delete, save)

BRARCHIVE -cds

4 2

Mon

Tue

42

42

Wed 42 A01

ARCHIVED 43

4 2

SAVED

43

4 A01 2

42 44

4 2

COPIED

42

DELETED

../saparch

SAP AG 1999

43

44

A02

43 44

4 A02 2

4 5 4 6

../saparch

44 A03

45 46

Performing Offline Redo Log File Backups Schedule an Action for Tue 05 CCMS Planning Calendar Planning Goto Listing Help System

18:00:00

Start time

MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

Full database offline + redo log backup

MON TUE WED THU

FRI

SAT SUN

Full database offline backup

MON TUE WED THU

FRI

SAT SUN

DB13

Period (weeks):

1

Calendar

Full database online + redo log backup Full database online backup Redo log backup Partial database offline backup Partial database online backup Check optimizer statistics Adapt next extents

-Check database SAPDBA

x

i - Back up offline redo log files -

SAPDBA

a - Archive function c - Archive device type

-

Command prompt

x s - Start BRARCHIVE

brarchive -cds -d tape -c -u system

SAP AG 1999

Copy, delete, and save archive logs tape

x

Monitoring Offline Redo Log File Backups CCMS Planning Calendar Planning Goto Listing Help System MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

MON TUE WED THU

FRI

SAT SUN

DB13

Database Operations Monitor (DB24) sapr3.SDBAH sapr3.SDBAD

CCMS Calendar (DB13)

SAPDBA

-

x

L - Show / Cleanup SAPDBA

Log files

x

a - Show log files / profiles -

SAPDBA f - BRARCHIVE log files

-

Command prompt

x

cd /oracle//sapbackup cat arch.log | more cat a.cds | more

SAP AG 1999

x

.... .... .... .... .... .... .... .... ....

Log File Cleanup SAPDBA

-

x

L - Show/Cleanup SAPDBA

x

b - Cleanup log files / directories x SAPDBA a - SAPDBA log files and dump directories b - SAPDBA daily check log files c - BRBACKUP log files d - BRARCHIVE log files f - ORACLE traces and audits

init.dba .... .... .... ....

expir_period_SAPDBA_normal expir_period_daily_check expir_period_BRBACKUP expir_period_BRARCHIVE expir_period_oracle_trace

= = = = =

.... .... .... .... .... .... .... .... ....

-

Command prompt

delete b.and

SAP AG 1999

sapr3.SDBAH sapr3.SDBAD

Log files

11 5 30 30 1

x

Freespace Problems in Directory saparch Monitoring the directory saparch

Detecting an archiver stuck - .…

RZ20

SAPGUI .… ....

__________ __________ __________ __________

x

alert_.log: Thread cannot allocate new log, All online logs needed archiving

DB24 Resolving an archiver stuck

SAPDBA

-

x

f - Archive mode SAPDBA

x

c - Show all archive information SAPDBA FREE SPACE : ...

x

Dummy

Remove dummy files

and Run BRARCHIVE -

Command prompt

x

cd saparch df-k.

SAP AG 1999

../saparch

A01

Database Errors

Error types: Statement Process Instance User Media

SAP AG 1999

“Most importantly, be prepared for disasters. Don’t think you will never see a failure. Every DBA will experience a database failure. It’s just a matter of when... Good luck.” Rama Velpuri

How to Handle Problems

Do not make any rash decisions Analyze the problem in detail Create a problem-solving strategy Before restoring any files, check: What is causing the problem Whether there is enough disk space to save and restore files Whether a hardware extension is necessary The file system and mount points The availability of backups The availability of offline redo log files

SAP AG 1999

Scenario

10

3 38 38 8

10

10

9

10

11

12

...

36

Offline and online redo log files

3 8

37

Control files

38

10 1 0

38

10

Data files

38

Full backup Legend: Log sequence number SAP AG 1999

9

Partial Restore and Complete Recovery

mount

38

open

38

38

38

Control files

38

10

11

38

...

3 8

37

39

Complete recovery 38

10

38

Partial restore

...

Offline and online redo log files

38

10

SAP AG 1999

38

Data files

Database Reset

open

10

10

Control files

10

10

11

...

10

11

...

37

9

38

Offline and online redo log files

10

Data files

10 10

Full restore

SAP AG 1999

Point in Time Recovery

mount

10

Open resetlogs

10

25

25

25

10

10

11

...

1

1

Control files

1

26

...

1

2

25

Incomplete recovery 25 25

Full restore

SAP AG 1999

24

25

10

...

Offline and online redo log files

24

10

10

37

1 1

1

Data files

Partial Restore and Complete Recovery (1)

Detail logs

arch.log

back.log

Check Database

Find backups

Restore data files

Find offline redo log files

Recovery script

Restore offline redo log files

saparch

Database SAP AG 1999

Recover database

Partial Restore and Complete Recovery (2)

Detail logs

arch.log

back.log

Check Database

Find backups

Restore backup files

Find offline redo log files

Recovery script

Restore offline redo log files

saparch

Database SAP AG 1999

Recover database

Partial Restore and Complete Recovery Limitations

Problem

Solutions

No data or offline redo log file backups available

Perform a database reset Perform a point in time recovery

Logs

No BRBACKUP/BRARCHIVE logs available

Use the SAPDBA function Restore individual files

init.*

No init.* files available

Restore these files from tape using command brrestore -n init_ora

Control files damaged

Online redo log files damaged

SAP AG 1999

Copy one of its mirrors

See R/3 Online Help

Database Reset Using a Full Offline Backup

Detail logs

back.log

Find full offline backups

Save current online redo log files and control file

Overwrite all data files, control files, and online redo log files

Database mount

Database open sapreorg

Database SAP AG 1999

Database Reset Using a Consistent Online Backup

Detail logs Recovery script back.log

Find Online_Cons backups

Save online redo log files and control file

Overwrite all Data files and Offline control files redo log files

sapreorg

saparch

Database SAP AG 1999

Recover database until cancel

Database open resetlogs

Full Restore and Recovery (1)

Detail logs

back.log

Find full offline/ online backups

Input: time

arch.log

reorg.log

Recover until?

Find offline redo log files

Status: allowed?

Not allowed if (for example): - No backup specified - No offline redo log files found - Recovery over tablespace reorg - Backup before open resetlogs

Database SAP AG 1999

Full Restore and Recovery (2)

Recovery script

Save online redo log files and control file

sapreorg

Overwrite all Data files and control files Offline redo log files (if necessary)

saparch

Database SAP AG 1999

Recover database (until time)

Database open (resetlogs)

Backup Requirements and Costs

Duration of a backup Time window for a recovery High availability Training Administrative workload Acquisition costs

SAP AG 1999

BRBACKUP and BRARCHIVE: One-Run Strategy Database backup

Offline redo log file backup

Offline redo log files in saparch

Data files

brbackup -m all -c -a -cds -c

Tape init header files SAP AG 1999

...

...

Logs

Consistent Online Backups

...begin backup

10

...end backup

10

25

10

9

25

Control files

25

10

...

11

10

Offline redo log files in saparch

...

25

24

25

...

10 10

Data files

25 25

brbackup -t online_cons

init files

10

10

...

25

10

...

25

Logs

Legend: Log sequence number SAP AG 1999

9

Parallel Tape Support

Control files

Offline redo log files in saparch

init.sap exec_parallel tape_address archive_function tape_address_arch

=0 = (dev1, dev2, dev3) = double_save_delete = (dev4, dev5)

BRARCHIVE

BRBACKUP

DDS DLT

DLT SAP AG 1999

DLT

DDS

Partial Database Backups

Mon

Tues

Sun

...

brbackup -m PSAPSTABD BRARCHIVE

SAP AG 1999

brbackup -m PSAPBTABD BRARCHIVE

Data files

brbackup -m all -f 7 BRARCHIVE

Backing Up Data Tablespaces Only

System, roll, and temp tablespace

brbackup -m all_data

SAP AG 1999

Data tablespaces

Pure index tablespaces

Two-Step Disk Backup

Data files

dir2

Fast 1st step brbackup -d disk -e 4

init.sap exec_parallel tape_address backup_root_dir

SAP AG 1999

=0 = (dev1, dev2, dev3) = (lvol1, lvol2)

dir1

“Slow” 2nd step brbackup -b last -d tape

Structure-Retaining Database Copy

$ORACLE_HOME

NEW_DB_HOME dbs

dbs sapdata1

btabd_1

. . .

sapdata

sapdata1

btabd_1

. . .

brbackup -d disk_copy

sapdata origlogA

origlogA origlogB mirrlogA

origlogB mirrlogA

mirrlogB

mirrlogB sapbackup

sapbackup saparch sapcheck

init.sap sapreorg

saptrace

SAP AG 1999

new_db_home = /oracle/NEW

Split Mirror Disk Backups

Production server

M

o irr

r

Backup server

1. Split mirror 2. Mount 3. Backup 4. Unmount 5. Resync brbackup -t online/offline_split -d tape

SAP AG 1999

SAP Tools and the Oracle Standby Database

Production server

Standby server

Database open

Database mounted standby or offline for backup

saparch

saparch

Recovery

ftp ftp OR NFS NFS sapr3.SDBAH sapr3.SDBAD

SAP AG 1999

brarchive -sd

-d disk -f -w

brarchive -ssd -f -m 60

brbackup -t offline_standby

External Backup Tools Using BC-BRI

init.sap

SAPDBA brbackup

backup_dev_type util_par_file

brarchive Backup

= util_file_online = init.utl

brrestore Inquire

BACKINT

External backup server

Restore init.utl

$ORACLE_HOME brbackup init.sap sapdata1 . . .

sapdata saparch

SAP AG 1999

btabd_1

Space Management: Review Table ABC Tablespace

Segment (table/index)

Index XYZ Table DEF Datafile2

Datafile1 Segment 192K

Segment 224K

Segment 192K Extent 48K 1

Initial Extent

Segment 80K

...

Segment 192K Segment 640K

...

...

Next Extent

Extent 144K

8K

8K2 8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

8K

...

Disks File system

Data block

SAP AG 1999

Segment 256K

Datafile1

Datafile2

Space Management: Fragmentation Types

Objects (tables/indexes)

Tablespace

.data1 External fragmentation

2

Critical object

1

4

2

0 Oracle block

1 2 3 4 5 ...

5

4

Internal fragmentation

0

2 0

3

3 2

4

3

5

1

1

Free 0

0

Used Extents

SAP AG 1999

1

Gaps

Daily Monitoring: sapdba -check R/3 CCMS Planning Calendar Planning Goto Listing Help System

Schedule an Action for Tue 05

x

MON TUE WED THU CheckDB CheckDB CheckDB CheckDB

FRI

SAT CheckDB

MON TUE WED THU CheckDB CheckDB CheckDB CheckDB

FRI

SAT CheckDB

MON TUE WED THU CheckDB CheckDB CheckDB CheckDB

FRI

SAT CheckDB

MON TUE WED THU CheckDB CheckDB CheckDB CheckDB

FRI

SAT CheckDB

SUN sapdba -next SUN sapdba -next SUN

Start time

18:00:00

Period (weeks):

1

Calendar

Full database offline + redo log backup Full database offline backup Full database online + redo log backup Full database online backup Redo log backup Partial database offline backup

SUN sapdba -next

Partial database online backup

Database table Update optimizer statistics DBMSGORA Check optimizer statistics

in /oracle//sapcheck

Adapt next extents Check database

sapdba -check

Start immediately

Database Check: Overview of Results Start check

Configure check

Database operations monitor

History

Database table Standard DBMSGORA

Check results

Settings

Warnings

6

Refresh every

10

seconds

Errors

5

Delete after

100 days

Total

11

View the last

10 days

inaktiv aktiv

DB16

History: All messages Result

W E E E

Date

08/21/1999 08/21/1999 08/21/1999 08/21/1999 SAP AG 1999

Time

22:00:00 22:00:00 22:00:00 22:00:00

Days Error type

10 10 10 10

PROF DBO DBO DBA

Error name LOG_SMALL_EN... OPT ALY NO_OPT_STATS

Text LOG_SMALL_ENTRY_MAX_SIZE DB Operation opt never started or finished successful DB Operation aly never started or finished successful 6 INDEXE(S) DDXTF 0,DDXTT 0, TATAF 0, ... WITHOU

Configuring sapdba -check DB17

Number of SAPDBA check parameters

Total Status

67 19

Profile param. Oracle alerts

28 13 7

Operations Typ

Parameter

Active

66

Inaktiv

1

O... Actv. S.. Op... Val. U... Per... Unit

DBA

ARCHIVE_STUCK

E >

DBA

CONTROL_FILE_MISSING

E

DBA

CONTROL_MIRROR

E

DBA

CRITICAL_SEGS

W >

DBA

DF_OFFLINE

E

DBA

FILE_MISMATCH

DBA

FILE_MISSING

DBA

FILE_TYPE_UNKNOWN

DBA

FS_FULL

80

Date

U... C CorrMeasure

P

P

1

E

INIT.ORA

CONTROL FILE(S) ARE NOT...

P

SAPDBA: TABLESPACE A...

SEGMENT(S) #1 WOULD CAU... FILE TYPE DOES NOT MATC...

SAPDBA: RESTORE/RECO... DATA FILES ARE MISSING FILE TYPE ( DATAFILE , RA... 01 / 22 / 1999 CA...

Actv.

Yes

Condition

Error

Description

ARCHIVE STUCK - FS SPACE #1 MORE THAN #2 FULL

if

E

DO A “CREATE CONTROL...

P

Database analysis tool (SAPDBA) E ARCHIVE_STUCK W > 95 P

Object

ARCHIVE STUCK - FS SPAC...

DATAFILE(S) #1 OFFLINE

E

Parameter

SAPDBA: BACKUP ARCH...

CONTROL FILES ARE MISSI...

Change Database Check Parameter

Type

Description

greater than (old

80

EXTEND FILESYSTEM OR...

# 1 FILE SYSTEM(S) # 2 ARE...

Percentage

Repeat period Duration

Time Unit

Configuration table DBCHECKORA

Corrective measure Type

Program

Operation

SAPDBA: BACKUP ARCHIVE LOGS

Changed by User

SAP AG 1999

Date

Tablespace Extension ADD A DATA FILE: File size depends on the estimated increase of the tablespace objects. Check for the number of data files in the database.

RESIZE THE DATA FILE: Extend the size of the data file depending on the space available on the file system and size of critical object.

New file .data1

.data1

.data2

Critical object

Critical object

OR

Extents

Original size

Back up extended tablespace and control files

SAP AG 1999

After Resize

Storage Categories of SAP Database Objects R/3

ABAP Dictionary: Display technical settings

Name Short text Last changed Status

x

ZPROGRAM Transparent Table Test Table for technical settings TND 24/08/1999 Active Saved

Logical storage parameters Data class APPL1 Size category 4

Transaction data. transparent tables Data records expected: 39.000 to 3.100.000

Table TGORA (storage parameters for R/3 tables) Category INIT NEXT MINEXTENT MAXEXTENT 0 16 40 1 300 1 16 160 1 300 2 16 640 1 300 3 16 2560 1 300 4 16 10240 1 300 5 16 20480 1 300 6 16 40960 1 300 7 16 81920 1 300 8 16 163840 1 300 9 16 327680 1 300 10 16 655360 1 150 11 16 1310720 1 150 12 16 2621440 1 150 13 16 5242880 1 150 14 16 10485760 1 150 SAP AG 1999

Table IGORA (storage parameters for R/3 indexes) Category INIT 0 16 1 16 80 16 160 1 640 1 300 1 300 5 300 6 16 7 16 20480 1640960 1 81920 1 300 1 150 11 3276801 150 6553601 150 1310720 1 2621440 1

NEXT MINEXTENT MAXEXTENT 40 1 300 1 300 2 300 3 16 4 16 2560 16 5120 1 10240 1 300 1 300 8 300 9 16 10 16 163840 16 12 16 13 16 150 14 16 150

Using sapdba -next Example sapdba -next: Table size: 900 MB 10 % value: 90000KB Current NEXT: 20480KB NEXT candidate: 90000KB

R/3

TGORA values (KB)

x

CCMS Planning Calendar

Planning Goto Listing Help System MON

TUE

WED

THU

FRI

SAT

10485760 5242880 MON

TUE

WED

THU

FRI

SAT

Schedule an Action for Tue 05

1310720

New value for NEXT: 163840

MON

655360

TUE

WED

327680 MON

163840

THU

1

Full database offline + redo log backup

TUE

WED

THU

Full database offline backup

81920

40960

Partial database offline backup

Current NEXT:

20480

20480

Database table Update optimizer statistics DBMSGORA

10240 2560 640 160 40

SAP AG 1999

Redo log backup Partial database online backup Check optimizer statistics Adapt next extents Check database

2560

FRI

Full database online + redo log backup Full database online backup

90000

Technical settings: Category 3:

SAT

Start immediately

SUN

sapdba -next

Calendar

NEXT candidate: Next smaller TGORA value:

81920

FRI

18:00:00

Start time Period (weeks):

163840

SUN

sapdba -next

2621440

Next larger TGORA value:

SUN

sapdba -next

SAT

SUN

sapdba -next

Daily Monitoring: Tables and Indexes

Database System

Database ORACLE Name TCC

Date/time of this analysis 08/24/1999 07:01:30 Refresh Refresh

Checks Checks

Space Space statistics statistics

Tablespaces

Total number 27 Total size/kb 12.123.016 Total free/kb 3.050.320 Minimum free/kb 4.024 Max. autoextensible/kb AutoExtend off Tables and indexes

Total number Total size/kb More than 1 extent Missing in database Missing in R/3 DDIC Space-critical objects

Current Current sizes sizes

25%

Tables Indexes 13.064 15.305 5.915.664 2.979.432 1.082 1.591 0 1 0 0 0 0 Maintain table: TCOLL

Space Space statistics statistics Freespace Freespace statistics statistics

Detailed Detailed analysis analysis Missing Missing indexes indexes Space Space critical critical objects objects Space Space statistics statistics

Day Time M T W T F S S 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 ------------------------------------------------------------------------------RSDBPREV 1 C X:X:X:X:X:X:X X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X:X RSORATDB 1

SAP AG 1999

C X:X:X:X:X:X:X

: : : : : : :X: : : : : : : : : : :X: : : : :

Tables and Indexes: Important Reports

Critical growth of tables/indexes in the last 4 weeks 09/05/1999 10:40:26 DB_SERVER Critical table/index growth Intervals: 08/06/1999 - 09/05/1999 Table Name ATAB~0 SMENINTT~ SKAT~001

Type INDEX INDEX INDEX

Size(KB) Total 72.720 2.416 9.960

Measurements:

NextExtSize Growth (Kbyte) 10.880 160 2.400 80 2.080 80

29

Extents Total Growth 168 68 31 30 89 26

MaxExtents defined %used 300 56 300 10 300 30

FirstExtent (Kbyte) 45.936 16 2.904

Tablespace PSAPPOOLI PSAPSTABI PSAPSTABI

Table Space History 09/05/1999 10:40:26 DB_SERVER Intervals: 08/06/1999 - 09/05/1999 Measurements: 30 Scale: Day Scale: Day Size (KB) Free(KB) Used (KB) %-Used Tablespace Total Chg/day Total Total Chg/day Total Chg PSAPPOOLD 747.512 0 38.944 708.568 2.415 94 0 PSAPPOOLI 563.200 0 97.888 465.312 1.793 82 0 PSAPSTABI 921.600 0 70.288 851.312 1.909 92 0 PSAPSTABD 1.013.744 0 31.024 982.720 3.384 96 0 PSAPROLL 307.200 0 182.392 124.808 3.084 40 1 PSAPBTABD 735.216 0 231.336 503.880 1.249 68 0 PSAPBTABI 409.600 0 125.768 283.832 157 69 0 PSAPSOURCED 102.400 0 69.296 33.104 6 32 0 PSAPSOURCEI 102.400 0 47.520 54.880 3 53 0 PSAPTEMP 307.200 0 307.192 8 36 0 0 PSAPUSER1D 8.192 0 4.024 4.168 36 50 0 PSAPPROTI 33.792 0 16.592 17.200 52 50 0

SAP AG 1999

Tables/Indexes Total Chg/day 6.619 0 6.751 0 4.404 0 3.325 0 15 0 2.344 0 3.222 0 47 0 54 0 0 0 4 0 114 0

Extents Total Chg/day 8.692 22 9.296 21 6.156 10 4.307 8 120 3 2.471 1 3.542 1 49 0 60 0 0 0 4 0 131 0

Analyzing Internal Fragmentation sapdba -analyze DB02 >> Detail Analysis

SAPDBA Detail Log

Detail log: 9908221023.aly *********************************************************************** (12690 tables analyzed - sorted by empty space in descending order) TABLESPACE_NAME TABLE_NAME EMPTY(kb) NEVER_BEEN_USED(kb) USED(kb)

Detailed analysis for Index ATAB~0 PSAPSTABD E071K 42032 42032 5128 Data from DBA_SEGMENTS/DBA_INDEXES PSAPEL40AD D010L 31112 31112 3168 Space PSAPPOOLD ATAB 17112 17112 58368 72.720 PSAPES40AD DOKCLU 13136 13136 323448 Allocated space..Kbyte blocks...... 9.090 PSAPES40AD D010S 10088 10088 774848 *********************************************************************** extents..... 168 CHARTS OF 20 EMPTY INDEXES - USING: VALIDATE STRUCTURE (6751 indexes analyzed. sorted by empty space within BTREE = Block structure USED_BY_BTREE - USED) 8.192 TABLESPACE_NAME INDEX_NAME TOTAL(kb) USED_BY_BTREE(kb) USED(kb) Blocksize.........byte PSAPPOOLI PSAPPOOLI PSAPPOOLI PSAPPOOLI

ATAB~0 T512T~0 RTXTL~0 T800Y~0

SAP AG 1999

72720 7280 11920 8800

72648 7200 10072 8744

50563 4641 7745 6726

Pct_free.............. Transactions initial.. maximum.. Header minimum....byte Data maximum......byte

10 2 255 159 7.230

Process freelists..... Freelist groups.......

1 1

Reorganization: Basics

Example for a table reorganization

Example for an index recreation

1) Export table

1) Drop index Data_1 Data_1

2 1 0

2

2 1 0

1 0

2 1 0

2) Recreate index sapreorg 2) Drop table

Additional storage space required

PSAPTEMP

Data_1 Data_1

0 2

3) Import table Recreate index SAP AG 1999

1 0

0 2 1 0

Reorganization: Reasons

Disk accesses [%]

0

1 2 3 4 5 ...

Oracle block Disk_1 2

Disk_2 3

2

2

3

0

4 1 0

1

2

3

0

3

4 1 0

0

3

4 1 0

1

2

3

0

3

4 1 0

1

2

3

0

4 1 0

2

1

2

1

2

3

0

3

4 1 0

4 1 0

Free

5 1

2

3

Internal fragmentation

0

5 1

3

0

0 5

3 0

5

2

0 5

Disk_4

3 0

5

2

0 2

2

0 5

2

Disk_3 3

0

2 0

3

4 1 0

5 1

Disk "hot spots" Disk

SAPDBA Detail Log Detail log: 9909050715.aly CHARTS OF 20 EMPTY INDEXES - USING: VALIDATE STRUCTURE (6751 indexes analyzed. sorted by empty space within BTREE = USED_BY_BTREE - USED) TABLESPACE_NAME INDEX_NAME TOTAL(kb) USED_BY_BTREE(kb) USED(kb) PSAPPOOLI PSAPPOOLI PSAPPOOLI PSAPPOOLI PSAPPOOLI PSAPPOOLI SAP AG 1999

ATAB~0 T512T~0 RTXTL~0 T800Y~0 T52C5~0 USOBX_C~0

72720 7280 11920 8800 5480 4160

72648 7200 10072 8744 5424 4032

20563 4641 7745 6726 3760 2627

Fragmented Indexes

Used

Reorganization: Phases and Types Phases Create script and restart file Check the free space Perform a reorganization

Directories sapreorg//.. sapreorg, PSAPTEMP. PSAPROLL, objects tablespace

Reorganization of a single object or a list of objects

Reorganization of a tablespace

Data file_1

Data file_1

2

0 2 1 0

2 0

6

3

2

3

4

0

6

5

1

1

0

1

2

3

4 1 0

Data file_1 2

2

5 0

Disk "hot spots" SAP AG 1999

3

4 1 0

0

4

Data file_new

4 1 0

2 5 1

Data file_2 8

5 1

3

0

4 3 0

1

0

5

0

0 1 2

0 0

7

11

Reorganization of a tablespace with data files

Disk y 4 3

7 9 10

Internal and external fragmentation Fragmented indexes

Moving / Renaming data files

1 2

Data file_2

0

Disk "hot spots" Fragmented indexes Internal fragmentation

Disk x

Data file_1 0

8

0

2 1 0

1

Data file_2

Data file_1

6

9

1

4

0 0

10 7

0 0

11 5

0

0

Reorganization: Methods Reorganization Reorganizationof ofindexes indexes Method Runtime Method Runtime Security Security Additional Additionalspace space Parallel Parallel Alter ++ ++ Objects II Alterindex index ++ ++ Objectstablespace tablespace rebuild PSAPTEMP rebuild PSAPTEMP Index Index recreate recreate

++

++

PSAPTEMP PSAPTEMP

II

Reorganization Reorganizationofoftables tables Method Runtime Method Runtime Security Security Additional Additionalspace space R3Chop R3Chop Parallel ParallelCompress Compress Restrictions Restrictions Create ++ ++ Objects TT NA No Createtable table ++ ++ Objectstablespace tablespace NA NA NA Notables tableswith with as select PSAPROLL LONG/RAW as select PSAPROLL LONG/RAWfields fields SAPDBA ++ sapreorg NO PP NO export SAPDBAunload. unload. ++ sapreorg NO NO export 1/2 DBWR scan depth?

No 4.1 Scan LRU for dirty blocks

No

4. Full write batch in the dirty queue ?

Yes 4.2 Stop if dirty queue has a write batch or max scan depth reached

5. Release LRU latch

6. Write batch of dirty blocks to the data files

SAP AG 1999

Yes

No

8. New signals?

7. Move blocks from dirty queue to tail of LRU

Synchronous I/O Versus Asynchronous I/O

Read/Write without Async I/O

Read/Write with Async I/O

20 ms

1. I/O request Wait for completion

20 ms

20 ms

Generate I/O requests 1-4 in parallel

20 ms

2. I/O request 20 ms Wait for completion

20 ms

Wait for completion 20 ms

3. I/O request Wait for completion

20 ms

20 ms Generate I/O requests 5-8 in parallel

4. I/O request 20 ms

Wait for completion Completion of 4 I/O Requests:

SAP AG 1999

80 ms

Completion of 8 I/O Requests:

20 ms 20 ms 40 ms

DBWR Processes

DB_WRITER_PROCESS 1

DB_WRITER_PROCESS 2

Data files

DB_WRITER_PROCESS n

Buffer cache

SAP AG 1999

LRU latches

DBWR I/O Slaves

Slave 1

DB_WRITER PROCESS

Slave 2

Slave n

Buffer cache

SAP AG 1999

LRU latches

Data files

Monitoring I/O problems of the DBWR

Before tuning the DBWR analyze view V$SYSTEM_EVENT for I/O problems causing a slow DBWR For details refer to unit Monitoring File System Requests write writecomplete completewaits waits free freebuffer bufferwaits waits ----------------------Should Should not nothave have any anymajor major impact impact

SAP AG 1999

Monitoring the DBWR

WHERE name like 'summed dirty%’ or name like 'write requests’

Query V$SYSSTAT with RSORAVDV

Summed dirty queue length / write requests < 1 WHERE name like 'free buffer%' or name like 'dirty buffers%' Free buffers requested / dirty buffers inspected < 1 % Free buffers requested / free buffers inspected < 1 %

SAP AG 1999

Rules and Thresholds: DBWR

Check first for I/O problems causing a slow DBWR DBWR problem if Dirty buffers inspected / free buffers requested > 1% Free buffers inspected / free buffers requested > 1% Summed dirty queue length / write requests > 1

Possible reasons: I/O problems DB_BLOCK_BUFFERS too small Throughput I/O subsystem (expensive statements, DBWR problems, ...)

Possible solutions: Tune I/O, tune expensive SQL statements, use async I/O if possible Increase DB_BLOCK_BUFFERS if possible Use multiple database writer processes

In any case don’t change any parameterization without detailed problem analysis

SAP AG 1999

Checkpoint

Start Checkpoint 1 DBWR CHKPT

Stop Checkpoint 1 Start Checkpoint 2 Stop Checkpoint 2 Start Checkpoint 3

Stop Checkpoint 3 Start Checkpoint 4

SAP AG 1999

Monitoring Checkpoints

Query V$SYSSTAT with RSORAVDV

WHERE name like 'background checkpoint%'

background checkpoints started = background checkpoints completed Alert file:

Thread 1 advanced to log sequence 248 Current log# 2 seq# 248 mem# 0: /oracle/PRD/orig_logA/redologs02.log Thread 1 cannot allocate new log, sequence 249 Checkpoint not complete -> Checkpoint not complete -> System is inaccessible

SAP AG 1999

Rules and Thresholds: Checkpoint

Check first for I/O problems causing a slow DBWR Checkpoint problem if Background checkpoints started = background checkpoints completed Checkpoint not complete in the alert log file

Possible reasons: I/O problems DB_BLOCK_BUFFERS too small SAP Recommendation ififproblems: Throughput I/O subsystem (expensive statements, DBWR problems, ...) SAP Recommendation problems:

Possible solutions:

LOG_CHECKPOINT_TO_ALERT LOG_CHECKPOINT_TO_ALERT==TRUE TRUE

Tune I/O, tune expensive SQL statements, use async I/O if possible Increase DB_BLOCK_CHECKPOINT_BATCH Increase DB_BLOCK_BUFFERS if possible Increase DB_FILE_ SIMULTANEOUS_WRITES Use multiple database writer processes In case of Checkpoint not complete increase the number of Online Redo Log Groups or increase the size of the existing groups but only if the time between two Online Redo Log switches is lower the 1 minute

In any case don’t change any parameterization without detailed problem analysis

SAP AG 1999

Checkpoint Analysis in the Alert File

Time stamp starting checkpoint Logswitch Information Time stamp checkpoint completed

SAP AG 1999

Exercise: Monitor the DBWR

Query V$SYSSTAT with RSORAVDV

SAP AG 1999

Exercise: Monitor the DBWR

Name Background checkpoints completed Background checkpoints started Dirty buffers inspected Free buffer inspected Free buffer requested Summed dirty queue length Write requests

SAP AG 1999

Value

Ratios

Oracle Database: Possible Errors

When running multiple database writer (DBWR) slaves your system crashes and you receive the following error: ORA-07404: sfareq: Timeout occurred waiting for request to complete Because: The master database writer timed out waiting for a write or close to complete. One of the database writers may have stopped running. Action: Check all database writer trace files. Shut down the database and try to warm start.

SAP AG 1999

Log Writer Fundamentals

REDO Log Buffer Change vector for RB-Block LGWR

Change vector for Data-Block

LGWR

OrigLog

MirrorLog

Online REDO log files SAP AG 1999

LGWR: Normal Operation Versus Online Backup

I/O request DBWR

OS OS splits 1 Oracle write request to 4 I/O request

2048 Byte

OS Block

2048 Byte

OS Block

2048 Byte

OS Block

2048 Byte

OS Block

1 Oracle Block 8192 Byte = 4 OS Blocks 2048 Byte

2048 Byte I/O request DBWR

2048 Byte OS

2048 Byte 2048 Byte

SAP AG 1999

Backup tool read 2 new and 2 old blocks

LGWR Events

Each COMMIT When the REDO log buffer is 30% filled up After 3 seconds idle time When DBWR writes

SAP AG 1999

Oracle REDO Log Buffer

SAP AG 1999

Transaction ST04

REDO Buffer Size 1 MB

SAP SAPRecommendation: Recommendation: LOG_BUFFER_SIZE LOG_BUFFER_SIZE==1024000 1024000 SAP AG 1999

Monitoring I/O problems of the LGWR Before tuning the LGWR analyze view V$SYSTEM_EVENT for I/O problems causing a slow LGWR For details refer to unit Monitoring File System Requests

log logfile filesync sync Average Average = 2 Otherwise: Solution:

Too many recursive calls, transactions become slow Check first historical data Increase SHARED_POOL_SIZE

Reloads / pins 5000 Byte

Shared pool defined with SHARED_POOL_SIZE

SAP AG 1999

Monitoring Shared Pool Reserved Size

Query V$SHARED_POOL_RESERVED with ST04

NAME

FREE_SPACE MAX_USED_SIZE REQUESTS MAX_FREE_SIZE REQUEST_MISSES LAST_MISS_SIZE MAX_MISS_SIZE REQUEST_FAILURES

VALUE (NOT OPTIMIZED)

VALUE (OPTIMIZED)

8397744 714128 474 2964240 0 0 0 0

741392 2559392 101609 6152 12021 6568 6952 0

Recommended Settings for the init.ora: SHARED_POOL_RESERVED_SIZE = 10% of Shared Pool Size SHARED_POOL_RESERVED_MIN_ALLOC between 2048 and 3072

SAP AG 1999

Shared Pool: CURSOR_SPACE_FOR_TIME

?

SAP SAPRecommendation: Recommendation: Don’t Don’tchange changethe thedefault defaultvalue value(FALSE) (FALSE)

SAP AG 1999

Problems Related to the Oracle Shared Pool

ORA-04031, 00000, "out of shared memory when trying to allocate %s bytes (%s,%s)" Because: More shared memory is needed than was allocated in the shared pool. Action: Reduce your use of shared memory, or increase the amount of available shared memory by increasing the value of the init.ora parameter "shared_pool_size".

SAP AG 1999

Rollback Segments: Where Needed COMMIT Job B: Update AUFNR SET OFFEN = ‘N’ Where ...

Job A: SELECT * from AUFNR Where OFFEN = ‘J’ Shadow process A MANDT 001 001 001 …..

AUFTR OFFEN 101 J 102 J 103 J ….. ...

Rollback Image of AUFNR

Shadow process B te Crea k ac Rollb

MANDT 001 001 001 Original Table AUFNR 001 …..

SAP AG 1999

MANDT 001 001 Upd 001 ate 001 …..

AUFTR OFFEN 100 J 101 J 102 J 103 J ….. ...

AUFTR OFFEN 100 J 101 N 102 N 103 N ….. ...

Changed Table AUFNR

Rollback Segments: Fundamentals

PRS_1

PRS _2

PRS _3

PRS _4

PRS _5

PRS _6

PRS _7

Transaction is assigned to the rollback segment with the least number of active transactions

….

A

B C

SAP work process uses DML (insert, update, delete)

SAP AG 1999

PRS _8

PRS _9

PRS _n

Rollback Segments: Block Allocation

E3 T1

SAP AG 1999

T1

T2

E2 T2

T1

...

Rollback Segments: Cyclical Reuse

SAP AG 1999

E4 T1

T1

T2

E1 T1

T2

...

Rollback Segments: Space Management

SAP AG 1999

First Approach Rollback Segments

Undo header + undo block : Buffer Busy Waits / Reads * 100 !!! More memory on DB server may be needed !!!

3. Don’t tune your system for expensive sort operations Otherwise: Solution:

SAP AG 1999

Necessary resources (like memory) are wasted for few expensive sort operations Try to avoid expensive sort operations

Temporary Segments: Possible Errors

ORA-01575: time-out waiting for space management resource Because: The system timed out before it could acquire the necessary resources to do space management. Action: Wait a few minutes, then retry the operation.

SAP AG 1999