April 27, 2017 | Author: G.R.THIYAGU ; Oracle DBA | Category: N/A
Download ARCHIVE GAP DETECTION AND RESOLUTION IN DG ENVIRONMENT.pdf...
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
ARCHIVE GAP
An archive gap means, a set of archived redo logs that could not be transmitted to the Standby site from the Primary database. Mostly this problem occurs the network connectivity becomes unavailable between the Primary and Standby site. When the network is available again Data Guard resumes redo data transmission from the Primary to Standby site. Oracle DG provides 2 methods for GAP resolution. They are AUTOMATIC and FAL. Automatic Gap Resolution
&
FAL Gap Resolution
Let’s consider an extended n/w failure occurred between the Primary and Standby machines which causes the Standby is very far behind the Primary database, then an RMAN INCREMENTAL BACKUP can be used to roll the Standby database forward faster than redo log apply. When the archived logs are missing on the Standby database, simply we can ship missing logs from the Primary to Standby database; (If missing logs are very less count e.g. (below 15). We need to register all Shipped logs in the Standby database so that Gap can be resolved. In this article I will demonstrate how to resolve archive log gaps using following methods. Manually resolving a Gap (by Shipping logs from Primary to Standby) To Roll Forward a Standby database using RMAN Incremental backup. DISASTER RECOVERY CONFIGURATION
Primary Database
192.168.222.131
DEV Database Server
Standby Database
192.168.222.132
UAT Database Server
Primary database db_unique_name
crms
Standby database db_unique_name
stbycrms
MANUALLY RESOLVING GAPS # On Primary database SYS> select name, db_unique_name, database_role from v$database; NAME
DB_UNIQUE_NAME
DATABASE_ROLE
--------- ------------------------------ ---------------CRMS
crms
PRIMARY
# On Standby database SYS> select name, db_unique_name, database_role from v$database; NAME
DB_UNIQUE_NAME
DATABASE_ROLE
OPEN_MODE
--------- ------------------------------ ---------------- -----------CRMS
stbycrms
PHYSICAL STANDBY
MOUNTED
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
# On Primary database SYS> select thread#, max(sequence#) from v$archived_log group by thread#; THREAD# MAX(SEQUENCE#) ---------- -------------1 551 # On Standby database SYS> select thread#, max(sequence#) from v$archived_log where applied='YES' group by thread#; THREAD# MAX(SEQUENCE#) ---------- -------------1 551
# Standby database is waiting for SEQ# 552 SYS> select process, sequence#, status from v$managed_standby; PROCESS
SEQUENCE# STATUS
--------- ---------- -----------ARCH
551 CLOSING
ARCH
558 CLOSING
ARCH
0 CONNECTED
ARCH
0 CONNECTED
RFS
0 IDLE
RFS
0 IDLE
RFS
559 IDLE
RFS
0 IDLE
MRP0
552 WAIT_FOR_GAP
9 rows selected.
# Snippet of Standby alert log $ cd /u01/app/oracle/diag/rdbms/stbycrms/stbycrms/trace $ tail -f alert_stbycrms.log Fetching gap sequence in thread 1, gap sequence 552-556 .. ... FAL[client]: Failed to request gap sequence GAP - thread 1 sequence 552-556 DBID 1613387466 branch 913081878 FAL[client]: All defined FAL servers have been attempted.
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
DETECTING GAPS
Oracle Data Guard provide us with a simple view (v$archive_gap) to detect a gap. On Standby database SYS> select * from v$archive_gap; THREAD# LOW_SEQUENCE# HIGH_SEQUENCE# ---------- ------------- -------------1
552
556
Output of Standby database is currently missing log files from sequence# 552 to 556; the Standby database is 5 logs behind the Primary database. ORACLE NOTE: Refer BUG #10072528 V$ARCHIVE_GAP may not detect archive gap when Physical Standby is open read only. FIND ARCHIVE LOGS AT PRIMARY SERVER
After identifying a gap (as shown above), as a DBA you need to query the primary database to locate the archived redo logs on the primary database. I have configured the local archive destination on primary is LOG_ARCHIVE_DEST_1.
# On Primary database (SEQUENCE 552 to 556) SYS> select name FROM v$archived_log WHERE thread# = 1 AND dest_id = 1 AND sequence# BETWEEN 552 and 556; NAME -------------------------------------------------------------------------------------/u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_552_cnvodqq5_.arc /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_553_cnvohcy1_.arc /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_554_cnvokk6z_.arc /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_555_cnvom1l4_.arc /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/o1_mf_1_556_cnvomxkz_.arc
Copy the above redo log files to the Physical Standby database and register them using the ALTER DATABASE REGISTER LOGFILE ... SQL statement on the Physical Standby database. COPY ARCHIVELOG FILES TO STANDBY
$ ls /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/ o1_mf_1_553_cnvohcy1_.arc
o1_mf_1_555_cnvom1l4_.arc
o1_mf_1_554_cnvokk6z_.arc
o1_mf_1_556_cnvomxkz_.arc
o1_mf_1_552_cnvodqq5_.arc
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
COPY ARCHIVE LOG FILES TO STANDBY
# Example to transfer archive logs to Standby Server $ scp log_file_n.arc oracle@standby:/archive_log_location/log_file_n.arc $ scp o1_mf_1_552_cnvodqq5_.arc o1_mf_1_553_cnvohcy1_.arc o1_mf_1_554_cnvokk6z_.arc o1_mf_1_555_cnvom1l4_.arc o1_mf_1_556_cnvomxkz_.arc
[email protected]:/u01/app/oracle/ flash_recovery_area/STBYCRMS/archivelog/2016_06_01/
[email protected]'s password:
o1_mf_1_552_cnvodqq5_.arc
100%
29KB
29.0KB/s
00:00
o1_mf_1_553_cnvohcy1_.arc
100%
73KB
72.5KB/s
00:00
o1_mf_1_554_cnvokk6z_.arc
100%
179KB
179.0KB/s
00:00
o1_mf_1_555_cnvom1l4_.arc
100%
61KB
60.5KB/s
00:00
o1_mf_1_556_cnvomxkz_.arc
100%
19KB
19.0KB/s
00:00
As per above example you need to transfer all archive logs to the Standby Server. REGISTER LOGFILES AT STANDBY SERVER
SQL> alter database register logfile '/directory path/filename'; SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive log/2016_06_01/o1_mf_1_552_cnvodqq5_.arc'; Database altered. SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive log/2016_06_01/ o1_mf_1_553_cnvohcy1_.arc'; Database altered. .
SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive log/2016_06_01/o1_mf_1_554_cnvokk6z_.arc'; Database altered. SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive log/2016_06_01/o1_mf_1_555_cnvom1l4_.arc'; Database altered. SYS> alter database register logfile '/u01/app/oracle/flash_recovery_area/STBYCRMS/archive log/2016_06_01/o1_mf_1_556_cnvomxkz_.arc'; Database altered. Recovery process would start automatically or else stop the managed recovery process (MRP) and re-start it once again; that’s it!
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
WHY WE NEED TO REGISTER ARCHIVE LOGS AT STANDBY ? On Primary to Standby Server, archive logs were not transferred by the log transfer service, (but through SCP) so the managed recovery process will not have any info about these logs. Then manually transferred logs need to be registered with the managed recovery process before they applied by the log apply service.
CHECK WITH FOLLOWING QUERIES
SQL> archive log list; SQL> select max(sequence#) from v$archived_log; SQL> select sequence#, archived, applied from v$archived_log where applied='YES' order by 1; This is simple way to handle manually Resolving Gaps.
ROLL FORWARD PHYSICAL STANDBY USING RMAN INCREMENTAL BACKUP The Standby database lags far behind from the Primary database then i noticed that there is a huge sequence mismatch between Primary & Standby database. There could be some reasons. Network unavailable between Primary and Standby database. Archive logs deleted/corrupted on Primary database. In cases where a Physical Standby database is out of Sync with Primary database. In case of archive logs are missing/corrupt we have to rebuild the Standby from scratch. If the database size in terabytes again rebuilding the Standby database could be a tedious job; but we a solution to resolve this kind of issues. As a DBA you can go for an RMAN Incremental backup to sync a Physical Standby with the Primary database; using the command RMAN BACKUP INCREMENTAL FROM SCN … create a backup on the Primary database that starts at the standby database’s current SCN, which can then be used to roll the Standby database forward in time. Please assume bunch of archive logs are deleted/corrupted on the Primary database Server before they are transferring to the Standby database Server. In this case, I demonstrate an efficient way to Sync Standby with Primary (an alternative to rebuild the Standby DB)! DISASTER RECOVERY
11g DATABASE SERVER FOR PRIMARY
192.168.222.133
DB_UNIQUE_NAME
CRMS
11g DATABASE SERVER FOR STANDBY
192.168.222.134
DB_UNIQUE_NAME
STBYCRMS
Oracle Database Software Version is 11.2.0.1.0 on OEL(5.5)
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
# On Primary database SYS> select name, db_unique_name, database_role from v$database; NAME
DB_UNIQUE_NAME
DATABASE_ROLE
--------- ------------------------------ ---------------CRMS
crms
PRIMARY
# On Standby database SYS> select name, db_unique_name, database_role, open_mode from v$database; NAME
DB_UNIQUE_NAME
DATABASE_ROLE
OPEN_MODE
--------- ------------------------------ ---------------- -------------------CRMS
stbycrms
PHYSICAL STANDBY MOUNTED
# On Primary database (Capture SYS> select thread#, max(sequence#) from v$archived_log group by thread#; THREAD# MAX(SEQUENCE#) ---------- -------------1
653
# On Standby database SYS> select thread#, max(sequence#) from v$archived_log where applied='YES' group by thread#; THREAD# MAX(SEQUENCE#) ---------- -------------1
558
Note the difference, On Standby last applied SEQUENCE# 558 but on Primary SEQUENCE# 653 # Finding Archive Gap SEQ# at Standby database SYS> select * from v$archive_gap; THREAD# LOW_SEQUENCE# HIGH_SEQUENCE# ---------- ------------- -------------1
559
644
SYS> @archive_gap.sql DB_NAME
HOSTNAME
LOG_ARCHIVED LOG_APPLIED APPLIED_TIME
LOG_GAP
---------- -------------- ------------ ----------- -------------- ------CRMS
OEL5
653
558
01-JUN/04:33
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
95
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
CANCEL MANAGED RECOVERY PROCEES
# On Standby database SYS> alter database recover managed standby database cancel; Database altered.
GET LAST SCN OF THE STANDBY DATABASE
Last SCN of the STANDBY, will be used for the RMAN Incremental backup at Primary DB Server.
# On the Standby database Server (Checking last SCN of the Standby) SYS> select current_scn from v$database; CURRENT_SCN ----------3473651 # On the Primary database Server (Capture current_scn at primary) SYS> select current_scn from v$database; CURRENT_SCN ----------4538393
Note the SCN DIFFERENCE 4538393 – 3473651 = 1064742 (Now the Standby is lag behind). But I want to know how long the Standby database is far behind in terms of (hours/days). To know that I used the scn_to_timestamp function to translate the SCN to timestamp.
# On Primary database SYS> select scn_to_timestamp(4538393) from dual; SCN_TO_TIMESTAMP(4538393) --------------------------------------01-JUN-16 11.12.52.000000000 PM
# I execute following SQL
Statement
at Primary database (NOT workable at Standby)
SYS> select scn_to_timestamp(3473651) from dual; SCN_TO_TIMESTAMP(3473651) -------------------------------------------------01-JUN-16 04.32.44.000000000 AM The Standby database far behind more than 18 hours from the Primary database. NOTE: Query NOT WORKABLE in OPEN READ ONLY MODE or MOUNTED MODE.
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
CHECKING ARCHIVE LOGS AT PRIMARY
$ ls /u01/app/oracle/flash_recovery_area/CRMS/archivelog/2016_06_01/ o1_mf_1_645_cnxo02c4_.arc
o1_mf_1_647_cnxo9ndy_.arc
o1_mf_1_649_cnxoc4d6_.arc
o1_mf_1_651_cnxpl4fd_.arc
o1_mf_1_653_cnxpyh0c_.arc
o1_mf_1_646_cnxo2329_.arc
o1_mf_1_648_cnxob5cl_.arc
o1_mf_1_650_cnxp29x7_.arc
o1_mf_1_652_cnxpyd51_.arc
To Sync Standby with Primary we need archive logs SEQUENCE# 559 to SEQUENCE# 644. But these archives are NOT found at Primary Site; then only choice is RMAN Incremental backup.
# Shutdown the Standby database SYS> shutdown immediate; ...
On the Primary, take an Incremental backup from the SCN# (3473651) of the Standby database. The last recorded SCN# (3473651) of the Standby database. # Connect target as Primary database $ rman target sys/passwd@CRMSDB .. connected to target database: CRMS (DBID=1613387466) RMAN> backup incremental from SCN 3473651 database format '/u03/rman-bkp/stby_%U' TAG='FOR_STBY'; Starting backup at 02-JUN-16 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=52 device type=DISK backup will be obsolete on date 09-JUN-16 archived logs will not be kept or backed up channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00003 name=/u01/app/oracle/oradata/crms/undotbs01.dbf input datafile file number=00004 name=/u01/app/oracle/oradata/crms/users01.dbf input datafile file number=00001 name=/u01/app/oracle/oradata/crms/system01.dbf input datafile file number=00002 name=/u01/app/oracle/oradata/crms/sysaux01.dbf input datafile file number=00005 name=/u01/app/oracle/oradata/crms/example01.dbf input datafile file number=00007 name=/u01/app/oracle/oradata/crms/users02.dbf input datafile file number=00006 name=/u01/app/oracle/oradata/crms/users03.dbf input datafile file number=00008 name=/u01/app/oracle/oradata/crms/mytbs01.dbf channel ORA_DISK_1: starting piece 1 at 02-JUN-16 channel ORA_DISK_1: finished piece 1 at 02-JUN-16
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
piece handle=/u03/rman-bkp/stby_03r73ftc_1_1 tag=FOR_STBY comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:13:13 using channel ORA_DISK_1 backup will be obsolete on date 09-JUN-16 archived logs will not be kept or backed up channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set including current control file in backup set channel ORA_DISK_1: starting piece 1 at 02-JUN-16 channel ORA_DISK_1: finished piece 1 at 02-JUN-16 piece handle=/u03/rman-bkp/stby_04r73gm6_1_1 tag=FOR_STBY comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:04 Finished backup at 02-JUN-16
CREATE A NEW STANDBY CONTROLFILE
On the Primary database, create a new Standby control file. We can use (SQL*Plus or RMAN) to create a Standby control file; Anyone method is enough.
SYS> alter database create standby controlfile as '/home/oracle/stdb_ctrl.ctl'; Database altered.
Or
# Connect target as Primary & Create control file backup for Standby database RMAN> backup current controlfile for standby format '/u03/rman-bkp/stdb_ctrl.ctl'; Starting backup at 02-JUN-16 using channel ORA_DISK_1 channel ORA_DISK_1: starting full datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set including standby control file in backup set channel ORA_DISK_1: starting piece 1 at 02-JUN-16 channel ORA_DISK_1: finished piece 1 at 02-JUN-16 piece handle=/u03/rman-bkp/stdb_ctrl.ctl tag=TAG20160602T024937 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01 Finished backup at 02-JUN-16
LISTING CREATED BACKUP SETS
$ ls -l
/u03/rman-bkp/
total 4707832 -rw-r----- 1 oracle oinstall 4792238080 Jun
2 02:42 stby_03r73ftc_1_1
-rw-r----- 1 oracle oinstall
11927552 Jun
2 02:42 stby_04r73gm6_1_1
-rw-r----- 1 oracle oinstall
11927552 Jun
2 02:49 stdb_ctrl.ctl
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
TRANSFER BACKUP SETS TO STANDBY SERVER
Using OS utility to transfer these backups to the Standby Server from Primary Server.
# SCP all backup sets to the Standby Server at Primary Server $ scp *
[email protected]:/u03/bkp/
[email protected]'s password: ****** stby_03r73ftc_1_1
100% 4570MB
2.4MB/s
31:43
stby_04r73gm6_1_1
100%
11MB
5.7MB/s
00:02
stdb_ctrl.ctl
100%
11MB
2.8MB/s
00:04
VERIFY RECEIVED BACKUP SETS ON THE STANDBY SERVER
$ ls -l /u03/bkp/ total 4707832 -rw-r----- 1 oracle oinstall 4792238080 Jun
2 05:40 stby_03r73ftc_1_1
-rw-r----- 1 oracle oinstall
11927552 Jun
2 05:40 stby_04r73gm6_1_1
-rw-r----- 1 oracle oinstall
11927552 Jun
2 05:40 stdb_ctrl.ctl
Note the location of all data files & Control file(s) at the Standby Database Server. SYS> startup mount; SYS> spool datafile_names.txt SYS> set lines 100; SYS> column name format a100; SYS> show parameter control_files; SYS> select count(*) from v$datafile; SYS> select file#, name from v$datafile order by file#; SYS> spool off SYS> shut immediate; SYS> ! cat datafile_names.txt
RELOCATE EXISTING CONTROL FILES
# Renamed old control files at Standby Server $ cd /u01/app/oracle/flash_recovery_area/stbycrms/ $ mv control02.ctl /tmp/control02.ctl.bkp $ cd /u01/app/oracle/oradata/stbycrms/ $ mv control01.ctl /tmp/control01.ctl.bkp
NOTE: If you do not want rename control files, you can remove these files at OS level.
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
BRINGUP THE STANDBY INSTANCE – NOMOUNT PHASE
$ export ORACLE_SID=stbycrms $ rman target / connected to target database (not started) RMAN> startup nomount; Oracle instance started Total System Global Area Fixed Size
912306176 bytes 1340244 bytes
Variable Size
536874156 bytes
Database Buffers
369098752 bytes
Redo Buffers
4993024 bytes
# Restore Standby control files from the backup # Replacing newly created controlfile with old files RMAN> restore standby controlfile from '/u03/bkp/stdb_ctrl.ctl'; Starting restore at 02-JUN-16 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=19 device type=DISK channel ORA_DISK_1: restoring control file channel ORA_DISK_1: restore complete, elapsed time: 00:00:03 output file name=/u01/app/oracle/oradata/stbycrms/control01.ctl output file name=/u01/app/oracle/flash_recovery_area/stbycrms/control02.ctl Finished restore at 02-JUN-16
SHUTDOWN THE STANDBY DATABASE & STARTUP MOUNT
RMAN> shutdown immediate; Oracle instance shut down # Startup the database with new controlfile RMAN> startup mount; connected to target database (not started) Oracle instance started database mounted Total System Global Area .. ...
912306176 bytes
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
CATALOG THE INCREMENTAL BACKUP FILES AT STANDBY
RMAN does not know about these backup sets. We must catalog the new backup sets on the Standby database. I.e. when we register these backup files to the Standby database control file with RMAN CATALOG command, the Standby control file gets updated about these backup files. It helps to recover the Standby using these RMAN backup files. # REGISTERING BACKUP SETS RMAN> catalog start with '/u03/bkp'; Starting implicit crosscheck backup at 02-JUN-16 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=18 device type=DISK Crosschecked 3 objects Finished implicit crosscheck backup at 02-JUN-16 Starting implicit crosscheck copy at 02-JUN-16 using channel ORA_DISK_1 Crosschecked 2 objects Finished implicit crosscheck copy at 02-JUN-16 searching for all files in the recovery area cataloging files... cataloging done List of Cataloged Files ======================= File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_646_cnxof4nl_.arc File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_647_cnxocnkr_.arc File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_652_cnxpz1lz_.arc File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_645_cnxodgj5_.arc File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_651_cnxodqq5_.arc File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_650_cnxp48lh_.arc File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_649_cnxoc5oq_.arc File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_653_cnxpzpvs_.arc File Name: /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_01/o1_mf_1_648_cnxobwyc_.arc
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
searching for all files that match the pattern /u03/bkp List of Files Unknown to the Database ===================================== File Name: /u03/bkp/stby_04r73gm6_1_1 File Name: /u03/bkp/stby_03r73ftc_1_1 File Name: /u03/bkp/stdb_ctrl.ctl Do you really want to catalog the above files (enter YES or NO)? Y cataloging files... cataloging done List of Cataloged Files ======================= File Name: /u03/bkp/stby_04r73gm6_1_1 File Name: /u03/bkp/stby_03r73ftc_1_1 File Name: /u03/bkp/stdb_ctrl.ctl
APPLY INCREMENTAL BACKUP TO THE STANDBY DATABASE
The recovery process will use cataloged incremental backup sets because we have registered. This backup taken for Physical Standby DB Sync; noredo key word is required. Check here
# Recover the database with Catalog incremental backup sets RMAN> recover database noredo; Starting recover at 02-JUN-16 using channel ORA_DISK_1 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 06/02/2016 05:55:09 RMAN-06094: datafile 6 must be restored
If data files have been added to the Primary database during the archive log gap time, they were not included in the incremental backup sets. We need to register newly created files to the Standby database. We can find newly created files using CURRENT_SCN of the Standby.
# Execute the following query at Primary DB - (Last SCN of the Standby database) SYS> select file#, name from v$datafile where creation_change# > 3473651; FILE# NAME ---------- -------------------------------------------6 /u01/app/oracle/oradata/crms/users03.dbf
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
Newly created files should be restored at Standby before recovery operation.
RMAN> run {restore datafile 6;} Starting restore at 02-JUN-16 using channel ORA_DISK_1 channel ORA_DISK_1: starting datafile backup set restore channel ORA_DISK_1: specifying datafile(s) to restore from backup set channel ORA_DISK_1: restoring datafile 00006 to /u01/app/oracle/oradata/stbycrms/users03.dbf channel ORA_DISK_1: reading from backup piece /u03/bkp/stby_03r73ftc_1_1 channel ORA_DISK_1: piece handle=/u03/bkp/stby_03r73ftc_1_1 tag=FOR_STBY channel ORA_DISK_1: restored backup piece 1 channel ORA_DISK_1: restore complete, elapsed time: 00:00:08 Finished restore at 02-JUN-16
# Recover the Standby database using Incremental backup
Sets
RMAN> recover database noredo; Starting recover at 02-JUN-16 using channel ORA_DISK_1 channel ORA_DISK_1: starting incremental datafile backup set restore channel ORA_DISK_1: specifying datafile(s) to restore from backup set destination for restore of datafile 00001: /u01/app/oracle/oradata/stbycrms/system01.dbf destination for restore of datafile 00002: /u01/app/oracle/oradata/stbycrms/sysaux01.dbf destination for restore of datafile 00003: /u01/app/oracle/oradata/stbycrms/undotbs01.dbf destination for restore of datafile 00004: /u01/app/oracle/oradata/stbycrms/users01.dbf destination for restore of datafile 00005: /u01/app/oracle/oradata/stbycrms/example01.dbf destination for restore of datafile 00007: /u01/app/oracle/oradata/stbycrms/users02.dbf destination for restore of datafile 00008: /u01/app/oracle/oradata/stbycrms/mytbs01.dbf channel ORA_DISK_1: reading from backup piece /u03/bkp/stby_03r73ftc_1_1 channel ORA_DISK_1: piece handle=/u03/bkp/stby_03r73ftc_1_1 tag=FOR_STBY channel ORA_DISK_1: restored backup piece 1 channel ORA_DISK_1: restore complete, elapsed time: 00:42:18 Finished recover at 02-JUN-16
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
All changed blocks has been captured in the incremental backup and also updated at the standby database, thus brings the Standby database up to date with the primary database.
# Start MRP at Standby database SYS> alter database recover managed standby database disconnect; Database altered.
# Check CURRENT_SCN of the Standby Database & Primary database SYS> select current_scn from v$database; ... Check the SCN’s in Primary and Standby it should be close to each other. ARCHIVE LOG SEQUENCE DETAILS AT PRIMARY & STANDBY DATABASE
# Maximum generated LOG SEQUENCE# at Primary SYS> select thread#, max(sequence#) from v$archived_log group bythread#; THREAD# MAX(SEQUENCE#) ---------- -------------1
773
# Maximum archive log SEQUENCE# applied at Standby SYS> select thread#, max(sequence#) from v$archived_log where applied='YES' group by thread#; THREAD# MAX(SEQUENCE#) ---------- -------------1
773
SYS> select process, status, thread#, sequence#, block#, blocks from v$managed_standby; ...
Wow! Successfully Standby database
Synced
with Primary database.
Check alert.log and status of processes if any problems;
REFERENCE DOCS :
https://docs.oracle.com/cd/E11882_01/server.112/e41134/rman.htm#SBYDB00759 http://docs.oracle.com/cd/B19306_01/backup.102/b14191/rcmdupdb.htm#sthref955 https://web.stanford.edu/dept/itss/docs/oracle/10gR2/backup.102/b14191/rcmdupdb008.htm
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
BELOW ARE THE STEPS I FOLLOWED FOR A SUCCESSFUL RECOVERY
# Check the archive log gap SEQUENCE# SQL> select * from v$archive_gap; # Find last SCN of the Standby database SQL> select current_scn from v$database; # Cancel the recovery SQL> alter database recover managed standby database cancel; # Shutdown the Standby SQL> shutdown immediate; # Backup the Source (Primary) using from SCN RMAN> backup incremental from SCN XXXXX database format '/bkp_location/stby_U%' # Backup current control file of the Source (Primary database) RMAN> backup current controlfile for standby format '/bkp_location/stby_ctrl.ctl'; # SCP backup
Sets
to the target(Standby) Server from Primary
$ scp /bkp_location/* oracle@standby:/standby_location/ # Startup Standby database at NOMOUNT mode RMAN> startup nomount; # Restore control files from the backup RMAN> restore standby controlfile from '/standby _location'; # Shutdown the Standby database RMAN> shutdown immediate; # Startup the Standby database at MOUNT RMAN> startup mount; # Register backup sets by a process called cataloging RMAN> catalog startwith '/standby _location'; # Perform Recover RMAN> recover database noredo; # Restart MRP at standby SQL> alter database recover managed standby database disconnect from session; # Check SCN of the Primary and Standby database SQL> select current_scn from v$database; These are
Steps
I have done to roll the Standby database forward in time. That’s it!
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
METHODS OF GAP RESOLUTION
Automatic Gap Resolution FAL (Fetch Archive Log) Gap Resolution Manual Gap Resolution Roll forward using RMAN Incremental Backup (Physical Standby) AUTOMATIC GAP RESOLUTION
If connectivity is lost between the Primary and Standby databases (due to network problems), redo data being generated on the primary database cannot be sent to the Standby database. Once a connection is reestablished, the missing archived redo log files are automatically detected by Data Guard, which then automatically transmits the missing archived redo log files to the Standby databases. The Standby databases are synchronized with the Primary database, without manual intervention by the DBA. How this happens? Let’s dig into deep. Automatic Gap Resolution is performed automatically by the Log Transport Services. Basically the currently transferred Redo is compared with the last received. On Primary ARCH/LGWR to transfer archived redo logs to the Standby. On the Standby RFS process receives archive redo log file. RFS process
Compares
the
Sequence
number of currently being archived file with the sequence
number of previously received archived redo file; (if currently being archived redo file sequence# is greater than the last sequence# received plus one, there is a gap). An archived redo log file is uniquely identified by its sequence number and thread number. o1_mf_1_648_cnxob5cl_.arc
Last archived log file.
o1_mf_1_652_cnxpyd51_.arc
Currently being archived file.
Now 3 files are missing. There is a gap between both files, then RFS automatically requests the missing redo log sequence# from the Primary DB again via the ARCH-RFS Heartbeat Ping. The archiver of the Primary then retransmits these missing archived redo files. o1_mf_1_649_cnxoc4d6_.arc, o1_mf_1_650_cnxp29x7_.arc, o1_mf_1_651_cnxpl4fd_.arc This type of Gap Resolution is using the Primary database
Serving
Service
defined in log_archive_dest_n on the
this Standby database.
SYS> show parameter log_archive_dest_2; NAME
TYPE
VALUE
----------------------- ----------- -------------------------------------------log_archive_dest_2
string
Service=stby_crmsdb
LGWR ASYNC
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=stbycrms
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
The archiver process of the Primary database polls the Standby databases every minute (Referred to as heartbeat) to see if there is a gap in the sequence of archived redo logs. If a gap is detected, the ARCH process sends the missing archived redo log files to the Standby databases that reported the gap. Once a gap is resolved then the Transport Process (ARCH/LGWR) is notified about the Resolution of the Gap. FAL GAP RESOLUTION
As I said above, RFS receives an archive log on the Standby database, then the archive logs are registered in the Standby Control file with name and location. Missing log files are typically detected by the Log Apply Services (MRP) on the Standby database. If the archived redo log file is missing/corrupted for any reason (eg. it got deleted). FAL is required to resolve a Gap, to obtain a new copy of the corrupted or deleted file. WHY ESPECIALLY FAL IS REQUIRED ? Since MRP has NO direct communications (LOG TRANSPORT SERVICES) of the primary database; it must use the FAL_SERVER and FAL_CLIENT initialization parameters to resolve the gap. Both of these parameters must be set in the Standby Initialization parameter file. The FAL_CLIENT parameter is no longer required since Oracle 11g R2. In earlier releases, you set the FAL_CLIENT parameter on the Standby database. FAL_SERVER AND FAL_CLIENT
FAL_CLIENT and FAL_SERVER need to be defined in the initialization parameter file of the Standby database(s). They mainly used for GAP RESOLUTION through FAL background process. Using FAL parameters the MRP in the Physical Standby database checks automatically and also Resolving gaps at the time redo is applied. FAL_SERVER = NET_SERVICE_NAME of Primary database. From where the missing archive log(s) Should be requested. FAL_CLIENT = NET_SERVICE_NAME of Standby database. The destination where the FAL_SERVER database Should send the archived log(s). SYS> show parameter fal; NAME
# On Standby database TYPE
VALUE
------------------------------------ ----------- -----------------------------fal_client
string
STBY_CRMSDB
fal_server
string
CRMSDB
In earlier release when you set FAL_CLIENT parameter on the Standby database, the Primary database (FAL_SERVER) uses Standby database (NET SERVICE NAME) to connect the Standby DB.
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
The Parameter log_archive_dest_n on FAL_SERVER database which is pointing to the Standby database to resolve the Gap Resolution.. # On Primary database SYS> show parameter log_Archive_dest_2; NAME
TYPE
VALUE
--------------------- ----------- -----------------------------log_archive_dest_2
string
service=stby_crmsdb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=stbycrms
FAL_SERVER:
An Oracle NET SERVICE NAME (TNS-Alias or Connect Descriptor). This parameter
must be configured on the Standby database System that points from where the missing archive log(s)
Should
be requested. It’s recommended to set FAL_SERVER on the Primary DB with the
value of the Standby database NET SERVICE NAME thus helps when you do Switch over event. Once the Log Apply Services (MRP) detect an ARCHIVE GAP, then it sends a FAL Request to the FAL_SERVER; Once communication has been established with the Primary database, it passes the Sequence number of archived files (which are causing for the archive gap) to retransmit from the archiver process of the Primary database; and additionally it passes the service name defined by the FAL_CLIENT parameter to the Primary ARCH process. Fetching gap sequence in thread 1, gap sequence 1076-1077 FAL[client]: Trying FAL server: CRMSDB
An ARCH Process on the FAL_SERVER tries to pick up the requested
Sequences
database and sends them to the FAL_CLIENT. I.e. The Primary database ARCH process
from that Ships
the
requested archived logs to the remote archive destination of corresponding service name. # SNIPPET OF STANDBY ALERT.LOG RFS[54]: Opened log for thread 1 sequence 1076 dbid 1613387466 branch 913081878 Media Recovery Log /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_04/o1_mf_1_1076_co5qn3sd_.arc
RFS[55]: Opened log for thread 1 sequence 1077 dbid 1613387466 branch 913081878 Media Recovery Log /u01/app/oracle/flash_recovery_area/STBYCRMS/archivelog/2016_06_04/o1_mf_1_1077_co5qn4db_.arc
In order to successfully complete a Gap Request the requested archive log Sequence(s) must be available on the FAL_SERVER database. When you have multiple Physical Standby databases, the FAL mechanism can automatically retrieve missing archived redo log files from another Physical Standby database.
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
RESOLVING ARCHIVE GAP BETWEEN PRIMARY & STANDBY DATBASE
IF THE FAL_SERVER IS NOT ABLE TO RESOLVE THE GAP? The FAL-Request fails and a corresponding Message will be put in the ALERT.LOG of the Standby database. Example taken for some other SEQUENCE (1078 – 1081).
FAL[client]: Failed to request gap sequence GAP - thread 1 sequence 1078-1081 DBID 1613387466 branch 913081878 FAL[client]: All defined FAL servers have been attempted. -------------------------------------------------------------
Every minute the Primary database polls its Standby databases to see if there are gaps in the sequence of archived redo log files. The FAL (Client) requests to transfer archived redo log files automatically. The FAL (Server) services the FAL requests coming from the FAL Client. A separate FAL server is created for each incoming FAL client. FAL is available since Oracle 9.2.0 for Physical Standby database and Oracle 10.1.0 for Logical Standby databases. In 11g, if you not set FAL_CLIENT parameter, the Primary database will obtain service name from related LOG_ARCHIVE_DEST_n parameter. REFERENCE LINKS
http://flylib.com/books/en/1.145.1.66/1/ https://docs.oracle.com/cd/B19306_01/server.102/b14239/log_transport.htm#i1268294 Oracle Note: Data Guard Gap Detection and Resolution Possibilities [ID 1537316.1]
Oracle DBA Technology Explored by Gunasekaran , Thiyagu
