1. standby exception reporting
Yesterday morning, a sudden surge in disk space led to an abnormal oracle backup, alarm, and then came to clean up the backup files of the disk. After that, the archive log can be synchronized. However, the standby found that the mrp did not start. The background error was as follows: google, said that this situation should be re-made backup (I do not know it is). Is there any other way to repair it? Error log error message is as follows:
Completed: alter database recover managed standby database using current logfile disconnect from session
Wed May 31 10:44:48 2017
Dumping diagnostic data in directory=[cdmp_20170531104448], requested by (instance=1, osid=43411 (PR00)), summary=[incident=600239].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors with log /oracle/app/oracle/archivelogs/1_18799_940127349.dbf
MRP0: Background Media Recovery terminated with error 600
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:
ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:
ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_mrp0_43408.trc (incident=600231):
ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave 2. Using duplicate target to restore the backup
duplicate target database for standby nofilenamecheck dorecover;
Starting Duplicate Db at 05-JUN-17
release channel c1;
release channel c2;
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=4708 device type=DISK
contents of Memory Script:
{
set until scn 14430447592;
restore clone standby controlfile;
}
executing Memory Script
executing command: SET until clause
Starting restore at 05-JUN-17
using channel ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00
channel ORA_AUX_DISK_1: ORA-19870: error while restoring backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00
ORA-19505: failed to identify file "/oracle/backup/data/controlfiles/c-3391761643-20170605-00"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Here the recovery failed, google down, did not find the right way, what to do? It needs a different way of thinking to recover. duplicate can't get through this road.
3. Try to use the most original rman backup to restore and recover directly
copy control files, backup files, backup files already exist, so no need to be generated again, but control files need to be generated again, so the latest backup control files are generated on the main library:
RMAN> backup current controlfile for standby format '/home/oracle/ctlfile4.bak';
Starting backup at 05-JUN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=948 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including standby control file in backup set
channel ORA_DISK_1: starting piece 1 at 05-JUN-17
channel ORA_DISK_1: finished piece 1 at 05-JUN-17
piece handle=/home/oracle/ctlfile4.bak tag=TAG20170605T223329 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-JUN-17
Starting Control File and SPFILE Autobackup at 05-JUN-17
piece handle=/oracle/backup/data/controlfiles/c-3391761643-20170605-02 comment=NONE
Finished Control File and SPFILE Autobackup at 05-JUN-17
RMAN>
scp will copy control files and backup files to the backup Library
scp -r 2017-06-05 101.251.31.131:/oracle/
scp /home/oracle/ctlfile4.bak 101.251.31.131:/home/oracle/
The next step is to restore the standby control file and restore the control file.
RMAN> restore standby controlfile from '/home/oracle/ctlfile4.bak';
Starting restore at 05-JUN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4708 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/oracle/app/oracle/oradata/powerdes/control01.ctl
output file name=/oracle/app/oracle/fast_recovery_area/powerdes/control02.ctl
Finished restore at 05-JUN-17
RMAN>
Start to mount state
RMAN> alter database mount;
database mounted
released channel: ORA_DISK_1
RMAN
Register backup set
RMAN> catalog start with '/oracle/2017-06-05';
Starting implicit crosscheck backup at 05-JUN-17
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4709 device type=DISK
Crosschecked 147 objects
Finished implicit crosscheck backup at 05-JUN-17
Starting implicit crosscheck copy at 05-JUN-17
using channel ORA_DISK_1
Crosschecked 2 objects
Finished implicit crosscheck copy at 05-JUN-17
searching for all files in the recovery area
cataloging files...
no files cataloged
searching for all files that match the pattern /oracle/2017-06-05
List of Files Unknown to the Database
=====================================
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
File Name: /oracle/2017-06-05/rman_backup.log
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bak
File Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bak
Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done
List of Cataloged Files
=======================
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bak
File Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bak
List of Files Which Where Not Cataloged
=======================================
File Name: /oracle/2017-06-05/rman_backup.log
RMAN-07517: Reason: The file header is corrupted
RMAN>
Blogger Huangshan (mchdba), Blog Source Address: http://blog.csdn.net/mchdba/article/details/72874642 Declined to reprint.
Start restore database operation
RMAN> restore database;
Starting restore at 05-JUN-17
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /home/oradata/powerdes/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /home/oradata/powerdes/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /home/oradata/powerdes/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /home/oradata/powerdes/users01.dbf
channel ORA_DISK_1: restoring datafile 00005 to /home/oradata/powerdes/powerdesk01.dbf
channel ORA_DISK_1: restoring datafile 00006 to /home/oradata/powerdes/plas01.dbf
channel ORA_DISK_1: restoring datafile 00007 to /home/oradata/powerdes/pl01.dbf
channel ORA_DISK_1: restoring datafile 00008 to /home/oradata/powerdes/help01.dbf
channel ORA_DISK_1: restoring datafile 00009 to /home/oradata/powerdes/adobelc01.dbf
channel ORA_DISK_1: restoring datafile 00010 to /home/oradata/powerdes/sms01.dbf
channel ORA_DISK_1: restoring datafile 00011 to /home/oradata/powerdes/plcrm01.dbf
channel ORA_DISK_1: restoring datafile 00012 to /home/oradata/powerdes/powerdesk02.dbf
channel ORA_DISK_1: restoring datafile 00013 to /home/oradata/powerdes/datagm01.dbf
channel ORA_DISK_1: restoring datafile 00014 to /home/oradata/powerdes/plimp01.DBF
channel ORA_DISK_1: restoring datafile 00015 to /home/oradata/powerdes/dwetl01.DBF
channel ORA_DISK_1: restoring datafile 00016 to /home/oradata/powerdes/dw02.DBF
channel ORA_DISK_1: restoring datafile 00017 to /home/oradata/powerdes/timdba01.DBF
channel ORA_DISK_1: restoring datafile 00018 to /home/oradata/powerdes/users02.dbf
channel ORA_DISK_1: restoring datafile 00019 to /home/oradata/powerdes/system02.dbf
channel ORA_DISK_1: restoring datafile 00020 to /home/oradata/powerdes/powerdesk03.dbf
channel ORA_DISK_1: restoring datafile 00021 to /home/oradata/powerdes/timdba02.dbf
channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
channel ORA_DISK_1: piece handle=/oracle/2017-06-05/full_POWERDES_20170605_9911.bak tag=TAG20170605T033303
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 01:29:37
Finished restore at 05-JUN-17
RMAN>
Start rocover database operation
RMAN> recover database;
Starting recover at 05-JUN-17
using channel ORA_DISK_1
starting media recovery
archived log for thread 1 with sequence 20421 is already on disk as file /oracle/app/oracle/archivelogs/1_20421_940127349.dbf
archived log for thread 1 with sequence 20422 is already on disk as file /oracle/app/oracle/archivelogs/1_20422_940127349.dbf
archived log for thread 1 with sequence 20423 is already on disk as file /oracle/app/oracle/archivelogs/1_20423_940127349.dbf
archived log for thread 1 with sequence 20424 is already on disk as file /oracle/app/oracle/archivelogs/1_20424_940127349.dbf
archived log for thread 1 with sequence 20425 is already on disk as file /oracle/app/oracle/archivelogs/1_20425_940127349.dbf
archived log for thread 1 with sequence 20426 is already on disk as file /oracle/app/oracle/archivelogs/1_20426_940127349.dbf
archived log for thread 1 with sequence 20427 is already on disk as file /oracle/app/oracle/archivelogs/1_20427_940127349.dbf
archived log for thread 1 with sequence 20428 is already on disk as file /oracle/app/oracle/archivelogs/1_20428_940127349.dbf
archived log for thread 1 with sequence 20429 is already on disk as file /oracle/app/oracle/archivelogs/1_20429_940127349.dbf
archived log for thread 1 with sequence 20430 is already on disk as file /oracle/app/oracle/archivelogs/1_20430_940127349.dbf
archived log for thread 1 with sequence 20431 is already on disk as file /oracle/app/oracle/archivelogs/1_20431_940127349.dbf
archived log for thread 1 with sequence 20432 is already on disk as file /oracle/app/oracle/archivelogs/1_20432_940127349.dbf
archived log for thread 1 with sequence 20433 is already on disk as file /oracle/app/oracle/archivelogs/1_20433_940127349.dbf
archived log for thread 1 with sequence 20434 is already on disk as file /oracle/app/oracle/archivelogs/1_20434_940127349.dbf
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20357
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20358
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20359
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20360
channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
channel ORA_DISK_1: piece handle=/oracle/2017-06-05/arch_POWERDES_20170605_9912.bak tag=TAG20170605T051030
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15
archived log file name=/oracle/app/oracle/archivelogs/1_20357_940127349.dbf thread=1 sequence=20357
archived log file name=/oracle/app/oracle/archivelogs/1_20358_940127349.dbf thread=1 sequence=20358
archived log file name=/oracle/app/oracle/archivelogs/1_20359_940127349.dbf thread=1 sequence=20359
archived log file name=/oracle/app/oracle/archivelogs/1_20360_940127349.dbf thread=1 sequence=20360
unable to find archived log
archived log thread=1 sequence=20361
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 06/05/2017 22:17:07
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 20361 and starting SCN of 14425991740
RMAN> recover database until scn 14425991740;
Starting recover at 05-JUN-17
using channel ORA_DISK_1
starting media recovery
Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '/home/oradata/powerdes/system01.dbf'
media recovery complete, elapsed time: 00:00:01
Finished recover at 05-JUN-17
RMAN>
Finally start the transfer and apply the archive log
SQL> alter database recover managed standby database using current logfile disconnect from session;
Database altered.
SQL> 4. Start checking the consistency of backup data in the main database
View the application of archive logs:
SQL> select sequence#,applied from v$archived_log order by sequence# asc;
SEQUENCE# APPLIED
---------- ---------
20379 YES
20380 YES
20381 YES
20382 YES
20383 YES
20384 YES
20385 YES
20386 YES
20387 YES
20388 YES
20389 YES
SEQUENCE# APPLIED
---------- ---------
20390 YES
20391 YES
20392 YES
20393 YES
20394 YES
20395 YES
20396 YES
20397 YES
20398 NO
20399 NO
20400 NOView the background alert log, which is being applied
Media Recovery Log /oracle/app/oracle/archivelogs/1_20391_940127349.dbf
Mon Jun 05 22:24:00 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20392_940127349.dbf
Mon Jun 05 22:24:10 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20393_940127349.dbf
Mon Jun 05 22:24:22 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20394_940127349.dbf
Mon Jun 05 22:24:33 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20395_940127349.dbf
Mon Jun 05 22:24:47 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20396_940127349.dbf
Mon Jun 05 22:25:00 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20397_940127349.dbf
Viewing the log information of the main repository is consistent, as follows:
SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /oracle/app/oracle/archivelogs
Oldest online log sequence 20430
Next log sequence to archive 20435
Current log sequence 20435
SQL>
SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /oracle/app/oracle/archivelogs
Oldest online log sequence 20430
Next log sequence to archive 0
Current log sequence 20435
SQL>
So far, the standby Library of dataguard has been restored by using rman backup and restore directly.
Think carefully, this problem has never been encountered before. When the disk is full, it can be restored directly after cleaning up the disk, but it can not be restored. The reason is that the real-time application archive log is enabled. This leads to confusion when the load is retransmitted again. To view previous dataguard build records http://blog.csdn.net/mchdba/article/details/71036947 Similar operating records were found:
(2)Add to current logfile Parameters, so that the application is currently reading and writing, and has not yet completed the archiving of the log
SQL> alter database recover managedstandby database using current logfile disconnect from session;