Yesterday I received page for ” ORA-17629: Cannot connect to the remote database server” reported in Primary Database alert log.
Fri Apr 10 06:47:41 2015
Errors in file /oracle/app/diag/rdbms/testdb/testdb/trace/testdb_ora_161991.trc:
ORA-17627:
ORA-17629: Cannot connect to the remote database server
Errors in file /oracle/app/diag/rdbms/testdb/testdb/trace/testdb_ora_161991.trc:
ORA-17629: Cannot connect to the remote database server
ORA-17627:
ORA-17629: Cannot connect to the remote database server
Fri Apr 10 06:48:22 2015
The error trace file shows
*** 2015-04-10 06:47:13.005
*** SESSION ID:(3024.45081) 2015-04-10 06:47:13.005
*** CLIENT ID:() 2015-04-10 06:47:13.005
*** SERVICE NAME:(testdb) 2015-04-10 06:47:13.005
*** MODULE NAME:(oracle@oracle2.testdbds.com (TNS V1-V3)) 2015-04-10 06:47:13.005
*** ACTION NAME:() 2015-04-10 06:47:13.005krbmsrvgrcf:Expected db_unique_name: testdb
krbmsrvgrcf:mydbuname = :testdb:
krbmsrvgrcf:Instance with connect id testdbSBY requested controlfile.
krbmsrvgrcf:Filespec: /oracle/app/product/11.2_1/dbs/snappcf_testdb.file*** 2015-04-10 06:47:41.467
OCI error val is 184283056 and errmsg is ”
ORA-17627:
ORA-17629: Cannot connect to the remote database server
******************** WARNING ***************************
The errors during Server autobackup are not fatal, as it
is attempted after sucessful completion of the command.
However, it is recomended to take an RMAN control file
backup as soon as possible because the Autobackup failed
with the following error:
ORA-17629: Cannot connect to the remote database server
ORA-17627:
ORA-17629: Cannot connect to the remote database server
******************** END OF WARNING *******************
ORA-17629: Cannot connect to the remote database server
ORA-17627:
ORA-17629: Cannot connect to the remote database server
The trace file mentions the SID,SERIAL# and the module name. The module shows “oracle@oracle2.testdbds.com” which is the standby database.
I thought to check the sid details from v$active_session_history and the session is coming in from standby database and definitely related to RMAN.
SQL> @ash_sid
Enter value for sid: 3024
Enter value for serial: 45081
old 17: where session_id = &sid and session_serial# = &serial
new 17: where session_id = 3024 and session_serial# = 45081
SQL Dur CPU DB Time
Sid,Serial<Blk STIME Module SqlId:ChildNo SqlExecId Start CPH S PRXJBZ Sta SEQ# EVENT P1:P2:P3 WCLASS ms ms ms
--------------- -------- -------------------- ---------------- --------- ----- ------------ ---- ------ ------------------------ ------------------ ----------- ------- ----- -------
3024,45081 06:47:13 oracle@oracle2.testdbds.com NNN N NNNNN Wait 129 control file single writ 9999999: 1024: 9 System I/O 105 8 113
06:47:14 NNN N NNNNN Wait 269 control file single writ 9999999: 5504: 9 System I/O
06:47:15 NNN N NNNNN Wait 395 control file single writ 9999999: 9536: 9 System I/O
06:47:16 NNN N NNNNN Wait 473 control file single writ 9999999: 12032: 9 System I/O
06:47:17 NNN N NNNNN Wait 560 control file sequential 9999999: 14848: 9 System I/O
06:47:18 NNN N NNNNN Wait 673 control file single writ 9999999: 18432: 9 System I/O
06:47:19 NNN N NNNNN Wait 768 control file sequential 9999999: 21504: 9 System I/O
06:47:20 NNN N NNNNN Wait 863 control file single writ 9999999: 24512: 9 System I/O
06:47:21 NNN N NNNNN Wait 959 control file single writ 9999999: 27584: 9 System I/O
06:47:22 NNN N NNNNN cpu 1044 9999999: 30336: 9
06:47:23 NNN N NNNNN cpu 1132 9999999: 33152: 9
06:47:24 NNN N NNNNN Wait 1220 control file sequential 9999999: 35968: 9 System I/O
06:47:25 NNN N NNNNN Wait 1318 control file sequential 9999999: 39104: 9 System I/O
06:47:26 NNN N NNNNN Wait 1405 control file single writ 9999999: 41856: 9 System I/O
06:47:27 NNN N NNNNN Wait 1489 control file single writ 9999999: 44544: 9 System I/O
06:47:28 NNN N NNNNN Wait 1567 control file single writ 9999999: 47040: 9 System I/O
06:47:29 NNN N NNNNN Wait 1649 control file single writ 9999999: 49664: 9 System I/O
06:47:30 NNN N NNNNN Wait 1733 control file single writ 9999999: 52352: 9 System I/O
06:47:31 NNN N NNNNN cpu 1817 9999999: 55040: 9
06:47:32 NNN N NNNNN Wait 1895 control file single writ 9999999: 57536: 9 System I/O
06:47:33 NNN N NNNNN Wait 1965 control file single writ 9999999: 59776: 9 System I/O
06:47:34 NNN N NNNNN Wait 2041 control file single writ 9999999: 62208: 9 System I/O
06:47:35 NNN N NNNNN Wait 2113 control file single writ 9999999: 64512: 9 System I/O
06:47:36 NNN N NNNNN Wait 2179 control file single writ 9999999: 66624: 9 System I/O
06:47:37 NNN N NNNNN Wait 2238 control file sequential 9999999: 68544: 9 System I/O
06:47:38 NNN N NNNNN Wait 2307 control file single writ 9999999: 70720: 9 System I/O
06:47:39 NNN N NNNNN Wait 2369 control file single writ 9999999: 72704: 9 System I/O
06:47:40 NNN N NNNNN Wait 2444 control file sequential 9999999: 75136: 9 System I/O
28 rows selected.
Looking into MOS I found “RMAN-06820 ORA-17629 During Backup at Standby Site (Doc ID 1616074.1)”. It stated something interesting, so I thought to mention it here, which I did not know
Change in 11.2.0.4 onward
Per ‘unpublished’ Bug 8740124, as of 11.2.0.4, we now include the current standby redo log as part of an RMAN archivelog backup at the standby site. This is achieved by forcing a log switch at the primary site.
I looked at the archive log backup trace file on standby database and there was no error reported. Then I started looking into the backup script and saw, after the backup is completed, resync function is called which connects to rcat database and perform resync catalog.
cle2.testdbs logs]$ more testdb_backup_testdb_resync_201504100617_Fri.log
Recovery Manager: Release 11.2.0.3.0 – Production on Fri Apr 10 06:46:55 2015
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
RMAN>
echo set onRMAN>
RMAN> connect target *
connected to target database: testdb (DBID=3948365078)RMAN> connect catalog *
connected to recovery catalog databaseRMAN> resync catalog;
starting partial resync of recovery catalog
ORA-20079: full resync from primary database is not done
doing automatic resync from primary
resyncing from database with DB_UNIQUE_NAME testdb
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync command on default channel at 04/10/2015 06:47:13
ORA-17629: Cannot connect to the remote database server
ORA-17628: Oracle error 17629 returned by remote Oracle serverRMAN> exit;
Recovery Manager complete.
Tried “show all for db_unique_name testdb;” from standby
RMAN> show all for db_unique_name testdb;
ORA-20079: full resync from primary database is not done
doing automatic resync from primary
resyncing from database with DB_UNIQUE_NAME testdb
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of show command at 04/10/2015 07:53:40
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of partial resync command on default channel at 04/10/2015 07:53:40
ORA-17629: Cannot connect to the remote database server
ORA-17628: Oracle error 17629 returned by remote Oracle server
Connected to Primary Database and ran the same command
[oracle@oracle1 anand]$ rman target /
Recovery Manager: Release 11.2.0.3.0 – Production on Fri Apr 10 07:55:43 2015
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: testdb (DBID=3948365078)
RMAN> connect catalog rcat/xxxxxxxx@rcat
connected to recovery catalog database
RMAN> show all for db_unique_name testdb;starting full resync of recovery catalog
full resync complete
RMAN configuration parameters for database with db_unique_name testdb are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 10 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO ‘%F’;
CONFIGURE DEVICE TYPE DISK PARALLELISM 8 BACKUP TYPE TO BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT ‘/mnt/oracle-backup/testdb/%U’;
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM ‘AES192’;
CONFIGURE COMPRESSION ALGORITHM ‘BASIC’ AS OF RELEASE ‘DEFAULT’ OPTIMIZE FOR LOAD FALSE;
CONFIGURE DB_UNIQUE_NAME ‘testdbSBY’ CONNECT IDENTIFIER ‘testdbSBY’;
CONFIGURE DB_UNIQUE_NAME ‘testdb’ CONNECT IDENTIFIER ‘testdb’;
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO ‘/oracle/app/product/11.2_1/dbs/snappcf_testdb.file’;RMAN>
RMAN> exit
The full resync completed on Primary database and then I performed “resync catalog” on standby as it completed successfully.
RMAN> resync catalog
2> ;starting partial resync of recovery catalog
partial resync completeRMAN> exit
Learn. Share. Repeat. 
Nice Post.