ORA-17629 ORA-20079

Yesterday I received page for ” ORA-17629: Cannot connect to the remote database server”  reported in Primary Database alert log.

Fri Apr 10 06:47:41 2015
Errors in file /oracle/app/diag/rdbms/testdb/testdb/trace/testdb_ora_161991.trc:
ORA-17627:
ORA-17629: Cannot connect to the remote database server
Errors in file /oracle/app/diag/rdbms/testdb/testdb/trace/testdb_ora_161991.trc:
ORA-17629: Cannot connect to the remote database server
ORA-17627:
ORA-17629: Cannot connect to the remote database server
Fri Apr 10 06:48:22 2015

The error trace file shows

*** 2015-04-10 06:47:13.005
*** SESSION ID:(3024.45081) 2015-04-10 06:47:13.005

*** CLIENT ID:() 2015-04-10 06:47:13.005
*** SERVICE NAME:(testdb) 2015-04-10 06:47:13.005
*** MODULE NAME:(oracle@oracle2.testdbds.com (TNS V1-V3)) 2015-04-10 06:47:13.005

*** ACTION NAME:() 2015-04-10 06:47:13.005

krbmsrvgrcf:Expected db_unique_name: testdb
krbmsrvgrcf:mydbuname = :testdb:
krbmsrvgrcf:Instance with connect id testdbSBY requested controlfile.
krbmsrvgrcf:Filespec: /oracle/app/product/11.2_1/dbs/snappcf_testdb.file

*** 2015-04-10 06:47:41.467
OCI error val is 184283056 and errmsg is ”
ORA-17627:
ORA-17629: Cannot connect to the remote database server
******************** WARNING ***************************
The errors during Server autobackup are not fatal, as it
is attempted after sucessful completion of the command.
However, it is recomended to take an RMAN control file
backup as soon as possible because the Autobackup failed
with the following error:
ORA-17629: Cannot connect to the remote database server
ORA-17627:
ORA-17629: Cannot connect to the remote database server
******************** END OF WARNING *******************
ORA-17629: Cannot connect to the remote database server
ORA-17627:
ORA-17629: Cannot connect to the remote database server

The trace file mentions the SID,SERIAL# and the module name. The module shows “oracle@oracle2.testdbds.com” which is the standby database.

I thought to check the sid details from v$active_session_history and the session is coming in from standby database and definitely related to RMAN.

SQL> @ash_sid
Enter value for sid: 3024
Enter value for serial: 45081
old  17: where    session_id = &sid and session_serial# = &serial
new  17: where    session_id = 3024 and session_serial# = 45081

                                                                         SQL                                                                                        Dur   CPU DB Time
Sid,Serial<Blk  STIME    Module               SqlId:ChildNo    SqlExecId Start CPH S PRXJBZ Sta    SEQ# EVENT                            P1:P2:P3   WCLASS           ms    ms      ms
--------------- -------- -------------------- ---------------- --------- ----- ------------ ---- ------ ------------------------ ------------------ ----------- ------- ----- -------
3024,45081      06:47:13 oracle@oracle2.testdbds.com                           NNN N NNNNN  Wait    129 control file single writ 9999999:  1024:  9 System I/O      105     8     113
                06:47:14                                                       NNN N NNNNN  Wait    269 control file single writ 9999999:  5504:  9 System I/O
                06:47:15                                                       NNN N NNNNN  Wait    395 control file single writ 9999999:  9536:  9 System I/O
                06:47:16                                                       NNN N NNNNN  Wait    473 control file single writ 9999999: 12032:  9 System I/O
                06:47:17                                                       NNN N NNNNN  Wait    560 control file sequential  9999999: 14848:  9 System I/O
                06:47:18                                                       NNN N NNNNN  Wait    673 control file single writ 9999999: 18432:  9 System I/O
                06:47:19                                                       NNN N NNNNN  Wait    768 control file sequential  9999999: 21504:  9 System I/O
                06:47:20                                                       NNN N NNNNN  Wait    863 control file single writ 9999999: 24512:  9 System I/O
                06:47:21                                                       NNN N NNNNN  Wait    959 control file single writ 9999999: 27584:  9 System I/O
                06:47:22                                                       NNN N NNNNN  cpu    1044                          9999999: 30336:  9
                06:47:23                                                       NNN N NNNNN  cpu    1132                          9999999: 33152:  9
                06:47:24                                                       NNN N NNNNN  Wait   1220 control file sequential  9999999: 35968:  9 System I/O
                06:47:25                                                       NNN N NNNNN  Wait   1318 control file sequential  9999999: 39104:  9 System I/O
                06:47:26                                                       NNN N NNNNN  Wait   1405 control file single writ 9999999: 41856:  9 System I/O
                06:47:27                                                       NNN N NNNNN  Wait   1489 control file single writ 9999999: 44544:  9 System I/O
                06:47:28                                                       NNN N NNNNN  Wait   1567 control file single writ 9999999: 47040:  9 System I/O
                06:47:29                                                       NNN N NNNNN  Wait   1649 control file single writ 9999999: 49664:  9 System I/O
                06:47:30                                                       NNN N NNNNN  Wait   1733 control file single writ 9999999: 52352:  9 System I/O
                06:47:31                                                       NNN N NNNNN  cpu    1817                          9999999: 55040:  9
                06:47:32                                                       NNN N NNNNN  Wait   1895 control file single writ 9999999: 57536:  9 System I/O
                06:47:33                                                       NNN N NNNNN  Wait   1965 control file single writ 9999999: 59776:  9 System I/O
                06:47:34                                                       NNN N NNNNN  Wait   2041 control file single writ 9999999: 62208:  9 System I/O
                06:47:35                                                       NNN N NNNNN  Wait   2113 control file single writ 9999999: 64512:  9 System I/O
                06:47:36                                                       NNN N NNNNN  Wait   2179 control file single writ 9999999: 66624:  9 System I/O
                06:47:37                                                       NNN N NNNNN  Wait   2238 control file sequential  9999999: 68544:  9 System I/O
                06:47:38                                                       NNN N NNNNN  Wait   2307 control file single writ 9999999: 70720:  9 System I/O
                06:47:39                                                       NNN N NNNNN  Wait   2369 control file single writ 9999999: 72704:  9 System I/O
                06:47:40                                                       NNN N NNNNN  Wait   2444 control file sequential  9999999: 75136:  9 System I/O


28 rows selected.

Looking into MOS I found “RMAN-06820 ORA-17629 During Backup at Standby Site (Doc ID 1616074.1)”. It stated something interesting, so I thought to mention it here, which I did not know

Change in 11.2.0.4 onward

Per ‘unpublished’ Bug 8740124, as of 11.2.0.4, we now include the current standby redo log as part of an RMAN archivelog backup at the standby site. This is achieved by forcing a log switch at the primary site.

I looked at the archive log backup trace file on standby database and there was no error reported. Then I started looking into the backup script and saw, after the backup is completed, resync function is called which connects to rcat database and perform resync catalog.

cle2.testdbs logs]$ more testdb_backup_testdb_resync_201504100617_Fri.log

Recovery Manager: Release 11.2.0.3.0 – Production on Fri Apr 10 06:46:55 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

RMAN>
echo set on

RMAN>

RMAN> connect target *
connected to target database: testdb (DBID=3948365078)

RMAN> connect catalog *
connected to recovery catalog database

RMAN> resync catalog;
starting partial resync of recovery catalog
ORA-20079: full resync from primary database is not done

doing automatic resync from primary
resyncing from database with DB_UNIQUE_NAME testdb
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync command on default channel at 04/10/2015 06:47:13
ORA-17629: Cannot connect to the remote database server

ORA-17628: Oracle error 17629 returned by remote Oracle server

RMAN> exit;

Recovery Manager complete.

Tried “show all for db_unique_name testdb;” from standby

RMAN> show all for db_unique_name testdb;

ORA-20079: full resync from primary database is not done

doing automatic resync from primary
resyncing from database with DB_UNIQUE_NAME testdb
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of show command at 04/10/2015 07:53:40
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of partial resync command on default channel at 04/10/2015 07:53:40
ORA-17629: Cannot connect to the remote database server
ORA-17628: Oracle error 17629 returned by remote Oracle server

Connected to Primary Database and ran the same command

[oracle@oracle1 anand]$ rman target /

Recovery Manager: Release 11.2.0.3.0 – Production on Fri Apr 10 07:55:43 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: testdb (DBID=3948365078)

RMAN> connect catalog rcat/xxxxxxxx@rcat

connected to recovery catalog database

RMAN> show all for db_unique_name testdb;

starting full resync of recovery catalog

full resync complete
RMAN configuration parameters for database with db_unique_name testdb are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 10 DAYS;
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO ‘%F’;
CONFIGURE DEVICE TYPE DISK PARALLELISM 8 BACKUP TYPE TO BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT ‘/mnt/oracle-backup/testdb/%U’;
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM ‘AES192’;
CONFIGURE COMPRESSION ALGORITHM ‘BASIC’ AS OF RELEASE ‘DEFAULT’ OPTIMIZE FOR LOAD FALSE;
CONFIGURE DB_UNIQUE_NAME ‘testdbSBY’ CONNECT IDENTIFIER ‘testdbSBY’;
CONFIGURE DB_UNIQUE_NAME ‘testdb’ CONNECT IDENTIFIER ‘testdb’;
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO ‘/oracle/app/product/11.2_1/dbs/snappcf_testdb.file’;

RMAN>
RMAN> exit

The full resync completed on Primary database and then I performed “resync catalog” on standby as it completed successfully.

RMAN> resync catalog
2> ;

starting partial resync of recovery catalog
partial resync complete

RMAN> exit

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s