Restoring OLR – 11gR2

OLR (Oracle Local Registry) stores the information used by OHASD during the startup sequence of the node.The OLR clusterware configuration file is present on all the nodes of the RAC database.It also stores the Oracle Restart’s metadata for Single instance database in 11gR2.

[oracle@anand-lab bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :        152
         Available space (kbytes) :     261968
         ID                       : 1434054292
         Device/File Name         : /u01/app/oracle/grid/11.2.0.2/db_1/cdata/localhost/anand-lab.olr
                                    Device/File integrity check succeeded
         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

To check for the logical corruption, login as root and do ocrcheck

[root@anand-lab bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :        152
         Available space (kbytes) :     261968
         ID                       : 1434054292
         Device/File Name         : /u01/app/oracle/grid/11.2.0.2/db_1/cdata/localhost/anand-lab.olr
                                    Device/File integrity check succeeded
         Cluster registry integrity check succeeded

         Logical corruption check succeeded

If the OLR is corrupt or inaccessible, the OHAS doesn’t start.

[oracle@anand-lab bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab                                    
ora.DATA02.dg
               ONLINE  ONLINE       anand-lab                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.LISTENR_TEST.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.asm
               ONLINE  ONLINE       anand-lab                Started             
ora.ons
               OFFLINE OFFLINE      anand-lab                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab                                    
ora.diskmon
      1        ONLINE  ONLINE       anand-lab                                    
ora.evmd
      1        ONLINE  ONLINE       anand-lab                                    
ora.matrix.db
      1        ONLINE  ONLINE       anand-lab                Open                
[oracle@anand-lab bin]$ 
[root@anand-lab bin]# ./ocrconfig -local -showbackup

anand-lab     2011/04/29 14:42:14     /u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110429_144214.olr
[root@anand-lab bin]# 
[root@anand-lab bin]# ./ocrconfig -local -manualbackup   --> Took a new backup

anand-lab     2011/08/20 12:18:06     /u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110820_121806.olr

anand-lab     2011/04/29 14:42:14     /u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110429_144214.olr
[root@anand-lab bin]# 

Lets rename the OLR

[oracle@anand-lab localhost]$ ls -lrth
-rw-r----- 1 oracle oinstall 261M Aug 20 12:18 anand-lab.olr
[oracle@anand-lab localhost]$ mv anand-lab.olr anand-lab.olr.bkp
[oracle@anand-lab localhost]$ ls -lrt
-rw-r----- 1 oracle oinstall 272756736 Aug 20 12:18 anand-lab.olr.bkp
[root@anand-lab bin]$ ./ocrcheck -local
PROTL-602: Failed to retrieve data from the local registry
PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
[oracle@anand-lab client]$ pwd
/u01/app/oracle/grid/11.2.0.2/db_1/log/anand-lab/client
[oracle@anand-lab client]$ cat ocrcheck_4189.log
Oracle Database 11g Clusterware Release 11.2.0.2.0 - Production Copyright 1996, 2010 Oracle. All rights reserved.
2011-08-20 12:29:39.216: [OCRCHECK][3086636736]ocrcheck starts...
2011-08-20 12:29:39.217: [  OCROSD][3086636736]utopen:6m': failed in stat OCR file/disk /u01/app/oracle/grid/11.2.0.2/db_1/cdata/localhost/anand-lab.olr, errno=2, os err string=No such file or directory
2011-08-20 12:29:39.217: [  OCROSD][3086636736]utopen:7: failed to open any OCR file/disk, errno=2, os err string=No such file or directory
2011-08-20 12:29:39.217: [  OCRRAW][3086636736]proprinit: Could not open raw device 
2011-08-20 12:29:39.217: [ default][3086636736]a_init:7!: Backend init unsuccessful : [26]
2011-08-20 12:29:39.217: [OCRCHECK][3086636736]Failed to access OCR repository. Retval [26] Error [PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]]
2011-08-20 12:29:39.217: [OCRCHECK][3086636736]Failed to initialize ocrchek2
2011-08-20 12:29:39.217: [OCRCHECK][3086636736]Exiting [status=failed]...

Stopped the OHASD using crsctl stop has and started back –

[root@anand-lab bin]# ./crsctl start has
CRS-4563: Insufficient user privileges.

CRS-4000: Command Start failed, or completed with errors.
[root@anand-lab localhost]# touch anand-lab.olr
[root@anand-lab localhost]# 
[root@anand-lab localhost]# ls -lrt
-rw-r----- 1 oracle oinstall 272756736 Aug 20 12:34 anand-lab.olr.bkp
-rw-r--r-- 1 root   root             0 Aug 20 12:45 anand-lab.olr
[root@anand-lab localhost]# chown oracle:oinstall anand-lab.olr
[root@anand-lab localhost]# chmod 640 anand-lab.olr
[root@anand-lab localhost]# ls -lrt
total 2828
-rw-r----- 1 oracle oinstall 272756736 Aug 20 12:34 anand-lab.olr.bkp
-rw-r----- 1 oracle oinstall         0 Aug 20 12:45 anand-lab.olr
[root@anand-lab localhost]# 

Restore OLR from backup –

[root@anand-lab bin]# ./ocrconfig -local -restore /u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110820_121806.olr
[root@anand-lab bin]# 

Log shows –

[oracle@anand-lab client]$ more ocrconfig_5059.log 
Oracle Database 11g Clusterware Release 11.2.0.2.0 - Production Copyright 1996, 2010 Oracle. All rights reserved.
2011-08-20 13:05:07.580: [ OCRCONF][3086165696]ocrconfig starts...
2011-08-20 13:05:07.580: [ OCRCONF][3086165696]Started to restore the Oracle Local Registry location from backup file [/u01/app/oracle/grid/11.2.0.2/db_1/cdata/anand-lab/backup_20110820_121806.olr].
2011-08-20 13:05:07.631: [  OCRMSG][3086165696]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2011-08-20 13:05:07.631: [  OCRMSG][3086165696]GIPC error [29] msg [gipcretConnectionRefused]
2011-08-20 13:05:07.632: [  OCRMSG][3086165696]prom_connect: error while waiting for connection complete [24]
2011-08-20 13:05:07.927: [ OCRCONF][3086165696]Successfully restored OCR and set block 0
2011-08-20 13:05:07.927: [ OCRCONF][3086165696]Exiting [status=success]...
[root@anand-lab bin]# ./crsctl stat res -t
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Status failed, or completed with errors.

Time to start OHASd 
[root@anand-lab bin]# ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[root@anand-lab bin]# 
[root@anand-lab bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab                                    
ora.DATA02.dg
               ONLINE  ONLINE       anand-lab                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.LISTENR_TEST.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.asm
               ONLINE  ONLINE       anand-lab                Started             
ora.ons
               OFFLINE OFFLINE      anand-lab                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab                                    
ora.diskmon
      1        ONLINE  ONLINE       anand-lab                                    
ora.evmd
      1        ONLINE  ONLINE       anand-lab                                    
ora.matrix.db
      1        ONLINE  ONLINE       anand-lab                Open                
[root@anand-lab bin]# 
Advertisements

2 thoughts on “Restoring OLR – 11gR2

  1. Anand, thank you so much for this article. It helped me to troubleshoot an issue that I was having while upgrading clusterware software from version 12.1.0.1.0 to 12.1.0.2.0.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s