Oracle Restart – 11gR2

Oracle Restart is the new feature introduced by Oracle in 11gR2 which ensures that various Oracle components gets automatically restarted after hardware or software failures.This feature depends on OHAS (Oracle High Availability Service) daemon and is only for standalone server environment.The various components which gets automatically restarted as part of Oracle Restart are :-

1. Database instance
2. Database listener
3. Database services
4. Oracle ASM instance
5. Oracle ASM disk group
6. Oracle ONS

CRSCTL utility is used to stop and start Oracle Restart.

[oracle@anand-lab bin]$ ps -ef | grep has
oracle    2458     1  0 13:00 ?        00:00:01 /u01/app/oracle/grid/11.2.0.2/db_1/bin/ohasd.bin reboot
root      2631     1  0 13:00 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
oracle    8003  3995  0 16:36 pts/2    00:00:00 grep has
[oracle@anand-lab bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.asm
               ONLINE  ONLINE       anand-lab                Started             
ora.ons
               OFFLINE OFFLINE      anand-lab                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab                                    
ora.diskmon
      1        ONLINE  ONLINE       anand-lab                                    
ora.evmd
      1        ONLINE  ONLINE       anand-lab                                    
ora.orcl.db
      1        ONLINE  ONLINE       anand-lab                Open                
[oracle@anand-lab bin]$ 

SRVCTL utility can be used to start and stop the Oracle Restart components manually.When you start/stop a component with SRVCTL, any components on which this component depends are automatically started/stopped first, and in the proper order.

[oracle@anand-lab bin]$ ./srvctl stop asm 
PRCR-1065 : Failed to stop resource ora.asm
CRS-2529: Unable to act on 'ora.asm' because that would require stopping or relocating 'ora.orcl.db', but the force option was not specified
[oracle@anand-lab bin]$ ./srvctl stop asm -f
[oracle@anand-lab bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  OFFLINE      anand-lab                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.asm
               OFFLINE OFFLINE      anand-lab                Instance Shutdown   
ora.ons
               OFFLINE OFFLINE      anand-lab                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab                                    
ora.diskmon
      1        ONLINE  ONLINE       anand-lab                                    
ora.evmd
      1        ONLINE  ONLINE       anand-lab                                    
ora.orcl.db
      1        ONLINE  OFFLINE                               Instance Shutdown   
[oracle@anand-lab bin]$ 

Though i stopped ASM instance but as Database instance component depends on it, its stopped automatically.How to check the dependencies?

[oracle@anand-lab bin]$ ./crsctl stat res ora.orcl.db -p | grep DEPENDENCIES
START_DEPENDENCIES=hard(ora.DATA.dg) weak(type:ora.listener.type,uniform:ora.ons,uniform:ora.eons) pullup(ora.DATA.dg)
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg)
[oracle@anand-lab bin]$ 

[oracle@anand-lab bin]$ ./crsctl stat res ora.asm -p | grep DEPENDENCIES
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
STOP_DEPENDENCIES=hard(ora.cssd)
[oracle@anand-lab bin]$ 

HARD :- Specify a hard start dependency for a resource when you want the resource to start only when a particular resource or resource of a particular type starts.Use intermediate to specify that Oracle Clusterware can start this resource if a resource on which it depends is in either the ONLINE or INTERMEDIATE state. If not specified, then resources must be in the ONLINE state for Oracle Clusterware to start this resource.

WEAK :- Specify a weak start dependency for a resource when you want that resource to start despite whether named resources are running, or not. An attempt to start this resource also attempts to start any resources on which this resource depends if they are not running.

PULLUP :- When you specify the pullup start dependency for a resource, then this resource starts as a result of named resources starting.

For more details read

http://download.oracle.com/docs/cd/E11882_01/rac.112/e16794/resatt.htm#CHDJHECG

What happens if you shutdown the database using sqlplus, and perform Oracle Restart.Does the instance start?The asnwer to this is NO (in default settings).

[oracle@anand-lab bin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sun May 1 16:56:14 2011

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> 
[oracle@anand-lab bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.asm
               ONLINE  ONLINE       anand-lab                Started             
ora.ons
               OFFLINE OFFLINE      anand-lab                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab                                    
ora.diskmon
      1        ONLINE  ONLINE       anand-lab                                    
ora.evmd
      1        ONLINE  ONLINE       anand-lab                                    
ora.orcl.db
      1        OFFLINE OFFLINE                               Instance Shutdown   
[oracle@anand-lab bin]$ 
[oracle@anand-lab bin]$ 
[oracle@anand-lab bin]$ 
[oracle@anand-lab bin]$ ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'anand-lab'
....................................
CRS-4133: Oracle High Availability Services has been stopped.
[oracle@anand-lab bin]$ ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[oracle@anand-lab bin]$ ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.asm
               ONLINE  ONLINE       anand-lab                Started             
ora.ons
               OFFLINE OFFLINE      anand-lab                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab                                    
ora.diskmon
      1        ONLINE  ONLINE       anand-lab                                    
ora.evmd
      1        ONLINE  ONLINE       anand-lab                                    
ora.orcl.db
      1        OFFLINE OFFLINE                                                   
[oracle@anand-lab bin]$ 

If you shut down the database with SQL*Plus, Oracle Restart does not interpret this as a database failure and does not attempt to restart the database.So, you need to manually startup the database using SQL*Plus which changes both the TARGET and STATE back to ONLINE.

Doing shutdown immediate using SQL*Plus cause the TARGET to be OFFLINE and by default the AUTO_START attribute is set to RESTORE which means, while starting up CRS will restore the resource to the same state that it was in when the server stopped. Oracle Clusterware attempts to restart the resource if the value of TARGET was ONLINE before the server stopped. So, be sure to bring down the database using SRVCTL utility instead of SQL*Plus.

[oracle@anand-lab bin]$ ./crsctl stat res ora.orcl.db -p | grep AUTO_START
AUTO_START=restore

From Doc

AUTO_START

Indicates whether Oracle Clusterware automatically starts a resource after a cluster server restart. Valid AUTO_START values are:

always: Restarts the resource when the server restarts regardless of the state of the resource when the server stopped.

restore: Restores the resource to the same state that it was in when the server stopped. Oracle Clusterware attempts to restart the resource if the value of TARGET was ONLINE before the server stopped.

never: Oracle Clusterware never restarts the resource regardless of the state of the resource when the server stopped.

Lets change the AUTO_START attribute to “always”

[oracle@anand-lab bin]$ ./crsctl modify resource ora.orcl.db -attr AUTO_START=always
[oracle@anand-lab bin]$ ./crsctl stat res ora.orcl.db -p | grep AUTO_START
AUTO_START=always
[oracle@anand-lab bin]$ 
SQL> shu immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> 
ora.orcl.db
      1        OFFLINE OFFLINE                               Instance Shutdown   
[oracle@anand-lab bin]$ ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'anand-lab'
CRS-2673: Attempting to stop 'ora.evmd' on 'anand-lab'
.........................
CRS-4133: Oracle High Availability Services has been stopped.

Starting HAS

[oracle@anand-lab bin]$ ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[oracle@anand-lab bin]$ 

Check the status of the resource, the ora.orcl.db resource’s TARGET and STATE should be online with db in OPEN mode.

[oracle@anand-lab bin]$ ./crsctl stat res -t 
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       anand-lab                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       anand-lab                                    
ora.asm
               ONLINE  ONLINE       anand-lab                Started             
ora.ons
               OFFLINE OFFLINE      anand-lab                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       anand-lab                                    
ora.diskmon
      1        ONLINE  ONLINE       anand-lab                                    
ora.evmd
      1        ONLINE  ONLINE       anand-lab                                    
ora.orcl.db
      1        ONLINE  ONLINE       anand-lab                Open                
[oracle@anand-lab bin]$ 

5 thoughts on “Oracle Restart – 11gR2

  1. You cannot run crsctl stop has for shutting down resources … as this is an abrupt shutdown. Please look into alert log file of database

    1. Yes stopping the OHAS or even CRS using crsctl stop crs on any version of oracle when the instance is up and running causes the instance to abort.As Oracle restart ensures that various Oracle components gets automatically restarted after hardware or software failures, hence showed crsctl stop has to simulate an hardware failure, as in hardware failure oracle instance wouldn’t be shutdown normally.

      Regards,
      Anand

  2. Hi Anand,
    Great post on Oracle Restart feature.
    i have a question on this feature-

    If you shut down the database using “srvctl” , it caused the TARGET
    to be OFFLINE and after starting up CRS it didnt bring up the database.

    Is this the correct behaviour?

    Please see the log –>

    $ srvctl stop database -d orcl

    $ crsctl stat res -t
    ——————————————————————————–
    NAME TARGET STATE SERVER STATE_DETAILS
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.DATA.dg
    ONLINE ONLINE temp-dba01
    ora.DG1.dg
    ONLINE ONLINE temp-dba01
    ora.FRA.dg
    ONLINE ONLINE temp-dba01
    ora.LISTENER.lsnr
    ONLINE ONLINE temp-dba01
    ora.asm
    ONLINE ONLINE temp-dba01 Started
    ora.ons
    OFFLINE OFFLINE temp-dba01
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.cssd
    1 ONLINE ONLINE temp-dba01
    ora.diskmon
    1 ONLINE ONLINE temp-dba01
    ora.evmd
    1 ONLINE ONLINE temp-dba01
    ora.orcl.db
    1 OFFLINE OFFLINE Instance Shutdown
    ora.tmp.db
    1 OFFLINE OFFLINE

    $ crsctl stop has

    CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.evmd’ on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.DG1.dg’ on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.FRA.dg’ on ‘temp-dba01’
    CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘temp-dba01’
    CRS-2677: Stop of ‘ora.DG1.dg’ on ‘temp-dba01’ succeeded
    CRS-2677: Stop of ‘ora.FRA.dg’ on ‘temp-dba01’ succeeded
    CRS-2677: Stop of ‘ora.DATA.dg’ on ‘temp-dba01’ succeeded
    CRS-2673: Attempting to stop ‘ora.asm’ on ‘temp-dba01’
    CRS-2677: Stop of ‘ora.asm’ on ‘temp-dba01’ succeeded
    CRS-2673: Attempting to stop ‘ora.cssd’ on ‘temp-dba01’
    CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘temp-dba01’ succeeded
    CRS-2677: Stop of ‘ora.cssd’ on ‘temp-dba01’ succeeded
    CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘temp-dba01’
    CRS-2677: Stop of ‘ora.evmd’ on ‘temp-dba01’ succeeded
    CRS-2677: Stop of ‘ora.diskmon’ on ‘temp-dba01’ succeeded
    CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘temp-dba01’ has completed
    CRS-4133: Oracle High Availability Services has been stopped.

    $ crsctl start has

    CRS-4123: Oracle High Availability Services has been started.

    $ crsctl stat res -t
    ——————————————————————————–
    NAME TARGET STATE SERVER STATE_DETAILS
    ——————————————————————————–
    Local Resources
    ——————————————————————————–
    ora.DATA.dg
    ONLINE OFFLINE temp-dba01
    ora.DG1.dg
    ONLINE OFFLINE temp-dba01
    ora.FRA.dg
    ONLINE OFFLINE temp-dba01
    ora.LISTENER.lsnr
    ONLINE OFFLINE temp-dba01 STARTING
    ora.asm
    ONLINE OFFLINE temp-dba01
    ora.ons
    OFFLINE OFFLINE temp-dba01
    ——————————————————————————–
    Cluster Resources
    ——————————————————————————–
    ora.cssd
    1 ONLINE OFFLINE
    ora.diskmon
    1 ONLINE OFFLINE
    ora.evmd
    1 ONLINE INTERMEDIATE temp-dba01
    ora.orcl.db
    1 OFFLINE OFFLINE
    ora.tmp.db
    1 OFFLINE OFFLINE

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s