hasGetClusterStatus Status verification failed due to cluvfy execution failure for node(s) – 10gR2 RAC

Received a mail from colleague reporting the below error from emgc Metric Collection Errors –

Target		crs_node1
Type		Cluster
Metric		Clusterware Status
Collection Timestamp	Dec 14, 2011 5:35:02 AM
Error Type		Collection Problem
Message		WARN|has::Common::hasGetClusterStatus Status verification failed due to cluvfy execution failure for node(s) node3:EFAIL,NODE_STATUS::node2:EFAIL,NODE_STATUS::node1:EFAIL,NODE_STATUS::node4:EFAIL,OVERALL_STATUS: 

The first thing i did is ran cluvfy


node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose

Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "node1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  node3                                yes
  node1                                yes
  node4                                yes
  node2                                yes
Result: Node reachability check passed from node "node1".


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Comment
  ------------------------------------  ------------------------
  node3                                passed
  node2                                passed
  node1                                passed
  node4                                passed
Result: User equivalence check passed for user "oracle".

ERROR:
The location "/tmp/CVU_10.2.0.5.0.1_dba/" is owned by another user on nodes:
        node3,node1,node2
Verification will proceed with nodes:
        node4

ERROR:
CRS is not installed on any of the nodes.
Verification cannot proceed.


Post-check for cluster services setup was unsuccessful on all the nodes.

The cluvfy verification failed. Checked for the permission on /tmp/CVU_10.2.0.5.0.1_dba

drwxr-----  3 em  dba      4096 May 27  2010 CVU_10.2.0.5.0.1_dba

Changed the permission to 770

drwxrwx---  3 em  dba      4096 May 27  2010 CVU_10.2.0.5.0.1_dba

After changing the permission of /tmp/CVU_10.2.0.5.0.1_dba/ on all the nodes ran cluvfy again

node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose

Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "node1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  node3                                yes
  node1                                yes
  node4                                yes
  node2                                yes
Result: Node reachability check passed from node "node1".


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Comment
  ------------------------------------  ------------------------
  node3                                passed
  node2                                passed
  node1                                passed
  node4                                passed
Result: User equivalence check passed for user "oracle".

WARNING:
CRS is not installed on nodes:
        node4
Verification will proceed with nodes:
        node3,node2,node1
.........................

Post-check for cluster services setup was unsuccessful.
Checks did not pass for the following node(s):
        node4

olsnodes showed all the 4 nodes, but cluvfy showed CRS is not installed on nodes: node4

To find out more, digged into the cluvfy log in $CRS_HOME/cv/log

[main] [5:36:32:622] [OUIData.readInventoryData:393]  ==== CRS home added: Oracle home properties:
Name     : OraCrs10g_home
Type     : CRS-HOME
Location : /app/oracle/crs
Node list: [node1, node2, node3]
...............................
[main] [5:43:44:502] [Stage.verify:359]  m_currentTaskSet.size=1; Wed Dec 14 05:43:44 CST 2011
[main] [5:43:44:502] [TaskNodeAppCreation.performTask:157]  Performing NodeApp Creation Verification Task... ; Wed Dec 14 05:43:44 CST 2011
[main] [5:43:44:502] [sVerificationUtil.getInventoryFileLocation:133]  Inventory Config File's name is:'/etc/oraInst.loc'; Wed Dec 14 05:43:44 CST 2011
[main] [5:43:44:503] [sVerificationUtil.getInventoryFileLocation:168]  inventory_loc=/app/oracle/crs/oraInventory; Wed Dec 14 05:43:44 CST 2011
[main] [5:43:44:503] [sVerificationUtil.getInventoryFileLocation:170]  Inventory File Location is-->/app/oracle/crs/oraInventory/ContentsXML/inventory.xml; Wed Dec 14 05:43:44 CST 2011
[main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208]  CRS found installed  on node: node3; Wed Dec 14 05:43:44 CST 2011
[main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208]  CRS found installed  on node: node2; Wed Dec 14 05:43:44 CST 2011
[main] [5:43:44:504] [VerificationUtil.isCRSInstalled:1208]  CRS found installed  on node: node1; Wed Dec 14 05:43:44 CST 2011
[main] [5:43:44:504] [TaskNodeAppCreation.performTask:178]  ==== Nodes with CRS installed is: 3; Wed Dec 14 05:43:44 CST 2011

..................

Checked the inventory.xml in /app/oracle/crs/oraInventory/ContentsXML/ which showed

{HOME NAME="OraCrs10g_home" LOC="/app/oracle/crs" TYPE="O" IDX="1" CRS="true"}
   {NODE_LIST}
      {NODE NAME="node1"/}
      {NODE NAME="node2"/}
      {NODE NAME="node3"/}
   {/NODE_LIST}
{/HOME}

Added NODE NAME “node4” to the inventory.xml using

node1: (matrix1) /app/oracle/crs/oui/bin> ./runInstaller -updateNodeList -silent "CLUSTER_NODES={node1,node2,node3,node4}" ORACLE_HOME=$CRS_HOME CRS=true

Starting Oracle Universal Installer...

No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /app/oracle/crs/oraInventory
'UpdateNodeList' was successful.

After it got completed, inventory.xml showed

{HOME_LIST}
{HOME NAME="O]raCrs10g_home" LOC="/app/oracle/crs" TYPE="O" IDX="1" CRS="true"}
   {NODE_LIST
      {NODE NAME="node1"/}
      {NODE NAME="node2"/}
      {NODE NAME="node3"/}
      {NODE NAME="node4"/}
   {/NODE_LIST}
{/HOME}
node1: (matrix1) /app/oracle/crs/bin> ./cluvfy stage -post crsinst -n node1,node2,node3,node4 -verbose

Performing post-checks for cluster services setup

..................
..................
Result: Check passed.


Post-check for cluster services setup was successful.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s