Pages

Tuesday 29 October 2013

Failed to start RAC 11gr2 database instance after applying one off patch

When I tried to start the database after applying patch, I got the error as below.

[oracle@test4201 ~]$ srvctl start database -d testms
PRCR-1079 : Failed to start resource ora.testms.db
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.124.148 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4202/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4202' failed
CRS-2632: There are no more servers to try to place resource 'ora.testms.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.167.131 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4201' failed
[oracle@test4201 ~]$ ps -ef|grep pmon
oracle    7221  6673  0 09:38 pts/1    00:00:00 grep --color=auto pmon
[oracle@test4201 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 09:39:04 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.167.131 failed bind. Check output from ifconfig command
SQL> exit

Solution:

I tried by setting cluster_interconnect parameter in init file, but it didn't help.

testms1.cluster_interconnects=10.243.51.15
testms2.cluster_interconnects=10.243.51.34



[oracle@test4201 ~]$ srvctl start database -d testms
PRCR-1079 : Failed to start resource ora.testms.db
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 10.243.51.15 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4201' failed
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 10.243.51.34 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4202/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4202' failed
CRS-2632: There are no more servers to try to place resource 'ora.testms.db' on that would satisfy its placement policy
[oracle@test4201 ~]$ vi /u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log



What mistake I had done is, I had missed post steps which does relinking of oracle binaries. I relinked the binaries on both the nodes with "oracle" user as below.

Note: If you are using rds protocol, command will be make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle . Check the below link on how to check protocol being used.
http://jongsma.wordpress.com/2012/08/08/mixing-udp-and-rds-on-an-exadata/

Node1 :

[oracle@test4201 ~]$ make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_g ioracle
make: Entering directory `/u01/app/oracle/product/11203/testms/rdbms/lib'
rm -f /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
cp /u01/app/oracle/product/11203/testms/lib//libskgxpg.so /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
chmod 755 /u01/app/oracle/product/11203/testms/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/11203/testms/rdbms/lib/oracle
gcc  -o /u01/app/oracle/product/11203/testms/rdbms/lib/oracle -m64 -L/u01/app/oracle/product/11203/testms/rdbms/lib/ -L/u01/app/oracle/product/11203/testms/lib/ -L/u01/app/oracle/product/11203/testms/lib/stubs/   -Wl,-E /u01/app/oracle/product/11203/testms/rdbms/lib/opimai.o /u01/app/oracle/product/11203/testms/rdbms/lib/ssoraed.o /u01/app/oracle/product/11203/testms/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/oracle/product/11203/testms/lib/nautab.o /u01/app/oracle/product/11203/testms/lib/naeet.o /u01/app/oracle/product/11203/testms/lib/naect.o /u01/app/oracle/product/11203/testms/lib/naedhs.o /u01/app/oracle/product/11203/testms/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/oracle/product/11203/testms/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/oracle/product/11203/testms/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/oracle/product/11203/testms/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/11203/testms/lib -lm    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/11203/testms/lib
test ! -f /u01/app/oracle/product/11203/testms/bin/oracle ||\
           mv -f /u01/app/oracle/product/11203/testms/bin/oracle /u01/app/oracle/product/11203/testms/bin/oracleO
mv /u01/app/oracle/product/11203/testms/rdbms/lib/oracle /u01/app/oracle/product/11203/testms/bin/oracle
chmod 6751 /u01/app/oracle/product/11203/testms/bin/oracle
make: Leaving directory `/u01/app/oracle/product/11203/testms/rdbms/lib'

Node2:

[oracle@test4202 ~]$ make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_g ioracle
make: Entering directory `/u01/app/oracle/product/11203/testms/rdbms/lib'
rm -f /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
cp /u01/app/oracle/product/11203/testms/lib//libskgxpg.so /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
chmod 755 /u01/app/oracle/product/11203/testms/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/11203/testms/rdbms/lib/oracle
gcc  -o /u01/app/oracle/product/11203/testms/rdbms/lib/oracle -m64 -L/u01/app/oracle/product/11203/testms/rdbms/lib/ -L/u01/app/oracle/product/11203/testms/lib/ -L/u01/app/oracle/product/11203/testms/lib/stubs/   -Wl,-E /u01/app/oracle/product/11203/testms/rdbms/lib/opimai.o /u01/app/oracle/product/11203/testms/rdbms/lib/ssoraed.o /u01/app/oracle/product/11203/testms/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/oracle/product/11203/testms/lib/nautab.o /u01/app/oracle/product/11203/testms/lib/naeet.o /u01/app/oracle/product/11203/testms/lib/naect.o /u01/app/oracle/product/11203/testms/lib/naedhs.o /u01/app/oracle/product/11203/testms/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/oracle/product/11203/testms/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/oracle/product/11203/testms/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/oracle/product/11203/testms/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/11203/testms/lib -lm    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/11203/testms/lib
test ! -f /u01/app/oracle/product/11203/testms/bin/oracle ||\
           mv -f /u01/app/oracle/product/11203/testms/bin/oracle /u01/app/oracle/product/11203/testms/bin/oracleO
mv /u01/app/oracle/product/11203/testms/rdbms/lib/oracle /u01/app/oracle/product/11203/testms/bin/oracle
chmod 6751 /u01/app/oracle/product/11203/testms/bin/oracle
make: Leaving directory `/u01/app/oracle/product/11203/testms/rdbms/lib'


Now started the database and it came up.

[oracle@test4201 dbs]$ . oraenv
ORACLE_SID = [testms] ? testms1
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/11203/testms
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@test4201 dbs]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 22:31:00 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2235240 bytes
Variable Size             889193624 bytes
Database Buffers         3372220416 bytes
Redo Buffers               12132352 bytes
Database mounted.
Database opened.


ORACLE_SID = [oracle] ? testms2
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/11203/testms
The Oracle base has been set to /u01/app/oracle
[oracle@test4202 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 22:31:34 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2235240 bytes
Variable Size             889193624 bytes
Database Buffers         3372220416 bytes
Redo Buffers               12132352 bytes
Database mounted.
Database opened.
SQL> 

Oracle Home inventory is corrupted in 11gr2 RAC

While checking oracle inventory with opatch, I got error as below.

[oracle@test4201 logs]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/opatch2013-10-29_12-47-27PM_1.log

List of Homes on this system:

  Home name= Ora11g_gridinfrahome1, Location= "/u01/app/11203/grid"
Inventory load failed... OPatch cannot load inventory for the given Oracle Home.
Possible causes are:
   Oracle Home dir. path does not exist in Central Inventory
   Oracle Home is a symbolic link
   Oracle Home inventory is corrupted
LsInventorySession failed: OracleHomeInventory gets null oracleHomeInfo

OPatch failed with error code 73
[oracle@test4201 logs]$


Solution:

Detach the oracle home and attach it again as below.

[oracle@test4201 logs]$ cd $ORACLE_HOME/oui/bin
[oracle@test4201 bin]$ ./runInstaller -detachHome ORACLE_HOME="/u01/app/oracle/product/11203/testms"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16378 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'DetachHome' failed.
[oracle@test4201 bin]$ ./runInstaller -silent -invPtrLoc /u01/app/oraInventory -attachHome ORACLE_HOME="/u01/app/oracle/product/11203/testms" ORACLE_HOME_NAME="OraDb11g_home1"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16378 MB    Passed
The inventory pointer is located at /u01/app/oraInventory
The inventory is located at /u01/app/oraInventory
'AttachHome' was successful.

Check now

[oracle@test4201 bin]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/opatch2013-10-29_12-52-50PM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/lsinv/lsinventory2013-10-29_12-52-50PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.3.0
There are 1 products installed in this Oracle Home.




OPatch failed with error code 41 while applying the database patch in 11gR2 RAC

OPatch failed with error code 41 while applying the database patch in 11gR2 RAC


oracle@test4201 12646746]$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_27_23/apply2013-10-29_05-27-22AM_1.log

Applying interim patch '12646746' to OH '/u01/app/oracle/product/11203/testms'
Verifying environment and performing prerequisite checks...
Patch 12646746: Optional component(s) missing : [ oracle.network.cman, 11.2.0.3.0 ]
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:


Following executables are active :
/u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
Log file location: /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_27_23/apply2013-10-29_05-27-22AM_1.log

Recommended actions: OPatch needs to modify files which are being used by some processes.

OPatch failed with error code 41


In log it's showing that

36 [Oct 29, 2013 5:27:27 AM]    Start fuser command /sbin/fuser /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 at Tue Oct 29 05:27:27 UTC 2013
 37 [Oct 29, 2013 5:27:27 AM]    Finish fuser command /sbin/fuser /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 at Tue Oct 29 05:27:27 UTC 2013
 38 [Oct 29, 2013 5:27:27 AM]    Following executables are active :
 39                              /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
 40 [Oct 29, 2013 5:27:27 AM]    Prerequisite check "CheckActiveFilesAndExecutables" failed.
 41                              The details are:
 42
 43
 44                              Following executables are active :
 45                              /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
 46 [Oct 29, 2013 5:27:27 AM]    OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
 47                              You will be still able to rollback patches after this cleanup.
 48                              Do you want to proceed? [y|n]
 49 [Oct 29, 2013 5:27:30 AM]    Y (auto-answered by -silent)
 50 [Oct 29, 2013 5:27:30 AM]    User Responded with: Y
 51 [Oct 29, 2013 5:27:30 AM]    Size of directory "/u01/app/oracle/product/11203/testms/.patch_storage" before cleanup is 282613767 bytes.
 52 [Oct 29, 2013 5:27:30 AM]    Deleting the directory "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/backup"
 53 [Oct 29, 2013 5:27:30 AM]    Deleted the directory "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/backup"
 54 [Oct 29, 2013 5:27:30 AM]    Size of directory "/u01/app/oracle/product/11203/testms/.patch_storage" after cleanup is 282613767 bytes.
 55 [Oct 29, 2013 5:27:30 AM]    UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
 56                              deleted, Please refer log file.
 57 [Oct 29, 2013 5:27:30 AM]    [ Error during Prerequisite for apply phase ] Detail:                              OUI-67074:ApplySession failed during prerequisite checks: Prerequisite check "Che
 58 [Oct 29, 2013 5:27:30 AM]    OUI-67035:System is intact, OPatch will not restore the system
 59 [Oct 29, 2013 5:27:30 AM]    Finishing ApplySession at Tue Oct 29 05:27:30 UTC 2013

Solution:

1.Identify the process which is accessing /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 executable.



[oracle@test4201 bp17]$ /sbin/fuser -v /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1

                     USER        PID ACCESS COMMAND
/u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1:
                     oracle    23612 ....m emagent

[oracle@test4201 bp17]$ ps -ef|grep 23612
oracle   22753 21386  0 22:31 pts/6    00:00:00 grep --color=auto 23612
oracle   23612 12053  0 10:13 pts/11   00:00:39 /u01/app/oracle/product/11203/testms/bin/emagent

In my case, it's found that EM agent was running so I killed that process as it was test environment. In production environment, check the status and stop the EM agent.

[oracle@test4201 bin]$ kill -9 23612
[oracle@test4201 bin]$ /sbin/fuser -v /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
[oracle@test4201 bin]$ pwd

2. Now try to apply the patch.


[oracle@test4201 12646746]$ opatch apply                                                                                                                                                            Oracle Interim Patch Installer version 11.2.0.3.5                                                                                                                                                    Copyright (c) 2013, Oracle Corporation.  All rights reserved.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Oracle Home       : /u01/app/oracle/product/11203/testms                                                                                                                                            Central Inventory : /u01/app/oraInventory                                                                                                                                                               from           : /u01/app/oracle/product/11203/testms/oraInst.loc                                                                                                                                OPatch version    : 11.2.0.3.5                                                                                                                                                                       OUI version       : 11.2.0.3.0                                                                                                                                                                       Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_44_58/apply2013-10-29_05-44-58AM_1.log                                                                                                                                                                                                                                                               Applying interim patch '12646746' to OH '/u01/app/oracle/product/11203/testms'                                                                                                                      Verifying environment and performing prerequisite checks...                                                                                                                                          Patch 12646746: Optional component(s) missing : [ oracle.network.cman, 11.2.0.3.0 ]                                                                                                                  All checks passed.                                                                                                                                                                                   Provide your email address to be informed of security issues, install and                                                                                                                            initiate Oracle Configuration Manager. Easier for you if you use your My                                                                                                                             Oracle Support Email address/User Name.                                                                                                                                                              Visit http://www.oracle.com/support/policies.html for details.                                                                                                                                       Email address/User Name:                                                                                                                                                                                                                                                                                                                                                                                  You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y



This node is part of an Oracle Real Application Cluster.
Remote nodes: 'test4202'
Local node: 'test4201'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11203/testms')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...

Patching component oracle.network.rsf, 11.2.0.3.0...

Patching component oracle.rdbms, 11.2.0.3.0...

Patching component oracle.rdbms.rsf, 11.2.0.3.0...

Verifying the update...

The local system has been patched.  You can restart Oracle instances on it.


Patching in rolling mode.


The node 'test4202' will be patched next.


Please shutdown Oracle instances running out of this ORACLE_HOME on 'test4202'.
(Oracle Home = '/u01/app/oracle/product/11203/testms')

Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'test4202'
   Apply-related files are:
     FP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt"
     DP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt"
     MP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt"
     RC = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/remote_cmds.txt"

Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt" with actual path.
Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2

Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/network/lib; /usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2


The node 'test4202' has been patched.  You can restart Oracle instances on it.

There were relinks on remote nodes.  Remember to check the binary size and timestamp on the nodes 'test4202' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11203/testms
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms
cd /u01/app/oracle/product/11203/testms/network/lib; /usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms
'

Patch 12646746 successfully applied
Log file location: /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_44_58/apply2013-10-29_05-44-58AM_1.log

OPatch succeeded.
[oracle@test4201 12646746]$ cd ..

OPatch failed with error code 73



I tried to apply the patch to database instance and got the error as below.

[oracle@test01db 13936066]$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testdbms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testdbms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/13936066_Oct_29_2013_09_24_25/apply2013-10-29_09-24-25AM_1.log

Applying interim patch '13936066' to OH '/u01/app/oracle/product/11203/testdbms'
Verifying environment and performing prerequisite checks...
[ Error during Oracle Home discovery Phase]. Detail: OPatchSession cannot load inventory for the given Oracle Home /u01/app/oracle/product/11203/testdbms. Possible causes are:
   No read or write permission to ORACLE_HOME/.patch_storage
   Central Inventory is locked by another OUI instance
   No read permission to Central Inventory
   The lock file exists in ORACLE_HOME/.patch_storage
   The Oracle Home does not exist in Central Inventory

[ Error during Oracle Home discovery Phase]. Detail: OPatch failed: ApplySession failed to prepare the system. Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/16311483 is corrupted
   No read permission to above directory
Please remove one-off entry [ 16311483 ] from /u01/app/oracle/product/11203/testdbms/inventory/ContentsXML/comps.xml and retry if above directory corrupted, or Please check the contents of the directory ORACLE_HOME/inventory/oneoffs/16311483 for read permission
Log file location: /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/13936066_Oct_29_2013_09_24_25/apply2013-10-29_09-24-25AM_1.log

Recommended actions: Please make sure no other OPatch or OUI processes is running. Try running $ORACLE_HOME/oui/bin/runInstsaller.

OPatch failed with error code 22


When I checked the inventory, got the error as below.

[oracle@test01db ContentsXML]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testdbms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testdbms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/opatch2013-10-29_09-50-44AM_1.log



Inventory load failed... OPatch cannot load inventory for the given Oracle Home.

LsInventorySession failed: Unable to create patchObject
Possible causes are:
   ORACLE_HOME/inventory/oneoffs/16311483 is corrupted
   No read permission to above directory
Please remove one-off entry [ 16311483 ] from /u01/app/oracle/product/11203/testdbms/inventory/ContentsXML/comps.xml and retry if above directory corrupted, or Please check the contents of the directory ORACLE_HOME/inventory/oneoffs/16311483 for read permission

OPatch failed with error code 73



So when I checked the patch 16311483 at $ORACLE_HOME/inventory/oneoffs, couldn't find it.

[oracle@test01db ContentsXML]$ ls -lrt $ORACLE_HOME/inventory/oneoffs/16311483
ls: /u01/app/oracle/product/11203/testdbms/inventory/oneoffs/16311483: No such file or directory
[oracle@test01db ContentsXML]$


Solution:

Take the backup of /u01/app/oracle/product/11203/testdbms/inventory/ContentsXML/comps.xml file and then remove  "16311483" corresponding entry(from ) from comps.xml. After that check the inventory as below.


[oracle@test01db ContentsXML]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/11203/testdbms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/11203/testdbms/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/opatch2013-10-29_10-13-01AM_1.log

Lsinventory Output file location : /u01/app/oracle/product/11203/testdbms/cfgtoollogs/opatch/lsinv/lsinventory2013-10-29_10-13-01AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.3.0
There are 1 products installed in this Oracle Home.


Interim patches (1) :

Patch  12646746     : applied on Tue Oct 29 05:45:31 UTC 2013
Unique Patch ID:  14825629
   Created on 14 May 2012, 16:17:31 hrs PST8PDT
   Bugs fixed:
     12646746



--------------------------------------------------------------------------------

OPatch succeeded.


Thursday 24 October 2013

Configure SSH Keys for RAC on Linux to establish user equivalance

1.Configure SSH on each node in the cluster. Log in as the "oracle" user and perform the following tasks on each RAC node.

[oracle@test4202 ~]$ mkdir ~/.ssh
[oracle@test4202 ~]$ ls -lart
total 28
-rw-r--r--  1 oracle oinstall  124 Oct  8 12:41 .bashrc
-rw-r--r--  1 oracle oinstall  176 Oct  8 12:41 .bash_profile
-rw-r--r--  1 oracle oinstall   33 Oct  8 12:41 .bash_logout
drwxr-xr-x 14 root   root     4096 Oct 24 08:55 ..
-rw-------  1 oracle oinstall 1736 Oct 24 09:05 .bash_history
drwxr-xr-x  2 oracle oinstall 4096 Oct 24 10:12 .ssh
drwx------  3 oracle oinstall 4096 Oct 24 10:12 .
[oracle@test4202 ~]$ chmod 700 ~/.ssh
[oracle@test4202 ~]$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
2e:83:65:b5:c6:0d:11:b1:db:a6:a5:98:bd:85:6d:25 oracle@test4202
The key's randomart image is:
+--[ RSA 2048]----+
|        +o       |
|         o       |
|        +        |
|       o *       |
|      o S E .    |
|     + * B o     |
|    . = * +      |
|       o +       |
|        .        |
+-----------------+
[oracle@test4202 ~]$ cd /home/oracle/.ssh/
[oracle@test4202 .ssh]$ ls -lrt
total 8
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:13 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Oct 24 10:13 id_rsa


Note:The RSA public key is written to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file.


2.Generate authorized_keys

Log in as "oracle" user on node1, generate an "authorized_keys" file on node1 and then copy it to node2 as below.

cd /home/oracle/.ssh/

[oracle@test4201 .ssh]$ ls -lrt
total 8
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:10 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Oct 24 10:10 id_rsa
[oracle@test4201 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@test4201 .ssh]$ ls -lrt
total 12
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:10 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Oct 24 10:10 id_rsa
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:11 authorized_keys
[oracle@test4201 .ssh]$ scp authorized_keys oracle@test4202:/home/oracle/.ssh/
Password:
authorized_keys                                                                                                                                          100%  398     0.4KB/s   0.4KB/s   00:00

Next, log in as the "oracle" user on node2 and perform as below.

[oracle@test4202 tmp]$ cd /home/oracle/.ssh/
[oracle@test4202 .ssh]$ ls -rlt
total 12
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:13 id_rsa.pub
-rw------- 1 oracle oinstall 1675 Oct 24 10:13 id_rsa
-rw-r--r-- 1 oracle oinstall  398 Oct 24 10:14 authorized_keys
[oracle@test4202 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0iKAmCxhxPJg+VLCHZuSzvix74Dw79ze5UJdx54hTfQ70Z/orYrTrOimSf0INSqBmqpiofTqu8W0Wu9acGtunsdJe++f2Ew58slF8svHCXaKyIMxBQheg7uCKEoDaaXbiZAD/30fGKAO5nL5IVZd9Pc7TVufutZbwGknbZxKwIXoFOUFDW3D0EkMiuFFiItWNjLoQvAx84hW5wJ2ImWc0mOSt2NDTk2O4pZ7plxeEYqr2Gy/bTwNba40r1ig7fwsRHh1j1t/cRYxY2s9IqlZeISynb6NaE4PmNHINypfuXb8f8kPt/g/GU8hsW8gbTBsnZPKDr+tdEDQjbnFk1eYjQ== oracle@test4201
[oracle@test4202 .ssh]$ cat id_rsa.pub >> authorized_keys
[oracle@test4202 .ssh]$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA0iKAmCxhxPJg+VLCHZuSzvix74Dw79ze5UJdx54hTfQ70Z/orYrTrOimSf0INSqBmqpiofTqu8W0Wu9acGtunsdJe++f2Ew58slF8svHCXaKyIMxBQheg7uCKEoDaaXbiZAD/30fGKAO5nL5IVZd9Pc7TVufutZbwGknbZxKwIXoFOUFDW3D0EkMiuFFiItWNjLoQvAx84hW5wJ2ImWc0mOSt2NDTk2O4pZ7plxeEYqr2Gy/bTwNba40r1ig7fwsRHh1j1t/cRYxY2s9IqlZeISynb6NaE4PmNHINypfuXb8f8kPt/g/GU8hsW8gbTBsnZPKDr+tdEDQjbnFk1eYjQ== oracle@test4201
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtdMGC7W1aGAyKuxR5HO2JsTprL3INqp4AGw5uBS3OyeA0pyrfqA+zB7MTxBTi7apJYqaGdVpTsmdawGQ46JQbfXu6Ob78KsQDHdJupzeSw5iBmCcsnzZ54LE/8Qv5FIh9R8HQH8Z/Yx69CfeRK8jPBPOPqdw9rHUhbpoy6u5PZhFgApEZyKM3IYqAU+ty/4lzeDAf/y+uBwK5kSRWzs5fAoSrWjzhyrzNERqN22pN8+OyYm295IPi2cDRb7NnInZdqfrTzWHpwzqrJL/88GTFWkgpC8oJp9sVl/v5KAJtnQCaXpyOsZraNwcZxyO07tIJLFcW0PPNMyk46vegO4tmw== oracle@test4202



[oracle@test4202 .ssh]$ scp authorized_keys oracle@test4201:/home/oracle/.ssh/
The authenticity of host 'test4201 (152.69.88.127)' can't be established.
RSA key fingerprint is a4:a5:a8:ef:6f:72:b1:02:ae:f5:27:9c:2f:a4:4c:16.
Are you sure you want to continue connecting (yes/no)? yes
Password:
authorized_keys                                                                                                                                          100%  796     0.8KB/s   0.8KB/s   00:00


3.Test the setup as below

Node1
[oracle@test4201 .ssh]$ ssh test4201 date
The authenticity of host 'test4201 (152.69.88.127)' can't be established.
RSA key fingerprint is a4:a5:a8:ef:6f:72:b1:02:ae:f5:27:9c:2f:a4:4c:16.
Are you sure you want to continue connecting (yes/no)? yes
Thu Oct 24 10:17:15 PDT 2013
[oracle@test4201 .ssh]$ ssh test4202 date
Thu Oct 24 10:17:29 PDT 2013
[oracle@test4201 .ssh]$


Node2
[oracle@test4201 .ssh]$ ssh test4201 date
The authenticity of host 'test4201 (152.69.88.127)' can't be established.
RSA key fingerprint is a4:a5:a8:ef:6f:72:b1:02:ae:f5:27:9c:2f:a4:4c:16.
Are you sure you want to continue connecting (yes/no)? yes
Thu Oct 24 10:17:15 PDT 2013
[oracle@test4201 .ssh]$ ssh test4202 date
Thu Oct 24 10:17:29 PDT 2013
[oracle@test4201 .ssh]$

Note:  You can also run the sshUserSetup.sh script available at /grid/sshsetup to setup user equivalance

Friday 18 October 2013

RMAN-06004: ORACLE error from recovery catalog database: RMAN-20001: target database not found in recovery catalog

I got error as below while doing resync of the database with catalog.

connected to target database: TESTDB (DBID=2269473865)
connected to recovery catalog database
PL/SQL package RMAN_UINLUPP.DBMS_RCVCAT version 11.02.00.02 in RCVCAT database is not current
PL/SQL package RMAN_UINLUPP.DBMS_RCVMAN version 11.02.00.02 in RCVCAT database is not current

RMAN> RUN
2> {
3> RESYNC CATALOG;
4> REPORT SCHEMA;
5> }
6> LIST BACKUP SUMMARY;
7>
8>

DBGSQL:     RCVCAT> begin dbms_rcvman.dumpPkgState('RCVMAN after sqlerror'); end;
DBGSQL:        sqlcode = 6550

DBGSQL:     RCVCAT> begin dbms_rcvcat.dumpPkgState('RCVCAT after sqlerror');end;
DBGSQL:        sqlcode = 6550
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of resync command at 10/18/2013 02:01:30
RMAN-06004: ORACLE error from recovery catalog database: RMAN-20001: target database not found in recovery catalog

Recovery Manager complete.

Reason:Database was not registered in the catalog. Register the target database and check it.

[oracle@uiddx4101 log]$ rman target / catalog rman_cat/rman_cat@rman

Recovery Manager: Release 11.2.0.3.0 - Production on Fri Oct 18 13:25:06 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: TESTDB (DBID=2269473865)
connected to recovery catalog database

RMAN> RESYNC CATALOG;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of resync command at 10/18/2013 13:25:12
RMAN-06004: ORACLE error from recovery catalog database: RMAN-20001: target database not found in recovery catalog



RMAN> register database;

database registered in recovery catalog
starting full resync of recovery catalog


full resync complete

RMAN>
RMAN>
RMAN> RESYNC CATALOG;

starting full resync of recovery catalog
full resync complete

RMAN> exit