Pages

Tuesday 29 October 2013

Failed to start RAC 11gr2 database instance after applying one off patch

When I tried to start the database after applying patch, I got the error as below.

[oracle@test4201 ~]$ srvctl start database -d testms
PRCR-1079 : Failed to start resource ora.testms.db
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.124.148 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4202/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4202' failed
CRS-2632: There are no more servers to try to place resource 'ora.testms.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.167.131 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4201' failed
[oracle@test4201 ~]$ ps -ef|grep pmon
oracle    7221  6673  0 09:38 pts/1    00:00:00 grep --color=auto pmon
[oracle@test4201 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 09:39:04 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 169.254.167.131 failed bind. Check output from ifconfig command
SQL> exit

Solution:

I tried by setting cluster_interconnect parameter in init file, but it didn't help.

testms1.cluster_interconnects=10.243.51.15
testms2.cluster_interconnects=10.243.51.34



[oracle@test4201 ~]$ srvctl start database -d testms
PRCR-1079 : Failed to start resource ora.testms.db
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 10.243.51.15 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4201' failed
CRS-5017: The resource action "ora.testms.db start" encountered the following error:
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:bind_fail failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvifconf
ORA-27303: additional information: requested interface 10.243.51.34 failed bind. Check output from ifconfig command
. For details refer to "(:CLSN00107:)" in "/u01/app/11203/grid/log/test4202/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.testms.db' on 'test4202' failed
CRS-2632: There are no more servers to try to place resource 'ora.testms.db' on that would satisfy its placement policy
[oracle@test4201 ~]$ vi /u01/app/11203/grid/log/test4201/agent/crsd/oraagent_oracle/oraagent_oracle.log



What mistake I had done is, I had missed post steps which does relinking of oracle binaries. I relinked the binaries on both the nodes with "oracle" user as below.

Note: If you are using rds protocol, command will be make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle . Check the below link on how to check protocol being used.
http://jongsma.wordpress.com/2012/08/08/mixing-udp-and-rds-on-an-exadata/

Node1 :

[oracle@test4201 ~]$ make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_g ioracle
make: Entering directory `/u01/app/oracle/product/11203/testms/rdbms/lib'
rm -f /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
cp /u01/app/oracle/product/11203/testms/lib//libskgxpg.so /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
chmod 755 /u01/app/oracle/product/11203/testms/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/11203/testms/rdbms/lib/oracle
gcc  -o /u01/app/oracle/product/11203/testms/rdbms/lib/oracle -m64 -L/u01/app/oracle/product/11203/testms/rdbms/lib/ -L/u01/app/oracle/product/11203/testms/lib/ -L/u01/app/oracle/product/11203/testms/lib/stubs/   -Wl,-E /u01/app/oracle/product/11203/testms/rdbms/lib/opimai.o /u01/app/oracle/product/11203/testms/rdbms/lib/ssoraed.o /u01/app/oracle/product/11203/testms/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/oracle/product/11203/testms/lib/nautab.o /u01/app/oracle/product/11203/testms/lib/naeet.o /u01/app/oracle/product/11203/testms/lib/naect.o /u01/app/oracle/product/11203/testms/lib/naedhs.o /u01/app/oracle/product/11203/testms/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/oracle/product/11203/testms/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/oracle/product/11203/testms/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/oracle/product/11203/testms/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/11203/testms/lib -lm    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/11203/testms/lib
test ! -f /u01/app/oracle/product/11203/testms/bin/oracle ||\
           mv -f /u01/app/oracle/product/11203/testms/bin/oracle /u01/app/oracle/product/11203/testms/bin/oracleO
mv /u01/app/oracle/product/11203/testms/rdbms/lib/oracle /u01/app/oracle/product/11203/testms/bin/oracle
chmod 6751 /u01/app/oracle/product/11203/testms/bin/oracle
make: Leaving directory `/u01/app/oracle/product/11203/testms/rdbms/lib'

Node2:

[oracle@test4202 ~]$ make -C $ORACLE_HOME/rdbms/lib -f ins_rdbms.mk ipc_g ioracle
make: Entering directory `/u01/app/oracle/product/11203/testms/rdbms/lib'
rm -f /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
cp /u01/app/oracle/product/11203/testms/lib//libskgxpg.so /u01/app/oracle/product/11203/testms/lib/libskgxp11.so
chmod 755 /u01/app/oracle/product/11203/testms/bin

 - Linking Oracle
rm -f /u01/app/oracle/product/11203/testms/rdbms/lib/oracle
gcc  -o /u01/app/oracle/product/11203/testms/rdbms/lib/oracle -m64 -L/u01/app/oracle/product/11203/testms/rdbms/lib/ -L/u01/app/oracle/product/11203/testms/lib/ -L/u01/app/oracle/product/11203/testms/lib/stubs/   -Wl,-E /u01/app/oracle/product/11203/testms/rdbms/lib/opimai.o /u01/app/oracle/product/11203/testms/rdbms/lib/ssoraed.o /u01/app/oracle/product/11203/testms/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/oracle/product/11203/testms/lib/nautab.o /u01/app/oracle/product/11203/testms/lib/naeet.o /u01/app/oracle/product/11203/testms/lib/naect.o /u01/app/oracle/product/11203/testms/lib/naedhs.o /u01/app/oracle/product/11203/testms/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/oracle/product/11203/testms/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/oracle/product/11203/testms/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/oracle/product/11203/testms/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/oracle/product/11203/testms/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/oracle/product/11203/testms/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/11203/testms/lib -lm    `cat /u01/app/oracle/product/11203/testms/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/11203/testms/lib
test ! -f /u01/app/oracle/product/11203/testms/bin/oracle ||\
           mv -f /u01/app/oracle/product/11203/testms/bin/oracle /u01/app/oracle/product/11203/testms/bin/oracleO
mv /u01/app/oracle/product/11203/testms/rdbms/lib/oracle /u01/app/oracle/product/11203/testms/bin/oracle
chmod 6751 /u01/app/oracle/product/11203/testms/bin/oracle
make: Leaving directory `/u01/app/oracle/product/11203/testms/rdbms/lib'


Now started the database and it came up.

[oracle@test4201 dbs]$ . oraenv
ORACLE_SID = [testms] ? testms1
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/11203/testms
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@test4201 dbs]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 22:31:00 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2235240 bytes
Variable Size             889193624 bytes
Database Buffers         3372220416 bytes
Redo Buffers               12132352 bytes
Database mounted.
Database opened.


ORACLE_SID = [oracle] ? testms2
ORACLE_HOME = [/home/oracle] ? /u01/app/oracle/product/11203/testms
The Oracle base has been set to /u01/app/oracle
[oracle@test4202 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.3.0 Production on Tue Oct 29 22:31:34 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 4275781632 bytes
Fixed Size                  2235240 bytes
Variable Size             889193624 bytes
Database Buffers         3372220416 bytes
Redo Buffers               12132352 bytes
Database mounted.
Database opened.
SQL> 

No comments:

Post a Comment