OPatch failed with error code 41 while applying the database patch in 11gR2 RAC
oracle@test4201 12646746]$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11203/testms
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11203/testms/oraInst.loc
OPatch version : 11.2.0.3.5
OUI version : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_27_23/apply2013-10-29_05-27-22AM_1.log
Applying interim patch '12646746' to OH '/u01/app/oracle/product/11203/testms'
Verifying environment and performing prerequisite checks...
Patch 12646746: Optional component(s) missing : [ oracle.network.cman, 11.2.0.3.0 ]
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
Log file location: /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_27_23/apply2013-10-29_05-27-22AM_1.log
Recommended actions: OPatch needs to modify files which are being used by some processes.
OPatch failed with error code 41
In log it's showing that
36 [Oct 29, 2013 5:27:27 AM] Start fuser command /sbin/fuser /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 at Tue Oct 29 05:27:27 UTC 2013
37 [Oct 29, 2013 5:27:27 AM] Finish fuser command /sbin/fuser /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 at Tue Oct 29 05:27:27 UTC 2013
38 [Oct 29, 2013 5:27:27 AM] Following executables are active :
39 /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
40 [Oct 29, 2013 5:27:27 AM] Prerequisite check "CheckActiveFilesAndExecutables" failed.
41 The details are:
42
43
44 Following executables are active :
45 /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
46 [Oct 29, 2013 5:27:27 AM] OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
47 You will be still able to rollback patches after this cleanup.
48 Do you want to proceed? [y|n]
49 [Oct 29, 2013 5:27:30 AM] Y (auto-answered by -silent)
50 [Oct 29, 2013 5:27:30 AM] User Responded with: Y
51 [Oct 29, 2013 5:27:30 AM] Size of directory "/u01/app/oracle/product/11203/testms/.patch_storage" before cleanup is 282613767 bytes.
52 [Oct 29, 2013 5:27:30 AM] Deleting the directory "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/backup"
53 [Oct 29, 2013 5:27:30 AM] Deleted the directory "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/backup"
54 [Oct 29, 2013 5:27:30 AM] Size of directory "/u01/app/oracle/product/11203/testms/.patch_storage" after cleanup is 282613767 bytes.
55 [Oct 29, 2013 5:27:30 AM] UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
56 deleted, Please refer log file.
57 [Oct 29, 2013 5:27:30 AM] [ Error during Prerequisite for apply phase ] Detail: OUI-67074:ApplySession failed during prerequisite checks: Prerequisite check "Che
58 [Oct 29, 2013 5:27:30 AM] OUI-67035:System is intact, OPatch will not restore the system
59 [Oct 29, 2013 5:27:30 AM] Finishing ApplySession at Tue Oct 29 05:27:30 UTC 2013
Solution:
1.Identify the process which is accessing /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 executable.
[oracle@test4201 bp17]$ /sbin/fuser -v /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
USER PID ACCESS COMMAND
/u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1:
oracle 23612 ....m emagent
[oracle@test4201 bp17]$ ps -ef|grep 23612
oracle 22753 21386 0 22:31 pts/6 00:00:00 grep --color=auto 23612
oracle 23612 12053 0 10:13 pts/11 00:00:39 /u01/app/oracle/product/11203/testms/bin/emagent
In my case, it's found that EM agent was running so I killed that process as it was test environment. In production environment, check the status and stop the EM agent.
[oracle@test4201 bin]$ kill -9 23612
[oracle@test4201 bin]$ /sbin/fuser -v /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
[oracle@test4201 bin]$ pwd
2. Now try to apply the patch.
[oracle@test4201 12646746]$ opatch apply Oracle Interim Patch Installer version 11.2.0.3.5 Copyright (c) 2013, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/11203/testms Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/11203/testms/oraInst.loc OPatch version : 11.2.0.3.5 OUI version : 11.2.0.3.0 Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_44_58/apply2013-10-29_05-44-58AM_1.log Applying interim patch '12646746' to OH '/u01/app/oracle/product/11203/testms' Verifying environment and performing prerequisite checks... Patch 12646746: Optional component(s) missing : [ oracle.network.cman, 11.2.0.3.0 ] All checks passed. Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name: You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'test4202'
Local node: 'test4201'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11203/testms')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Patching component oracle.network.rsf, 11.2.0.3.0...
Patching component oracle.rdbms, 11.2.0.3.0...
Patching component oracle.rdbms.rsf, 11.2.0.3.0...
Verifying the update...
The local system has been patched. You can restart Oracle instances on it.
Patching in rolling mode.
The node 'test4202' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'test4202'.
(Oracle Home = '/u01/app/oracle/product/11203/testms')
Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'test4202'
Apply-related files are:
FP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt"
DP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt"
RC = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/remote_cmds.txt"
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt" with actual path.
Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/network/lib; /usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2
The node 'test4202' has been patched. You can restart Oracle instances on it.
There were relinks on remote nodes. Remember to check the binary size and timestamp on the nodes 'test4202' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11203/testms
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms
cd /u01/app/oracle/product/11203/testms/network/lib; /usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms
'
Patch 12646746 successfully applied
Log file location: /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_44_58/apply2013-10-29_05-44-58AM_1.log
OPatch succeeded.
[oracle@test4201 12646746]$ cd ..
oracle@test4201 12646746]$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11203/testms
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11203/testms/oraInst.loc
OPatch version : 11.2.0.3.5
OUI version : 11.2.0.3.0
Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_27_23/apply2013-10-29_05-27-22AM_1.log
Applying interim patch '12646746' to OH '/u01/app/oracle/product/11203/testms'
Verifying environment and performing prerequisite checks...
Patch 12646746: Optional component(s) missing : [ oracle.network.cman, 11.2.0.3.0 ]
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
Log file location: /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_27_23/apply2013-10-29_05-27-22AM_1.log
Recommended actions: OPatch needs to modify files which are being used by some processes.
OPatch failed with error code 41
In log it's showing that
36 [Oct 29, 2013 5:27:27 AM] Start fuser command /sbin/fuser /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 at Tue Oct 29 05:27:27 UTC 2013
37 [Oct 29, 2013 5:27:27 AM] Finish fuser command /sbin/fuser /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 at Tue Oct 29 05:27:27 UTC 2013
38 [Oct 29, 2013 5:27:27 AM] Following executables are active :
39 /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
40 [Oct 29, 2013 5:27:27 AM] Prerequisite check "CheckActiveFilesAndExecutables" failed.
41 The details are:
42
43
44 Following executables are active :
45 /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
46 [Oct 29, 2013 5:27:27 AM] OPatch will clean up 'restore.sh,make.txt' files and 'rac,scratch,backup' directories.
47 You will be still able to rollback patches after this cleanup.
48 Do you want to proceed? [y|n]
49 [Oct 29, 2013 5:27:30 AM] Y (auto-answered by -silent)
50 [Oct 29, 2013 5:27:30 AM] User Responded with: Y
51 [Oct 29, 2013 5:27:30 AM] Size of directory "/u01/app/oracle/product/11203/testms/.patch_storage" before cleanup is 282613767 bytes.
52 [Oct 29, 2013 5:27:30 AM] Deleting the directory "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/backup"
53 [Oct 29, 2013 5:27:30 AM] Deleted the directory "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/backup"
54 [Oct 29, 2013 5:27:30 AM] Size of directory "/u01/app/oracle/product/11203/testms/.patch_storage" after cleanup is 282613767 bytes.
55 [Oct 29, 2013 5:27:30 AM] UtilSession: Backup area for restore has been cleaned up. For a complete list of files/directories
56 deleted, Please refer log file.
57 [Oct 29, 2013 5:27:30 AM] [ Error during Prerequisite for apply phase ] Detail: OUI-67074:ApplySession failed during prerequisite checks: Prerequisite check "Che
58 [Oct 29, 2013 5:27:30 AM] OUI-67035:System is intact, OPatch will not restore the system
59 [Oct 29, 2013 5:27:30 AM] Finishing ApplySession at Tue Oct 29 05:27:30 UTC 2013
Solution:
1.Identify the process which is accessing /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1 executable.
[oracle@test4201 bp17]$ /sbin/fuser -v /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
USER PID ACCESS COMMAND
/u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1:
oracle 23612 ....m emagent
[oracle@test4201 bp17]$ ps -ef|grep 23612
oracle 22753 21386 0 22:31 pts/6 00:00:00 grep --color=auto 23612
oracle 23612 12053 0 10:13 pts/11 00:00:39 /u01/app/oracle/product/11203/testms/bin/emagent
In my case, it's found that EM agent was running so I killed that process as it was test environment. In production environment, check the status and stop the EM agent.
[oracle@test4201 bin]$ kill -9 23612
[oracle@test4201 bin]$ /sbin/fuser -v /u01/app/oracle/product/11203/testms/lib/libclntsh.so.11.1
[oracle@test4201 bin]$ pwd
2. Now try to apply the patch.
[oracle@test4201 12646746]$ opatch apply Oracle Interim Patch Installer version 11.2.0.3.5 Copyright (c) 2013, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/oracle/product/11203/testms Central Inventory : /u01/app/oraInventory from : /u01/app/oracle/product/11203/testms/oraInst.loc OPatch version : 11.2.0.3.5 OUI version : 11.2.0.3.0 Log file location : /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_44_58/apply2013-10-29_05-44-58AM_1.log Applying interim patch '12646746' to OH '/u01/app/oracle/product/11203/testms' Verifying environment and performing prerequisite checks... Patch 12646746: Optional component(s) missing : [ oracle.network.cman, 11.2.0.3.0 ] All checks passed. Provide your email address to be informed of security issues, install and initiate Oracle Configuration Manager. Easier for you if you use your My Oracle Support Email address/User Name. Visit http://www.oracle.com/support/policies.html for details. Email address/User Name: You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'test4202'
Local node: 'test4201'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11203/testms')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Patching component oracle.network.rsf, 11.2.0.3.0...
Patching component oracle.rdbms, 11.2.0.3.0...
Patching component oracle.rdbms.rsf, 11.2.0.3.0...
Verifying the update...
The local system has been patched. You can restart Oracle instances on it.
Patching in rolling mode.
The node 'test4202' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'test4202'.
(Oracle Home = '/u01/app/oracle/product/11203/testms')
Is the node ready for patching? [y|n]
y
User Responded with: Y
Updating nodes 'test4202'
Apply-related files are:
FP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt"
DP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt"
MP = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt"
RC = "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/remote_cmds.txt"
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
Instantiating the file "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in "/u01/app/oracle/product/11203/testms/.patch_storage/12646746_May_14_2012_16_17_31/rac/make_cmds.txt" with actual path.
Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2
Running command on remote node 'test4202':
cd /u01/app/oracle/product/11203/testms/network/lib; /usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms || echo REMOTE_MAKE_FAILED::>&2
The node 'test4202' has been patched. You can restart Oracle instances on it.
There were relinks on remote nodes. Remember to check the binary size and timestamp on the nodes 'test4202' .
The following make commands were invoked on remote nodes:
'cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/11203/testms
cd /u01/app/oracle/product/11203/testms/rdbms/lib; /usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms
cd /u01/app/oracle/product/11203/testms/network/lib; /usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=/u01/app/oracle/product/11203/testms
'
Patch 12646746 successfully applied
Log file location: /u01/app/oracle/product/11203/testms/cfgtoollogs/opatch/12646746_Oct_29_2013_05_44_58/apply2013-10-29_05-44-58AM_1.log
OPatch succeeded.
[oracle@test4201 12646746]$ cd ..
No comments:
Post a Comment