给几个测试库打补丁，停完监听和实例后查看已经没有Oracle的进程了，于是执行opatch apply。但执行报如下错误：
$ opatch apply
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2017, Oracle Corporation. All rights reserved.
Oracle Home : /opt/oracle/app/oracle/product/11.2.4/db_1
Central Inventory : /opt/oracle/app/oracle/oraInventory
from : /opt/oracle/app/oracle/product/11.2.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.12
OUI version : 11.2.0.4.0
Log file location : /opt/oracle/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/opatch2017-03-29_00-16-10AM_1.log
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/opt/oracle/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
Log file location: /opt/oracle/app/oracle/product/11.2.4/db_1/cfgtoollogs/opatch/opatch2017-03-29_00-16-10AM_1.log
Oracle的相关进程已经没有了
[bbtst3s oracle@tst3 /home/oracle]
$ ps -ef |grep ora
oracle 8608 8606 0 02:04:59 ? 0:00 sshd: oracle@pts/2
oracle 8796 19274 0 02:05:24 ? 0:00 sleep 15
oracle 19274 1 3 Dec 20 ? 967:38 /bin/sh ./OSWatcher.sh 15 192
oracle 8825 8610 0 02:05:35 pts/2 0:00 grep ora
oracle 8668 19319 0 02:05:03 ? 0:00 sleep 60
oracle 8824 8610 2 02:05:35 pts/2 0:00 ps -ef
oracle 8610 8608 0 02:04:59 pts/2 0:00 -ksh
oracle 19319 19274 0 Dec 20 ? 23:31 /bin/sh ./OSWatcherFM.sh 192 /opt/bboss/tst3/oracle/arch/oswatcher/oswbb/archive
root 8606 21662 0 02:04:56 ? 0:00 sshd: oracle [priv]
于是使用fuser查看
$ fuser /opt/oracle/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1
/opt/oracle/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1: 29888m
看到文件确实被占用了，但不知道是被哪个进程占用的，再使用lsof命令
$ ./lsof /opt/oracle/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1
执行完后没有结果，这就没办法查了。
于是请教了同事，他用使用root用户去执行lsof命令，而且只指定到最上一层的目录去查
# lsof /opt
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
RaterServ 29888 rate3 mem REG 64,0x6 10800648 85874 /opt/oracle/app/oracle/product/11.2.4/db_1/lib/libnnz11.so
RaterServ 29888 rate3 mem REG 64,0x6 121875400 324102 /opt/oracle/app/oracle/product/11.2.4/db_1/lib/libclntsh.so.11.1
从上面的输出结果可以看出来确实是有进程在占用，进程号是29888。
再查29888对就的进程名
# ps -ef |grep 29888
找到是应用的一个服务，与测试沟通后，这个进程可以直接kill，于是
kill -9 29888
再次执行opatch apply成功。