一个存储档案的rac数据库起不来了，生产环境是linux rac 11.2.0.4,原因是因为用工具测试磁盘IO时损坏了ocr所在磁盘组与存储数据ASM磁盘的磁盘头。
下面是恢复过程:
1.检查crs的状态:
[grid@darac1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[root@darac1 crsd]# ps -ef|grep crs
root 3126 1 1 10:34 ? 00:00:31 /u01/app/product/11.2.0/crs/bin/ohasd.bin reboot
grid 3514 1 0 10:34 ? 00:00:07 /u01/app/product/11.2.0/crs/bin/oraagent.bin
grid 3525 1 0 10:34 ? 00:00:00 /u01/app/product/11.2.0/crs/bin/mdnsd.bin
grid 3537 1 0 10:34 ? 00:00:16 /u01/app/product/11.2.0/crs/bin/gpnpd.bin
grid 3549 1 1 10:34 ? 00:00:33 /u01/app/product/11.2.0/crs/bin/gipcd.bin
root 4128 1 0 10:54 ? 00:00:02 /u01/app/product/11.2.0/crs/bin/cssdmonitor
root 4144 1 0 10:54 ? 00:00:01 /u01/app/product/11.2.0/crs/bin/cssdagent
grid 4167 1 2 10:55 ? 00:00:14 /u01/app/product/11.2.0/crs/bin/ocssd.bin
root 4354 3680 0 11:04 pts/1 00:00:00 grep crs
2.强制关闭crs
[root@darac1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'darac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'darac1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'darac1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'darac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'darac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'darac1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'darac1'
CRS-2677: Stop of 'ora.gpnpd' on 'darac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'darac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
3.以exclusive模式启动crs
[root@darac1 bin]# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'darac1'
CRS-2676: Start of 'ora.mdnsd' on 'darac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'darac1'
CRS-2676: Start of 'ora.gpnpd' on 'darac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'darac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'darac1'
CRS-2676: Start of 'ora.gipcd' on 'darac1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'darac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'darac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'darac1'
CRS-2676: Start of 'ora.diskmon' on 'darac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'darac1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'darac1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'darac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'darac1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'darac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'darac1'
CRS-2676: Start of 'ora.ctssd' on 'darac1' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'darac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'darac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'darac1'
CRS-2676: Start of 'ora.asm' on 'darac1' succeeded
4.查看GI相关的alert.log日志文件如何
[ohasd(5040)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2016-10-13 11:20:47.302:
[gpnpd(5215)]CRS-2328:GPNPD started on node darac1.
2016-10-13 11:20:58.388:
[ohasd(5040)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2016-10-13 11:21:00.608:
[cssd(5318)]CRS-1713:CSSD daemon is started in clustered mode
2016-10-13 11:21:01.521:
[/u01/app/product/11.2.0/crs/bin/orarootagent.bin(5304)]CRS-5013:Agent "/u01/app/product/11.2.0/crs/bin/orarootagent.bin" failed to start process
"/u01/app/product/11.2.0/crs/bin/osysmond" for action "start": details at "(:CLSN00008:)" in
"/u01/app/product/11.2.0/crs/log/darac1/agent/ohasd/orarootagent_root//orarootagent_root.log"
2016-10-13 11:21:03.585:
[ohasd(5040)]CRS-2878:Failed to restart resource 'ora.crf'
2016-10-13 11:21:05.399:
[/u01/app/product/11.2.0/crs/bin/orarootagent.bin(5340)]CRS-5013:Agent "/u01/app/product/11.2.0/crs/bin/orarootagent.bin" failed to start process
"/u01/app/product/11.2.0/crs/bin/osysmond" for action "start": details at "(:CLSN00008:)" in