由于维护人员修改Oracle Linux 7中的/dev/shm大小造成其大小小于Oracle实例的MEMORY_TARGET或者SGA_TARGET而导致集群不能启动(CRS-4535,CRS-4000)
[grid@jtp1 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
检查asm磁盘的权限是否问题，发现磁盘权限正常
[root@jtp3 ~]# ls -lrt /dev/asm*
brw-rw----. 1 grid oinstall 8, 128 Apr 3 2018 /dev/asmdisk07
brw-rw----. 1 grid oinstall 8, 48 Apr 3 2018 /dev/asmdisk02
brw-rw----. 1 grid oinstall 8, 96 Apr 3 2018 /dev/asmdisk05
brw-rw----. 1 grid oinstall 8, 112 Apr 3 2018 /dev/asmdisk06
brw-rw----. 1 grid oinstall 8, 64 Apr 3 2018 /dev/asmdisk03
brw-rw----. 1 grid oinstall 8, 80 Apr 3 2018 /dev/asmdisk04
brw-rw----. 1 grid oinstall 8, 32 Apr 3 2018 /dev/asmdisk01
重启crs
[root@jtp1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'jtp1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'jtp1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'jtp1'
CRS-2677: Stop of 'ora.mdnsd' on 'jtp1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'jtp1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'jtp1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'jtp1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'jtp1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'jtp1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'jtp1'
CRS-2673: Attempting to stop 'ora.evmd' on 'jtp1'
CRS-2677: Stop of 'ora.ctssd' on 'jtp1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'jtp1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'jtp1'
CRS-2677: Stop of 'ora.cssd' on 'jtp1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'jtp1'
CRS-2673: Attempting to stop 'ora.driver.afd' on 'jtp1'
CRS-2677: Stop of 'ora.driver.afd' on 'jtp1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'jtp1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'jtp1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@jtp1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
查看crs的alert.log发现磁盘组不能加载
[root@jtp1 ~]# tail -f /u01/app/grid/diag/crs/jtp1/crs/trace/alert.log
2018-04-02 18:30:21.227 [OHASD(8143)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 8143
2018-04-02 18:30:21.230 [OHASD(8143)]CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
2018-04-02 18:30:21.245 [OHASD(8143)]CRS-2112: The OLR service started on node jtp1.
2018-04-02 18:30:21.262 [OHASD(8143)]CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2018-04-02 18:30:21.262 [OHASD(8143)]CRS-1301: Oracle High Availability Service started on node jtp1.
2018-04-02 18:30:21.567 [ORAROOTAGENT(8214)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 8214
2018-04-02 18:30:21.600 [CSSDAGENT(8231)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 8231
2018-04-02 18:30:21.607 [CSSDMONITOR(8241)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 8241
2018-04-02 18:30:21.620 [ORAAGENT(8225)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 8225
2018-04-02 18:30:22.146 [ORAAGENT(8316)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 8316
2018-04-02 18:30:22.211 [MDNSD(8335)]CRS-8500: Oracle Clusterware MDNSD process is starting with operating system process ID 8335
2018-04-02 18:30:22.215 [EVMD(8337)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 8337
2018-04-02 18:30:23.259 [GPNPD(8369)]CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 8369
2018-04-02 18:30:24.275 [GPNPD(8369)]CRS-2328: GPNPD started on node jtp1.
2018-04-02 18:30:24.283 [GIPCD(8433)]CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 8433
2018-04-02 18:30:26.296 [CSSDMONITOR(8464)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 8464
2018-04-02 18:30:28.299 [CSSDAGENT(8482)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 8482
2018-04-02 18:30:28.496 [OCSSD(8497)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 8497
2018-04-02 18:30:29.538 [OCSSD(8497)]CRS-1713: CSSD daemon is started in hub mode
2018-04-02 18:30:36.015 [OCSSD(8497)]CRS-1707: Lease acquisition for node jtp1 number 1 completed
2018-04-02 18:30:37.087 [OCSSD(8497)]CRS-1605: CSSD voting file is online: AFD:CRS1; details in /u01/app/grid/diag/crs/jtp1/crs/trace/ocssd.trc.
2018-04-02 18:30:37.103 [OCSSD(8497)]CRS-1672: The number of voting files currently available 1 has fallen to the minimum number of voting files required 1.
2018-04-02 18:30:46.237 [OCSSD(8497)]CRS-1601: CSSD Reconfiguration complete. Active nodes are jtp1 .
2018-04-02 18:30:48.514 [OCTSSD(9302)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 9302
2018-04-02 18:30:48.535 [OCSSD(8497)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
2018-04-02 18:30:50.626 [OCTSSD(9302)]CRS-2407: The new Cluster Time Synchronization Service reference node is host jtp1.
2018-04-02 18:30:50.627 [OCTSSD(9302)]CRS-2401: The Cluster Time Synchronization Service started on host jtp1.
2018-04-02 18:31:04.202 [ORAROOTAGENT(8214)]CRS-5019: All OCR locations are on ASM disk groups [CRS], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc".
2018-04-02 18:41:00.225 [ORAROOTAGENT(8214)]CRS-5818: Aborted command 'start' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:9:3} in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc.
2018-04-02 18:41:03.757 [ORAROOTAGENT(8214)]CRS-5017: The resource action "ora.storage start" encountered the following error:
2018-04-02 18:41:03.757+Storage agent start action aborted. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc".
2018-04-02 18:41:03.760 [OHASD(8143)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.storage'. Details at (:CRSPE00221:) {0:9:3} in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd.trc.
2018-04-02 18:42:09.921 [ORAROOTAGENT(8214)]CRS-5019: All OCR locations are on ASM disk groups [CRS], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc".