oracle 11.2 RAC 系列-安装新主机，识别老存储-1-识别ASM磁盘

在有些场景下，RAC环境中如果主机出现问题，比如硬件故障等，不能启动，我们需要尽快存储上的启动数据库，恢复业务，那么就需要迁移以前的RAC环境到新的主机环境下，我测试了11.2和12.1的RAC，恢复过程还是很快的，基本上就是安装软件的过程，如果真实场景恢复业务，有两种方法：
1，按照我这里的方法重新安装主机，恢复RAC和数据库
2，如果之前有可用的操作系统的备份（比如NBU备份了OS），那么直接使用NBU还原即可
.
我这里测试的是方法1，重新安装11204的GI（Grid Infrastructure）和ORACLE RDBMS软件，然后识别老存储。
测试环境：单节点RAC, 操作系统是OEL Linux 6.6, 数据库版本是11.2.0.4

首先，因为存储使用的是11204的ASM，测试过程只安装11204的GI（Grid Infrastructure）软件，不用OUI配置GI。
[root@lunar5 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
 
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@lunar5 ~]#
执行root.sh：
[root@lunar5 ~]# /u01/app/11.2.0.4/grid/root.sh
Performing root user operation for Oracle 11g
 
The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.4/grid
 
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
 
 
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
 
Using configuration parameter file: /u01/app/11.2.0.4/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'lunar5'
CRS-2676: Start of 'ora.mdnsd' on 'lunar5' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lunar5'
CRS-2676: Start of 'ora.gpnpd' on 'lunar5' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lunar5'
CRS-2672: Attempting to start 'ora.gipcd' on 'lunar5'
CRS-2676: Start of 'ora.cssdmonitor' on 'lunar5' succeeded
CRS-2676: Start of 'ora.gipcd' on 'lunar5' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lunar5'
CRS-2672: Attempting to start 'ora.diskmon' on 'lunar5'
CRS-2676: Start of 'ora.diskmon' on 'lunar5' succeeded
CRS-2676: Start of 'ora.cssd' on 'lunar5' succeeded
 
ASM created and started successfully.
 
Disk Group TEMPDG created successfully.
 
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 381226346c054f62bf5f219ee16d7e25.
Successfully replaced voting disk group with +TEMPDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   381226346c054f62bf5f219ee16d7e25 (/dev/asm-diski) [TEMPDG]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'lunar5'
CRS-2676: Start of 'ora.asm' on 'lunar5' succeeded
CRS-2672: Attempting to start 'ora.TEMPDG.dg' on 'lunar5'
CRS-2676: Start of 'ora.TEMPDG.dg' on 'lunar5' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@lunar5 ~]#
相应的checkpoint文件内容：
[root@lunar5 ~]# ll /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
-rw-r--r-- 1 grid oinstall 4176 Feb 15 19:31 /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
[root@lunar5 ~]#
[root@lunar5 ~]# tail  /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
  
  
  
  
  
  
  
  
  

[root@lunar5 ~]#
这里看到“DESC=”ROOTCRS_NODECONFIG” STATE=”SUCCESS””表示GI已经配置完成。
图形界面点击ok，继续执行其余配置,配置完成后，再次检查checkpoint文件：
[root@lunar5 ~]# ll  /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
-rw-r--r-- 1 grid oinstall 4176 Feb 15 19:31 /u01/app/grid/Clusterware/ckptGridHA_lunar5.xml
[root@lunar5 ~]# cd /u01/app/grid/Clusterware/
[root@lunar5 Clusterware]# ll
total 8
-rw-r--r-- 1 grid oinstall 4176 Feb 15 19:31 ckptGridHA_lunar5.xml
[root@lunar5 Clusterware]#
这里看到，checkpoint文件的日期没有变化，说明checkpoint文件是执行root.sh的时候才有用的，也就是这个过程是11.2中为了方便客户，增加了root.sh的失败后继续配置二设计的，非常体贴的功能。在12.2中，该功能更加方便，他将会只管的告诉你当前配置的检查点情况，如果有些步骤失败后，oracle会自动清除老的配置，以便可以失败安装后不用重装，而是纠正错误后继续配置，类似“断点续传”那种意思。
.
比如，下面是12.2beta上安装RAC时执行root.sh的过程：
[root@lunar1 ~]#  /stage/root.sh
Performing root user operation.
 
The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /stage
 
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
 
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
 
Using configuration parameter file: /stage/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/lunar1/crsconfig/rootcrs_lunar1_2016-01-24_12-44-34AM.log
2016/01/24 12:44:34 CLSRSC-46: Error: '/stage/crs/install/ParentDirPerm_lunar1.txt' does not exist
 
2016/01/24 12:44:34 CLSRSC-46: Error: '/stage/crs/install/ParentDirPerm_lunar1.txt' does not exist
 
2016/01/24 12:44:35 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
 
2016/01/24 12:44:35 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
 
2016/01/24 12:44:35 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
 
2016/01/24 12:44:35 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
 
2016/01/24 12:44:37 CLSRSC-363: User ignored prerequisites during installation
 
2016/01/24 12:44:37 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
 
2016/01/24 12:44:38 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
 
2016/01/24 12:44:39 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
 
2016/01/24 12:44:40 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
 
2016/01/24 12:44:41 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
 
2016/01/24 12:44:41 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
 
2016/01/24 12:44:42 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
 
2016/01/24 12:44:43 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
 
2016/01/24 12:45:30 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
 
2016/01/24 12:45:31 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
 
2016/01/24 12:45:46 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
 
2016/01/24 12:46:08 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
 
2016/01/24 12:46:11 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
 
2016/01/24 12:46:11 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
 
2016/01/24 12:46:14 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lunar1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'lunar1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'lunar1'
CRS-2677: Stop of 'ora.mdnsd' on 'lunar1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'lunar1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'lunar1'
CRS-2677: Stop of 'ora.evmd' on 'lunar1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'lunar1'
CRS-2677: Stop of 'ora.gipcd' on 'lunar1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lunar1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2016/01/24 12:46:23 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
 
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'lunar1'
CRS-2672: Attempting to start 'ora.evmd' on 'lunar1'
CRS-2676: Start of 'ora.mdnsd' on 'lunar1' succeeded
CRS-2676: Start of 'ora.evmd' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lunar1'
CRS-2676: Start of 'ora.gpnpd' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'lunar1'
CRS-2676: Start of 'ora.gipcd' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lunar1'
CRS-2676: Start of 'ora.cssdmonitor' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lunar1'
CRS-2672: Attempting to start 'ora.diskmon' on 'lunar1'
CRS-2676: Start of 'ora.diskmon' on 'lunar1' succeeded
CRS-2676: Start of 'ora.cssd' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lunar1'
CRS-2672: Attempting to start 'ora.ctssd' on 'lunar1'
CRS-2676: Start of 'ora.ctssd' on 'lunar1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lunar1'
CRS-2676: Start of 'ora.asm' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'lunar1'
CRS-2676: Start of 'ora.storage' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'lunar1'
CRS-2676: Start of 'ora.crf' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'lunar1'
CRS-2676: Start of 'ora.crsd' on 'lunar1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: lunar1
CRS-6016: Resource auto-start has completed for server lunar1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2016/01/24 12:47:34 CLSRSC-343: Successfully started Oracle Clusterware stack
 
2016/01/24 12:47:34 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
 
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'lunar1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lunar1'
CRS-2676: Start of 'ora.asm' on 'lunar1' succeeded
CRS-2672: Attempting to start 'ora.CRSDG.dg' on 'lunar1'
CRS-2676: Start of 'ora.CRSDG.dg' on 'lunar1' succeeded
2016/01/24 12:48:43 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
 
2016/01/24 12:49:04 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
 
[root@lunar1 ~]#
可以看到，这个过程比12.2以前的RAC执行root.sh的提示更加清晰了。
好吧，回到我们的环境中，继续检查老的asm磁盘组
[grid@lunar5 ~]$ kfod disks=all ds=true cluster=true status=true
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                     Disk Group   User     Group  
================================================================================
   1:        512 Mb MEMBER    /dev/asm-diskb                           DATADG1      grid     asmadmin
   2:        512 Mb MEMBER    /dev/asm-diskc                           DATADG1      grid     asmadmin
   3:        512 Mb MEMBER    /dev/asm-diskd                           DATADG1      grid     asmadmin
   4:       1024 Mb MEMBER    /dev/asm-diske                           CRSDG        grid     asmadmin
   5:       1024 Mb MEMBER    /dev/asm-diskf                           CRSDG        grid     asmadmin
   6:       1024 Mb MEMBER    /dev/asm-diskg                           CRSDG        grid     asmadmin
   7:       5120 Mb MEMBER    /dev/asm-diskh                           DATADG2      grid     asmadmin
   8:       3072 Mb MEMBER    /dev/asm-diski                           TEMPDG       grid     asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME                                      HOST_NAME          
================================================================================
     +ASM1 /u01/app/11.2.0.4/grid                           lunar5             
[grid@lunar5 ~]$
将上述磁盘组添加到ASM启动磁盘组的列表中：
[grid@lunar5 ~]$ ss
 
SQL*Plus: Release 11.2.0.4.0 Production on Mon Feb 15 21:31:29 2016
 
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
 
SQL> show parameter asm
 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string
asm_diskstring                       string      /dev/asm*
asm_power_limit                      integer     1
asm_preferred_read_failure_groups    string
SQL> alter system set asm_diskgroups='TEMPDG','CRSDG', 'DATADG1', 'DATADG2' scope=both;
 
System altered.
 
SQL>
对新添加的磁盘组执行mount和dismount后，这些磁盘组就会自动添加到ocr中：
[root@lunar5 ~]# crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
               ONLINE  ONLINE       lunar5                                      
ora.DATADG1.dg
               ONLINE  ONLINE       lunar5                                      
ora.DATADG2.dg
               ONLINE  ONLINE       lunar5                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       lunar5                                      
ora.TEMPDG.dg
               ONLINE  ONLINE       lunar5                                      
ora.asm
               ONLINE  ONLINE       lunar5                   Started            
ora.gsd
               OFFLINE OFFLINE      lunar5                                      
ora.net1.network
               ONLINE  ONLINE       lunar5                                      
ora.ons
               ONLINE  ONLINE       lunar5                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       lunar5                                      
ora.cvu
      1        ONLINE  ONLINE       lunar5                                      
ora.lunar5.vip
      1        ONLINE  ONLINE       lunar5                                      
ora.oc4j
      1        ONLINE  ONLINE       lunar5                                      
ora.scan1.vip
      1        ONLINE  ONLINE       lunar5                                      
[root@lunar5 ~]#
检查ASM磁盘组中的内容（比如包含哪些数据库），因为ASM是采用OMF的，因此大部分情况下，通过磁盘组的目录结构就可以看出来这个存储中以前包含哪些数据库和其他相关信息：
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576      3072     2769             1024             872              0             N  CRSDG/
MOUNTED  NORMAL  N         512   4096  1048576      1536     1251              512             369              0             N  DATADG1/
MOUNTED  EXTERN  N         512   4096  1048576      5120     3386                0            3386              0             N  DATADG2/
MOUNTED  EXTERN  N         512   4096  1048576      3072     2719                0            2719              0             Y  TEMPDG/
ASMCMD> cd +crsdg
ASMCMD> ls
ASMCMD> cd +datadg1
ASMCMD> ls
ASMCMD> cd +datadg2
ASMCMD> ls
LUNAR/
ASMCMD>
这里看到，该存储中，包含了一个数据库，crsdg中的信息已经被我删除了，因此，没有老的ocr和voting disk文件了
现在，已经确定只有datadg2上面有数据，因此可以重组磁盘组，以便合理利用空间（后续我要用这个vm做很多其他测试）：
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  4194304      1536     1296              512             392              0             N  DATADG1/
MOUNTED  EXTERN  N         512   4096  1048576      5120     3386                0            3386              0             N  DATADG2/
MOUNTED  EXTERN  N         512   4096  1048576      3072     3016                0            3016              0             N  DATADG3/
MOUNTED  EXTERN  N         512   4096  1048576      3072     2719                0            2719              0             Y  TEMPDG/
ASMCMD>
使用kfod查看：
[grid@lunar5 ~]$ kfod disks=all ds=true cluster=true status=true
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                     Disk Group   User     Group  
================================================================================
   1:        512 Mb MEMBER    /dev/asm-diskb                           DATADG1      grid     asmadmin
   2:        512 Mb MEMBER    /dev/asm-diskc                           DATADG1      grid     asmadmin
   3:        512 Mb MEMBER    /dev/asm-diskd                           DATADG1      grid     asmadmin
   4:       1024 Mb MEMBER    /dev/asm-diske                           DATADG3      grid     asmadmin
   5:       1024 Mb MEMBER    /dev/asm-diskf                           DATADG3      grid     asmadmin
   6:       1024 Mb MEMBER    /dev/asm-diskg                           DATADG3      grid     asmadmin
   7:       5120 Mb MEMBER    /dev/asm-diskh                           DATADG2      grid     asmadmin
   8:       3072 Mb MEMBER    /dev/asm-diski                           TEMPDG       grid     asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME                                      HOST_NAME          
================================================================================
     +ASM1 /u01/app/11.2.0.4/grid                           lunar5             
[grid@lunar5 ~]$
确认当前的crs状态都是正常的：
[root@lunar5 ~]# crsctl status res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATADG1.dg
ONLINE ONLINE lunar5
ora.DATADG2.dg
ONLINE ONLINE lunar5
ora.DATADG3.dg
ONLINE ONLINE lunar5
ora.LISTENER.lsnr
ONLINE ONLINE lunar5
ora.TEMPDG.dg
ONLINE ONLINE lunar5
ora.asm
ONLINE ONLINE lunar5 Started
ora.gsd
OFFLINE OFFLINE lunar5
ora.net1.network
ONLINE ONLINE lunar5
ora.ons
ONLINE ONLINE lunar5
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE lunar5
ora.cvu
1 ONLINE ONLINE lunar5
ora.lunar5.vip
1 ONLINE ONLINE lunar5
ora.oc4j
1 ONLINE ONLINE lunar5
ora.scan1.vip
1 ONLINE ONLINE lunar5
[root@lunar5 ~]#
至此老存储的ASM磁盘组已经在新的主机上可以识别了。