记一次TB级别的Oracle Exadata数据库恢复

现在它还提供了新的低成本1/8机架配置。 借助Oracle Exadata，客户可通过整合降低IT成本，使存储能力增加10倍，提升所有应用的性能，因消除系统集成的试错过程而加快上市速度，以及实时做出更好的业务决策

某客户的exadata环境，由于大量硬件故障，导致数据库宕机。经过分析确认是由于某个cell节点的某个disk出现异常，导致部分

cell节点的disk 离线后无法进行asm reblance；最终导致diskgroup 被强制dismount了。如下是alert log

Fri Oct 09 15:44:26 2015
NOTE: process _user4655_+asm1 (4655) initiating offline of disk 46.3915916191 (DATA_KLYX_CD_10_KLYXCEL02) with mask 0x7e in group 1
NOTE: checking PST: grp = 1
ERROR: Disk 18 cannot be offlined, since all the disks [18, 49] with mirrored data would be offline.
ERROR: too many offline disks in PST (grp 2)
GMON checking disk modes for group 1 at 55 for pid 41, osid 4655
Fri Oct 09 15:44:26 2015
NOTE: cache dismounting (not clean) group 2/0x4AD8D253 (DBFS_DG)
ERROR: too many offline disks in PST (grp 1)
NOTE: checking PST for grp 1 done.
WARNING: Disk 46 (DATA_KLYX_CD_10_KLYXCEL02) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 46/0xe968239f, mode = 0x6a, op = 4
GMON updating disk modes for group 1 at 56 for pid 41, osid 4655
ERROR: Disk 46 cannot be offlined, since all the disks [46, 72] with mirrored data would be offline.
ERROR: too many offline disks in PST (grp 1)
WARNING: Offline of disk 46 (DATA_KLYX_CD_10_KLYXCEL02) in group 1 and mode 0x7f failed on ASM inst 1
WARNING: Disk 18 (DBFS_DG_CD_10_KLYXCEL02) in group 2 mode 0x7f offline is being aborted
Fri Oct 09 15:44:26 2015
NOTE: process _user4663_+asm1 (4663) initiating offline of disk 46.3915916191 (DATA_KLYX_CD_10_KLYXCEL02) with mask 0x7e in group 1
NOTE: checking PST: grp = 1
WARNING: Offline of disk 18 (DBFS_DG_CD_10_KLYXCEL02) in group 2 and mode 0x7f failed on ASM inst 1
Fri Oct 09 15:44:26 2015
NOTE: halting all I/Os to diskgroup 2 (DBFS_DG)
NOTE: unable to offline disks after getting write error for diskgroup DBFS_DG
NOTE: write to disk 32 succeeded
NOTE: disk 18 had IO error
NOTE: write to disk 6 succeeded
NOTE: halting all I/Os to diskgroup 1 (DATA_KLYX)
NOTE: LGWR doing non-clean dismount of group 1 (DATA_KLYX)
NOTE: LGWR sync ABA=21.5465 last written ABA 21.5465
GMON checking disk modes for group 1 at 57 for pid 44, osid 4663
ERROR: too many offline disks in PST (grp 1)
NOTE: checking PST for grp 1 done.

我们可以看到，cell 2节点的第10号disk存在异常，通过cellcli命令行也可以进一步确认：

CellCLI> list physicaldisk;
         20:0            E07Q5N          normal
         20:1            E05SMD          normal
         20:2            E07GV6          normal
         20:3            E060YD          normal
         20:4            E05ZQE          normal
         20:5            E07L9M          normal
         20:6            E07FED          normal
         20:7            E05N88          normal
         20:8            E08DY5          normal
         20:9            E06ADQ          normal
         20:10           E08E5C          critical
         20:11           E08DYF          normal

通过对该磁盘进行修复检查，确实也发现了坏道，如下所示：

SEAGATE ST360057SSUN600G --> SEAGATE ST360057SSUN600G
Sector 0 --> Sector 0
1,172,123,568 Sectors
Sectors that could not be read:
102,578,137
102,578,139
102,578,140
102,578,141
102,578,142
102,578,149
102,578,152
......
1,172,123,561 sector(s) successfully copied.
7 bad source sectors encountered.
Corresponding destination sectors filled with: UNREADABLESECTOR

修复完毕后，成功mount diskgroup。但是最后检查数据库存在部分数据坏块，如下所示：

SQL>  select * from  V$DATABASE_BLOCK_CORRUPTION;
 
     FILE#     BLOCK#     BLOCKS CORRUPTION_CHANGE# CORRUPTION_TYPE
---------- ---------- ---------- ------------------ -----------------
         1    5186816          3                  0 CORRUPT
         1    5186813          1                  0 CORRUPT
         1    3343734          2                  0 CORRUPT
        25    2391372          1         1.4561E+13 CORRUPT
        35     538916          1                  0 CORRUPT
        36     543804          3                  0 CORRUPT

对于数据坏块的修复相对比较简单，只是由于是asm环境处理起来相对麻烦一些，加上客户没有备份，所以无法进行blockcover。对于system的几个数据坏块，经过检查发现为审计对象，因此通过truncate然后insert数据即可格式化掉；
另外第25号文件的数据坏块为逻辑坏块，通过查询确认为index，直接rebuild index即可。
至于说最后第第35，36号文件，由于均为数据文件，涉及的对象均为分区表的partiiton，因此处理相对麻烦一些。

这里我通过创建测试表，然后清理掉数据，构造一个数据空块，然后将该数据块复制到asm diskgroup中替换到这几个坏块即可，注意替换之前，需要修改数据坏块的rdba地址和obj id信息，如下是数据块的复制替换过程：

SQL> @copy_block_to_asm.sql
Enter value for file_with_patched_block: /u01/.../blk.dbf
old  15: v_FsFileName := '&file_with_patched_block';
new  15: v_FsFileName := '/u01/.../blk.dbf';
Enter value for file_to_patch_in_asm: +DATA_KLYX/xxxx1/datafile/epmrange107.dbf
old  16: v_AsmFileName := '&file_to_patch_in_ASM';
new  16: v_AsmFileName := '+DATA_KLYX/xxxx1/datafile/epmrange107.dbf';
Enter value for block_to_patch: 543804
old  17: v_offstart := '&block_to_patch';
new  17: v_offstart := '543804';
PL/SQL procedure successfully completed.

最后通过rman检查，确认一切正常。如下是rman的检测结果：

.......
RMAN> backup validate datafile 36;
 
Starting backup at 16-OCT-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1337 instance=xxx1 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00036 name=+DATA_KLYX/xxx1/datafile/epmrange107.dbf
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
List of Datafiles
=================
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
36   OK     0              315258       3276800         14561048845049
  File Name: +DATA_KLYX/xxx1/datafile/epmrange107.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              2926354
  Index      0              20
  Other      0              35168
......