oracle中Frequent generate a lot of cdmp* directories contain *bucket trace in bdump问题分析

最近有套库突然目录使用告警，发现TRACE 目录使用较高，发现在bdump 目录中每分钟都生成一个cdmp*目录，且每个目录中含大量的文件，这是一套oracle 11.2.0.3 rac on hpux 11.31的数据库， 如果没有在系统中启用dump event ，那就可能是bug ，这里简单的记录。

oracle@anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace> ls
alert_weejar2.log       cdmp_20160408221002    cdmp_20160409050508    cdmp_20160427102501    cdmp_20160503133006    cdmp_20160506124003 
cdmp_20160407113747    cdmp_20160408221101    cdmp_20160409050609    cdmp_20160427102602    cdmp_20160503133105    cdmp_20160506124103  
cdmp_20160407113801    cdmp_20160408221202    cdmp_20160409050708    cdmp_20160427103304    cdmp_20160503133205    cdmp_20160506124202
...

oracle@anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502> ls
weejar2_acms_15550_bucket.trc  weejar2_ora_16823_bucket.trc   weejar2_ora_18001_bucket.trc  
weejar2_ora_19458_bucket.trc   weejar2_ora_26892_bucket.trc   weejar2_ora_29042_bucket.trc
weejar2_acms_15550_bucket.trm  weejar2_ora_16823_bucket.trm   weejar2_ora_18001_bucket.trm  
weejar2_ora_19458_bucket.trm   weejar2_ora_26892_bucket.trm   weejar2_ora_29042_bucket.trm
...

oracle@anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502> cat weejar2_ora_8317_bucket.trc
Trace file /oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502/weejar2_ora_8317_bucket.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /oracle/app/oracle/product/11.2.0.3/dbhome_1
System name:    HP-UX
Node name:      anbob2
Release:        B.11.31
Version:        U
Machine:        ia64
Instance name: weejar2
Redo thread mounted by this instance: 2
Oracle process number: 531
Unix process pid: 8317, image: oracle@anbob2

*** 2016-05-09 17:35:07.273
*** SESSION ID:(832.4201) 2016-05-09 17:35:07.273
 

*** 2016-05-09 17:35:07.273
Process diagnostic dump for oracle@anbob2, OS id=8317,
pid: 531, proc_ser: 66, sid: 832, sess_ser: 4201
-------------------------------------------------------------------------------
current sql:
client details:
  O/S info: user: weblogic, term: unknown, ospid: 1234
  machine: qmweb86 program: JDBC Thin Client
  application name: JDBC Thin Client, hash value=2546894660
Current Wait Stack:
 0: waiting for 'SQL*Net message from client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=55 seq_num=56 snap_id=1
    wait times: snap=4.499210 sec, exc=4.499210 sec, total=4.499210 sec
    wait times: max=infinite, heur=4.499210 sec
    wait counts: calls=0 os=0
    in_wait=1 iflags=0x1a0
Wait State:
  fixed_waits=0 flags=0x22 boundary=0x0000000000000000/-1
Session Wait History:
    elapsed time of 0.000031 sec since current wait
 0: waited for 'SQL*Net message to client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=54 seq_num=55 snap_id=1
    wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000012 sec of elapsed time
 1: waited for 'SQL*Net message from client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=53 seq_num=54 snap_id=1
    wait times: snap=0.001062 sec, exc=0.001062 sec, total=0.001062 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000027 sec of elapsed time
 2: waited for 'SQL*Net message to client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=52 seq_num=53 snap_id=1
    wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000016 sec of elapsed time
 3: waited for 'SQL*Net message from client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=51 seq_num=52 snap_id=1
    wait times: snap=0.001342 sec, exc=0.001342 sec, total=0.001342 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000526 sec of elapsed time
 4: waited for 'SQL*Net message to client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=50 seq_num=51 snap_id=1
    wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000173 sec of elapsed time
 5: waited for 'SQL*Net message from client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=49 seq_num=50 snap_id=1
    wait times: snap=0.000847 sec, exc=0.000847 sec, total=0.000847 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000003 sec of elapsed time
 6: waited for 'SQL*Net message to client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=48 seq_num=49 snap_id=1
    wait times: snap=0.000002 sec, exc=0.000002 sec, total=0.000002 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000041 sec of elapsed time
 7: waited for 'SQL*Net message from client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=47 seq_num=48 snap_id=1
    wait times: snap=0.029814 sec, exc=0.029814 sec, total=0.029814 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000076 sec of elapsed time
 8: waited for 'SQL*Net message to client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=46 seq_num=47 snap_id=1
    wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000153 sec of elapsed time
 9: waited for 'SQL*Net message from client'
    driver id=0x74637000, #bytes=0x1, =0x0
    wait_id=45 seq_num=46 snap_id=1
    wait times: snap=0.002608 sec, exc=0.002608 sec, total=0.002608 sec
    wait times: max=infinite
    wait counts: calls=0 os=0
    occurred after 0.000002 sec of elapsed time
Sampled Session History of session 832 serial 4201
---------------------------------------------------
The sampled session history is constructed by sampling
the target session every 1 second. The sampling process
captures at each sample if the session is in a non-idle wait,
an idle wait, or not in a wait. If the session is in a
non-idle wait then one interval is shown for all the samples
the session was in the same non-idle wait. If the
session is in an idle wait or not in a wait for
consecutive samples then one interval is shown for all
the consecutive samples. Though we display these consecutive
samples  in a single interval the session may NOT be continuously
idle or not in a wait (the sampling process does not know).
 
The history is displayed in reverse chronological order.
 
sample interval: 1 sec, max history 120 sec
---------------------------------------------------
  [121 samples,                                            17:33:07 - 17:35:07]
    idle wait at each sample
-------------------------------------------------------------------------------
Process diagnostic dump actual duration=0.005000 sec
  (max dump time=30.000000 sec)

*** 2016-05-09 17:35:07.278
 
-------------------------------------------------------------------------------
Trace Bucket Dump Begin: default bucket for process 531 (osid: 8317)
TIME(*=approx):SEQ:COMPONENT:FILE@LINE:FUNCTION:SECT/DUMP: [EVENT#:PID:SID] DATA
-------------------------------------------------------------------------------
2016-05-09 17:32:17.785270 :GSIPC:kjctc.c@321:kjctccr(): GSIPC:CONN: Disconnect from inst 1, receiver 2
2016-05-09 17:32:17.785271 :89F5D5D1:db_trace:ksxp.c@4368:ksxpmsgcncl(): [10401:531:0] KSXPMSGCNCL: client 2 tid inst 1 ptid 257 flags 0x100
2016-05-09 17:32:17.785271 :GSIPC:kjctc.c@321:kjctccr(): GSIPC:CONN: Disconnect from inst 1, receiver 3
2016-05-09 17:32:17.785272 :89F5D5D2:db_trace:ksxp.c@4368:ksxpmsgcncl(): [10401:531:0] KSXPMSGCNCL: client 2 tid inst 1 ptid 257 flags 0x100
2016-05-09 17:32:17.785569 :GSIPC:kjcc.c@626:kjccdel(): GSIPC:CLNT: deleted client comm layer structures 2
2016-05-09 17:32:17.785576 :89F5D5D4:db_trace:kss.c@1414:kssdch(): [10809:531:0] kssdch(0xc0000010e1d0bdc8 = process, 2) 2 0 exit

查看系统event dump

SQL> oradebug setmypid
Statement processed.
SQL> oradebug eventdump system
10949 trace name context forever,level 1
28401 trace name context forever,level 1
SQL> oradebug eventdump process
28401 trace name context forever,level 1
10949 trace name context forever,level 1
SQL> oradebug eventdump session
28401 trace name context forever,level 1
10949 trace name context forever,level 1
Mos中查了下定位了一个bug,Bug 17062524 Diagnostic enhancement to write default trace bucket to trace file for database

The default RDBMS tracing bucket is not dumped to the trace file on a
failure – instead it is dumped to a cdmp* file.

Workaround
Find the cdmp* file containing the default tracing bucket’s contents. (This
file is anot included by default when packaging an incident so has to be
collected manually)

fixed:
12.2 (Future Release)
12.1.0.2 (Server Patch Set)