*Author's introduction*****
Original Creator: Lose Seriously
Contact number: 18081072613
WeChat, QQ:18081072613
Reprint to indicate origin
1. Failure Phenomena
Today, a customer's database started with an ORA-00600 [kkdlcob-objn-exists], [37] error, the database could not be opened, along with a good ORA-00704, requesting emergency rescue.
Error statement:
kkdlcob-objn-exists indicates that dataobj#already exists.
37 represents the value of dataobj#
If an error occurs during the database startup phase, accompanied by an ORA-00704 error, then the database fails during the bootstrap phase and is conscious of the error when creating system-owned objects.
The detailed code for the error is as follows:
2. Analysis process
2.1 View trace files
In the log file, you can see the following information:
*** 2020-05-06 15:47:40.272 dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0) ----- Current SQL Statement for this session (sql_id=8urmvm9u914ww) ----- CREATE UNIQUE INDEX I_OBJ2 ON OBJ$(OWNER#,NAME,NAMESPACE,REMOTEOWNER,LINKNAME,SUBNAME,TYPE#,SPARE3,OBJ#) PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 16K NEXT 104K MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 37 EXTENTS (FILE 1 BLOCK 344)) ----- Abridged Call Stack Trace ----- ksedsts()+585<-kjzdssdmp()+329<-kjzduptcctx()+288<-kjzdpcrshnfy()+77<-kstdmp()+364<-dbkedDefDump()+16515<-ksedmp()+43<-ksfdmp()+87<-dbgexPhaseII()+1819<-dbgexProcessError()+3008<-dbgeExecuteForError()+65<-dbgePostErrorKGE()+2269<-dbkePostKGE_kgsf()+77<-kgeadse()+349 <-kgerinv_internal()+76<-kgerinv()+49<-kgeasnmierr()+64<-kkdlcob()+2197<-kdicwpre()+2143<-kdicrws()+1062
This SQL statement is the statement that starts ORA-00600 [kkdlcob-objn-exists], [37] with an error. It can also be found by the keyword OBJNO 37, that is, when the system creates I_OBJ2 at startup, the object of OBJNO 37 is already stored in the system.
We know that OBJNO 37 is information that the system reads from the bootstrap $table during startup and that the contents of this table are fixed.
So we suspect that someone deliberately modified the bootstrap$content.
Continue looking down at the log contents:
Continuing with the trace analysis, you can see the block accessed before the error, 1/523 of which belongs to bootstrap$
BH (0x00000009CFD3DAB0) file#: 1 rdba: 0x0040020b (1/523) class: 1 ba: 0x00000009CB5EC000 set: 37 pool: 3 bsz: 8192 bsi: 0 sflg: 2 pwc: 0,0 dbwrid: 0 obj: 59 objn: 59 tsn: 0 afn: 1 hint: f hash: [0x0000000AD19BF560,0x0000000AD19BF560] lru: [0x0000000B02416BB8,0x0000000B02416BB8] ckptq: [NULL] fileq: [NULL] objq: [0x0000000ACECC2340,0x0000000ACECC2340] objaq: [0x0000000ACECC2330,0x0000000ACECC2330] st: XCURRENT md: NULL fpin: 'kdswh11: kdst_fetch' tch: 1 flags: only_sequential_access LRBA: [0x0.0.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [65535] buffer tsn: 0 rdba: 0x0040020b (1/523) scn: 0x0000.01565fa3 seq: 0x01 flg: 0x06 tail: 0x5fa30601 frmt: 0x02 chkval: 0x0ea2 type: 0x06=trans data Hex dump of block: st=0, typ_found=1 Dump of memory from 0x00000009CB5EC000 to 0x00000009CB5EE000 9CB5EC000 0000A206 0040020B 01565FA3 06010000 [......@.._V.....] 9CB5EC010 00000EA2 00000001 0000003B 01565FA2 [........;...._V.] 9CB5EC020 00000000 00030001 00000000 001D000A [................] 9CB5EC030 0001D51A 00C00241 0014616B 00002002 [....A...ka... ..] 9CB5EC040 01565FA3 000F0100 0030FFFF 0DC40AAD [._V.......0.....] 9CB5EC050 00000DC4 1D30000F 1B481C56 19AB1A82 [......0.V.H.....] 9CB5EC060 17C1189C 0AAD16F8 112012EA 0F891051 [.......... .Q...] 9CB5EC070 0DF50EC0 00000000 00000000 00000000 [................] 9CB5EC080 00000000 00000000 00000000 00000000 [................] Repeat 166 times 9CB5ECAF0 03012C00 0205C102 3CFE05C1 45524303 [.,.........<.CRE] 9CB5ECB00 20455441 4C424154 41542045 22282442 [ATE TABLE TAB$("] 9CB5ECB10 234A424F 554E2022 5245424D 544F4E20 [OBJ#" NUMBER NOT] 9CB5ECB20 4C554E20 44222C4C 4F415441 22234A42 [ NULL,"DATAOBJ#"] 9CB5ECB30 4D554E20 2C524542 23535422 554E2022 [ NUMBER,"TS#" NU] 9CB5ECB40 5245424D 544F4E20 4C554E20 46222C4C [MBER NOT NULL,"F] 9CB5ECB50 23454C49 554E2022 5245424D 544F4E20 [ILE#" NUMBER NOT] 9CB5ECB60 4C554E20 42222C4C 4B434F4C 4E202223 [ NULL,"BLOCK#" N] 9CB5ECB70 45424D55 4F4E2052 554E2054 222C4C4C [UMBER NOT NULL,"] 9CB5ECB80 4A424F42 4E202223 45424D55 54222C52 [BOBJ#" NUMBER,"T] 9CB5ECB90 22234241 4D554E20 2C524542 4C4F4322 [AB#" NUMBER,"COL] 9CB5ECBA0 4E202253 45424D55 4F4E2052 554E2054 [S" NUMBER NOT NU] 9CB5ECBB0 222C4C4C 43554C43 22534C4F 4D554E20 [LL,"CLUCOLS" NUM] 9CB5ECBC0 2C524542 54435022 45455246 4E202224 [BER,"PCTFREE$" N] 9CB5ECBD0 45424D55 4F4E2052 554E2054 222C4C4C [UMBER NOT NULL,"] 9CB5ECBE0 55544350 24444553 554E2022 5245424D [PCTUSED$" NUMBER] 9CB5ECBF0 544F4E20 4C554E20 49222C4C 5254494E [ NOT NULL,"INITR] 9CB5ECC00 22534E41 4D554E20 20524542 20544F4E [ANS" NUMBER NOT ] 9CB5ECC10 4C4C554E 414D222C 41525458 2022534E [NULL,"MAXTRANS" ] 9CB5ECC20 424D554E 4E205245 4E20544F 2C4C4C55 [NUMBER NOT NULL,] 9CB5ECC30 414C4622 20225347 424D554E 4E205245 ["FLAGS" NUMBER N] 9CB5ECC40 4E20544F 2C4C4C55 44554122 22245449 [OT NULL,"AUDIT$"] 9CB5ECC50 52415620 52414843 38332832 4F4E2029 [ VARCHAR2(38) NO] 9CB5ECC60 554E2054 222C4C4C 43574F52 2022544E [T NULL,"ROWCNT" ] 9CB5ECC70 424D554E 222C5245 434B4C42 2022544E [NUMBER,"BLKCNT" ] 9CB5ECC80 424D554E 222C5245 43504D45 2022544E [NUMBER,"EMPCNT" ] 9CB5ECC90 424D554E 222C5245 53475641 20224350 [NUMBER,"AVGSPC" ] 9CB5ECCA0 424D554E 222C5245 434E4843 2022544E [NUMBER,"CHNCNT" ] 9CB5ECCB0 424D554E 222C5245 52475641 20224E4C [NUMBER,"AVGRLN" ] 9CB5ECCC0 424D554E 222C5245 53475641 465F4350 [NUMBER,"AVGSPC_F] 9CB5ECCD0 2022424C 424D554E 222C5245 43424C46 [LB" NUMBER,"FLBC] 9CB5ECCE0 2022544E 424D554E 222C5245 4C414E41 [NT" NUMBER,"ANAL] 9CB5ECCF0 54455A59 22454D49 54414420 53222C45 [YZETIME" DATE,"S] 9CB5ECD00 4C504D41 5A495345 4E202245 45424D55 [AMPLESIZE" NUMBE] 9CB5ECD10 44222C52 45524745 4E202245 45424D55 [R,"DEGREE" NUMBE] 9CB5ECD20 49222C52 4154534E 5345434E 554E2022 [R,"INSTANCES" NU] 9CB5ECD30 5245424D 4E49222C 4C4F4354 4E202253 [MBER,"INTCOLS" N] 9CB5ECD40 45424D55 4F4E2052 554E2054 222C4C4C [UMBER NOT NULL,"] 9CB5ECD50 4E52454B 4F434C45 2022534C 424D554E [KERNELCOLS" NUMB] 9CB5ECD60 4E205245 4E20544F 2C4C4C55 4F525022 [ER NOT NULL,"PRO] 9CB5ECD70 54524550 4E202259 45424D55 4F4E2052 [PERTY" NUMBER NO] 9CB5ECD80 554E2054 222C4C4C 47495254 47414C46 [T NULL,"TRIGFLAG] 9CB5ECD90 554E2022 5245424D 5053222C 31455241 [" NUMBER,"SPARE1] 9CB5ECDA0 554E2022 5245424D 5053222C 32455241 [" NUMBER,"SPARE2] 9CB5ECDB0 554E2022 5245424D 5053222C 33455241 [" NUMBER,"SPARE3] 9CB5ECDC0 554E2022 5245424D 5053222C 34455241 [" NUMBER,"SPARE4] 9CB5ECDD0 41562022 41484352 31283252 29303030 [" VARCHAR2(1000)] 9CB5ECDE0 5053222C 35455241 41562022 41484352 [,"SPARE5" VARCHA] 9CB5ECDF0 31283252 29303030 5053222C 36455241 [R2(1000),"SPARE6] 9CB5ECE00 41442022 20294554 524F5453 20454741 [" DATE) STORAGE ] 9CB5ECE10 4F202028 4F4E4A42 20373320 4E424154 [( OBJNO 37 TABN] --Scarlet 9CB5ECE20 2931204F 554C4320 52455453 4F5F4320 [O 1) CLUSTER C_O] 9CB5ECE30 28234A42 234A424F 03002C29 0239C102 [BJ#(OBJ#),....9.] 9CB5ECE40 43C139C1 54414552 4E492045 20584544 [.9.CREATE INDEX ] 9CB5ECE50 44435F49 20344645 43204E4F 24464544 [I_CDEF4 ON CDEF$] 9CB5ECE60 414E4528 44454C42 43502029 45524654 [(ENABLED) PCTFRE] 9CB5ECE70 30312045 494E4920 4E415254 20322053 [E 10 INITRANS 2 ] 9CB5ECE80 5458414D 534E4152 35353220 4F545320 [MAXTRANS 255 STO] 9CB5ECE90 45474152 20202820 54494E49 204C4149 [RAGE ( INITIAL ] 9CB5ECEA0 204B3436 5458454E 32303120 4D204B34 [64K NEXT 1024K M] 9CB5EDB40 20312053 4558414D 4E455458 32205354 [S 1 MAXEXTENTS 2] 9CB5EDB50 34373431 34363338 43502035 434E4954 [147483645 PCTINC] 9CB5EDB60 53414552 20302045 4E4A424F 3833204F [REASE 0 OBJNO 38] 9CB5EDB70 54584520 53544E45 49462820 3120454C [ EXTENTS (FILE 1] 9CB5EDB80 4F4C4220 33204B43 29293235 0203002C [ BLOCK 352)),...] 9CB5EDB90 C10226C1 0102FE26 41455243 55204554 [.&..&...CREATE U] 9CB5EDBA0 5551494E 4E492045 20584544 424F5F49 [NIQUE INDEX I_OB] 9CB5EDBB0 4F20324A 424F204E 4F28244A 52454E57 [J2 ON OBJ$(OWNER] 9CB5EDBC0 414E2C23 4E2C454D 53454D41 45434150 [#,NAME,NAMESPACE] 9CB5EDBD0 4D45522C 4F45544F 52454E57 4E494C2C [,REMOTEOWNER,LIN] 9CB5EDBE0 4D414E4B 55532C45 4D414E42 59542C45 [KNAME,SUBNAME,TY] 9CB5EDBF0 2C234550 52415053 4F2C3345 29234A42 [PE#,SPARE3,OBJ#)] 9CB5EDC00 54435020 45455246 20303120 54494E49 [ PCTFREE 10 INIT] 9CB5EDC10 534E4152 4D203220 52545841 20534E41 [RANS 2 MAXTRANS ] 9CB5EDC20 20353532 524F5453 20454741 49202028 [255 STORAGE ( I] 9CB5EDC30 4954494E 31204C41 4E204B36 20545845 [NITIAL 16K NEXT ] 9CB5EDC40 4B343031 4E494D20 45545845 2053544E [104K MINEXTENTS ] 9CB5EDC50 414D2031 54584558 53544E45 34313220 [1 MAXEXTENTS 214] 9CB5EDC60 33383437 20353436 49544350 4552434E [7483645 PCTINCRE] 9CB5EDC70 20455341 424F2030 204F4E4A 45203733 [ASE 0 OBJNO 37 E] --Scarlet 9CB5EDC80 4E455458 28205354 454C4946 42203120 [XTENTS (FILE 1 B] 9CB5EDC90 4B434F4C 34343320 002C2929 25C10203 [LOCK 344)),....%] 9CB5EDCA0 D025C102 41455243 55204554 5551494E [..%.CREATE UNIQU] 9CB5EDCB0 4E492045 20584544 424F5F49 4F20314A [E INDEX I_OBJ1 O] 9CB5EDCC0 424F204E 4F28244A 2C234A42 454E574F [N OBJ$(OBJ#,OWNE] 9CB5EDCD0 542C2352 23455059 43502029 45524654 [R#,TYPE#) PCTFRE] 9CB5EDCE0 30312045 494E4920 4E415254 20322053 [E 10 INITRANS 2 ] 9CB5EDCF0 5458414D 534E4152 35353220 4F545320 [MAXTRANS 255 STO]
From the trace, you can see that before I_OBJ2 was created, tab$was created and OBJ#happened to be 37.Friends who are familiar with the bootstrap core object should know that TAB$obj#should be 4.
Therefore, it is estimated that the modification of bootstrap$will result.
2.2 View the contents of a block
Through the dump information of the block, confirm that the fast information has been modified.
Block header dump: 0x0040020b Object id on Block? Y seg/obj: 0x3b csc: 0x00.1565fa2 itc: 1 flg: O typ: 1 - DATA fsl: 0 fnx: 0x0 ver: 0x01 Itl Xid Uba Flag Lck Scn/Fsc 0x01 0x000a.01d.0001d51a 0x00c00241.616b.14 --U- 2 fsc 0x0000.01565fa3 --Lck 2, 2 lines of records have been modified bdba: 0x0040020b tab 0, row 8, @0xaad tl: 840 fb: --H-FL-- lb: 0x1 cc: 3 --lb 1, indicating that the line has been modified col 0: [ 2] c1 05 col 1: [ 2] c1 05 --OBJ#=4 col 2: [828] 43 52 45 41 54 45 20 54 41 42 4c 45 20 54 41 42 24 28 22 4f 42 4a 23 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 44 41 54 41 4f 42 4a 23 22 20 4e 55 4d 42 45 52 2c 22 54 53 23 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 46 49 4c 45 23 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 42 4c 4f 43 4b 23 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 42 4f 42 4a 23 22 20 4e 55 4d 42 45 52 2c 22 54 41 42 23 22 20 4e 55 4d 42 45 52 2c 22 43 4f 4c 53 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 43 4c 55 43 4f 4c 53 22 20 4e 55 4d 42 45 52 2c 22 50 43 54 46 52 45 45 24 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 50 43 54 55 53 45 44 24 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 49 4e 49 54 52 41 4e 53 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 4d 41 58 54 52 41 4e 53 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 46 4c 41 47 53 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 41 55 44 49 54 24 22 20 56 41 52 43 48 41 52 32 28 33 38 29 20 4e 4f 54 20 4e 55 4c 4c 2c 22 52 4f 57 43 4e 54 22 20 4e 55 4d 42 45 52 2c 22 42 4c 4b 43 4e 54 22 20 4e 55 4d 42 45 52 2c 22 45 4d 50 43 4e 54 22 20 4e 55 4d 42 45 52 2c 22 41 56 47 53 50 43 22 20 4e 55 4d 42 45 52 2c 22 43 48 4e 43 4e 54 22 20 4e 55 4d 42 45 52 2c 22 41 56 47 52 4c 4e 22 20 4e 55 4d 42 45 52 2c 22 41 56 47 53 50 43 5f 46 4c 42 22 20 4e 55 4d 42 45 52 2c 22 46 4c 42 43 4e 54 22 20 4e 55 4d 42 45 52 2c 22 41 4e 41 4c 59 5a 45 54 49 4d 45 22 20 44 41 54 45 2c 22 53 41 4d 50 4c 45 53 49 5a 45 22 20 4e 55 4d 42 45 52 2c 22 44 45 47 52 45 45 22 20 4e 55 4d 42 45 52 2c 22 49 4e 53 54 41 4e 43 45 53 22 20 4e 55 4d 42 45 52 2c 22 49 4e 54 43 4f 4c 53 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 4b 45 52 4e 45 4c 43 4f 4c 53 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 50 52 4f 50 45 52 54 59 22 20 4e 55 4d 42 45 52 20 4e 4f 54 20 4e 55 4c 4c 2c 22 54 52 49 47 46 4c 41 47 22 20 4e 55 4d 42 45 52 2c 22 53 50 41 52 45 31 22 20 4e 55 4d 42 45 52 2c 22 53 50 41 52 45 32 22 20 4e 55 4d 42 45 52 2c 22 53 50 41 52 45 33 22 20 4e 55 4d 42 45 52 2c 22 53 50 41 52 45 34 22 20 56 41 52 43 48 41 52 32 28 31 30 30 30 29 2c 22 53 50 41 52 45 35 22 20 56 41 52 43 48 41 52 32 28 31 30 30 30 29 2c 22 53 50 41 52 45 36 22 20 44 41 54 45 29 20 53 54 4f 52 41 47 45 20 28 20 20 4f 42 4a 4e 4f 20 33 37 20 54 41 42 4e 4f 20 31 29 20 43 4c 55 53 54 45 52 20 43 5f 4f 42 4a 23 28 4f 42 4a 23 29
3, Solution
Knowing that bootstrap$has been modified makes the solution very simple.
3.1 Modify block content directly through bbed.
The way bbed is modified is as follows: http://www.htz.pw/?p=846
3.2 Find a database of the same version and copy bootstrap$
Modified methods are as follows: http://www.htz.pw/?p=841