IBM P55A主机宕机dump分析(2)

(2)> f 2
pvthread+000200 STACK:
Use current context [F00000002FF47600] of cpu 0
[002934FC].h_cede+000014 ()
[00049BE0]waitproc+00047C ()
[0013B0F4]procentry+000010 (??, ??, ??, ??)
(2)> f 14
pvthread+000E00 STACK:
Use current context [01941E00] of cpu 6
[002B0528]slock+0001D8 (0000000003B90010, 0000000001941B70 [??])
[00009558].simple_lock+000058 ()
[0020D6EC]v_prelru_remlist+000060 (??, ??, ??, ??)
[002A25FC]begfst+0000A0 ()
____ Exception (F000000030017780) ____
iar   : 0000000000065090  msr   : 8000000000009032  cr    : 22022024
lr    : 00000000000BBBD0  ctr   : 00000000000D0508  xer   : 20000000
mq    : 00000000  asr   : 00000001EF72A001
r0  : 0000000022022024  r1  : 0FFFFFFFF4017BE0  r2  : 0000000001491C28
r3  : 0000000000065090  r4  : 8000000000001032  r5  : 0000000022022024
r6  : 0FFFFFFFF4017C70  r7  : 0000000000000000  r8  : 00000000010209C0
r9  : 0000000000000001  r10 : 0000000000000000  r11 : 0000000000000000
r12 : 0000000000297EC8  r13 : F10001002CB82C00  r14 : 0000000001000085
r15 : F1000100100B0600  r16 : 00000000510000F0  r17 : 0000000000000003
r18 : 0000000000000001  r19 : 0000000000000000  r20 : F10001002CB82D78
r21 : 00000000FFFEFBFF  r22 : 0000000000000000  r23 : F00000002FF47600
r24 : 0000000000000000  r25 : 0000000000000000  r26 : F100070F00001800
r27 : F100070F10000E00  r28 : F100010010088000  r29 : F10001002CB82C00
r30 : 0000000000000001  r31 : 0000000000000001
prev      0000000000000000 stackfix  0000000000000000 int_ticks 00
kjmpbuf   0000000000000000 excbranch 0000000000000000 no_pfault 00
intpri    0B backt     00 flags     00
fpscr     0000000000000000 fpscrx    00000000 fpowner   00
fpeu      00 fpinfo    00 alloc     F000
o_iar     0000000000000000 o_toc     0000000000000000
o_arg1    0000000000000000 o_vaddr   0000000000000000
krlockp   0000000000000000   
Except :                     
 csr   0000000000000000 dsisr 0000000040000000  bit set: DSISR_PFT
 esid  0000000009101400 dar   F1000110151EB000 dsirr 0000000000000106
[00065090]et_wait+000344 (0000000000065090, 8000000000001032,
   0000000022022024 [??])    
[000CF134]vm_psmd_flush_pending+0000D8 (??, ??, ??)
[000CE78C]vm_psmd_demote+0002BC (??, ??, ??, ??)
[000CFEA8]psmd_kthread+000098 (??)
[0013DAC0]threadentry+000014 (??, ??, ??, ??)
(2)> symptom
[kdb_get_memory] no real storage @ FFFFFFFF4017EA0
PIDS/5765G0300 LVLS/530 PCSS/SPI1 MS/300 FLDS/v_delpft VALU/7c20f800 FLDS/v_relfram VALU/464
p550a:/tmp/ibmsupt#kdb vmcore.1 /unix
The specified kernel file is a 64-bit kernel
vmcore.1 mapped from @ 700000000000000 to @ 700000031cd9029
Preserving 1317350 bytes of symbol table
First symbol __mulh
Component Names:
 1)  minidump [2 entries]
 2)  dmp_minimal [9 entries]
 3)  proc [315 entries]
 4)  thrd [2310 entries]
 5)  rasct [1 entries]
 6)  ldr [2 entries]
 7)  errlg [3 entries]
 8)  mtrc [26 entries]
 9)  lfs [2 entries]
10)  bos [3 entries]
11)  ipc [7 entries]
12)  vmm [13 entries]
13)  alloc_kheap [512 entries]
14)  alloc_other [21 entries]
15)  rtastrc [8 entries]
16)  eidedd [1 entries]
17)  sisraid [4 entries]
18)  aixpcm [9 entries]
19)  scdisk [19 entries]
20)  lvm [2 entries]
21)  jfs2 [1 entries]
22)  tty [4 entries]
23)  netstat [10 entries]
24)  goent_dd [7 entries]
25)  dump_statistics [1 entries]
Component Dump Table has 3292 entries
           START              END
0000000000001000 0000000003BBA050 start+000FD8
F00000002FF47600 F00000002FFDC920 __ublock+000000
000000002FF22FF4 000000002FF22FF8 environ+000000
000000002FF22FF8 000000002FF22FFC errno+000000
F100070F00000000 F100070F10000000 pvproc+000000
F100070F10000000 F100070F18000000 pvthread+000000
PFT:
PVT:
id....................0002
raddr.....0000000000686000 eaddr.....F200800030000000
size..............00040000 align.............00001000
valid..1 ros....0 fixlmb.1 seg....0 wimg...2
Dump analysis on CHRP_SMP_PCI POWER_PC POWER_5 machine with 8 available CPU(s)  (64-bit registers)
Processing symbol table...
.......................done
(4)> stat
SYSTEM_CONFIGURATION:
CHRP_SMP_PCI POWER_PC POWER_5 machine with 8 available CPU(s)  (64-bit registers)
SYSTEM STATUS:
sysname... AIX
nodename.. p550a
release... 3
version... 5
build date Jan 10 2006
build time 10:56:32
label..... 0602A_53E
machine... 000B27ACD600
nid....... 0B27ACD6
time of crash: Thu Jul 22 01:17:43 2010
age of system: 14 day, 21 min., 51 sec.
xmalloc debug: disabled
CRASH INFORMATION:
CPU 4 CSA 018FFE00 at time of crash, error code for LEDs: 30000000
pvthread+000C00 STACK:
[00075FEC]v_delpft+000108 (F200800020000008 [??])
[0010AA88]v_relframe+000464 (??, ??, ??)
[001027E4]v_pageout+0006D0 (??, ??, ??)
[00141A20]v_steal+00043C (??, ??, ??, ??)
[00144EF4]v_fblru_scan+0003B8 (??)
[001403D4]v_lru+00035C (??)  
[001414D0]v_memp_lru+00023C (??)
[00207FEC]v_prememp_lru+000020 (??)
[002A2474].backt+000080 ()   
____ Exception (F000000030017780) ____
iar   : 00000000002A23F4  msr   : 80000000000010B2  cr    : 42000024
lr    : 00000000001408D4  ctr   : 0000000000000000  xer   : 00000000
mq    : 00000000  asr   : 00000000F376A001
r0  : 0000000000207FCC  r1  : 0FFFFFFFF4017E90  r2  : 0000000001491C28
r3  : 0000000000000000  r4  : F10001002CBA1100  r5  : 0000000003B90000
r6  : 0000000000000000  r7  : 0000000000000000  r8  : 0000000000000106
r9  : 0000000000000000  r10 : 00000000001408D4  r11 : F000000030017780
r12 : 80000000000010B2  r13 : F10001002CB82400  r14 : 00000000DEADBEEF
r15 : 000000000101A9C0  r16 : 00000000DEADBEEF  r17 : 00000000DEADBEEF
r18 : 00000000DEADBEEF  r19 : 00000000DEADBEEF  r20 : 00000000DEADBEEF
r21 : 00000000DEADBEEF  r22 : 00000000DEADBEEF  r23 : 00000000DEADBEEF
r24 : 00000000DEADBEEF  r25 : 00000000DEADBEEF  r26 : 00000000DEADBEEF
r27 : 00000000DEADBEEF  r28 : 00000000DEADBEEF  r29 : 00000000DEADBEEF
r30 : 0000000003B90000  r31 : 0000000000000000
                             
prev      0000000000000000 stackfix  0000000000000000 int_ticks 00
kjmpbuf   0000000000000000 excbranch 0000000000000000 no_pfault 00
intpri    0B backt     00 flags     00
fpscr     0000000000000000 fpscrx    00000000 fpowner   00
fpeu      00 fpinfo    00 alloc     F000
o_iar     0000000000000000 o_toc     0000000000000000
o_arg1    0000000000000000 o_vaddr   0000000000000000
krlockp   0000000000000000   
Except :                     
 csr   0000000000000000 dsisr 0000000000000000
 esid  0000000000000000 dar   0000000000000000 dsirr 0000000000000106
[002A23F4].backt+000000 ()   
[kdb_get_memory] no real storage @ FFFFFFFF4017EA0
(4)> status                  
CPU     TID  TSLOT     PID  PSLOT  PROC_NAME
  0     E01D     14    600C      6  psmd
  1    12025     18    D01A     13  wait
  2   12809F    296   370C4     55  bpbkar
  3    1502B     21    F01E     15  wait
  4     C019     12    4008      4  lrud
  5     3133  32771    3126  16387  wait
  6     4135  32772    4128  16388  wait
  7     5137  32773    512A  16389  wait
  8-63   Disabled
根据这里分析，宕机的另一个CPU正在做页面交换及bpbkar操作，其中bpbkar是NBU的进程。