Failed to analyze Xen vmcore file with error: invalid kernel virtual address: 1ef8 type: "fill_pcpu_struct"

[skvl]

Info:

• Commit: 46344aa2f92b07ded52cf9841f8db24dd7fe67d7
• Xen version: 4.16
• Dom0: Ubuntu 18.04.6 (5.4.0-137-generic)
• makedumpfile args: -E -d 0
• Error output:

crash 8.0.2++
...
GNU gdb (GDB) 10.2
...
crash: invalid kernel virtual address: 1ef8  type: "fill_pcpu_struct"
WARNING: cannot fill pcpu_struct.

crash: cannot read cpu_info.

• Part of dump file:

00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  04 00 3e 00 01 00 00 00  00 00 00 00 00 00 00 00  |..>.............|
00000020  40 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |@...............|
00000030  00 00 00 00 40 00 38 00  06 00 00 00 00 00 00 00  |[email protected].........|
00000040  04 00 00 00 00 00 00 00  90 01 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  b0 16 00 00 00 00 00 00  b0 16 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  01 00 00 00 07 00 00 00  |................|
00000080  40 18 00 00 00 00 00 00  00 00 00 00 80 88 ff ff  |@...............|
00000090  00 00 00 00 00 00 00 00  00 f0 09 00 00 00 00 00  |................|
000000a0  00 f4 09 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  01 00 00 00 07 00 00 00  40 08 0a 00 00 00 00 00  |........@.......|
000000c0  00 00 10 00 80 88 ff ff  00 00 10 00 00 00 00 00  |................|
000000d0  00 00 de bf 00 00 00 00  00 00 de bf 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  01 00 00 00 07 00 00 00  |................|
000000f0  40 08 e8 bf 00 00 00 00  00 00 f0 bf 80 88 ff ff  |@...............|
00000100  00 00 f0 bf 00 00 00 00  00 00 10 00 00 00 00 00  |................|
00000110  00 00 10 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  01 00 00 00 07 00 00 00  40 08 f8 bf 00 00 00 00  |........@.......|
00000130  00 00 00 00 81 88 ff ff  00 00 00 00 01 00 00 00  |................|
00000140  00 40 a7 3a 02 00 00 00  00 40 a7 3a 02 00 00 00  |.@.:.....@.:....|
00000150  00 00 00 00 00 00 00 00  01 00 00 00 07 00 00 00  |................|
00000160  40 48 9f fa 02 00 00 00  00 40 a7 ba 83 88 ff ff  |@H.......@......|
00000170  00 40 a7 ba 03 00 00 00  00 c0 58 05 00 00 00 00  |[email protected].....|
00000180  00 c0 58 05 00 00 00 00  00 00 00 00 00 00 00 00  |..X.............|
00000190  05 00 00 00 50 01 00 00  01 00 00 00 43 4f 52 45  |....P.......CORE|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000210  00 00 00 00 00 00 00 00  00 00 00 00 ff ff a5 3a  |...............:|
00000220  03 83 ff ff 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000230  00 00 00 00 30 af a5 3a  03 83 ff ff 00 00 00 00  |....0..:........|
00000240  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000260  00 00 00 00 04 b1 a4 3a  03 83 ff ff 04 00 00 00  |.......:........|
00000270  00 00 00 00 04 00 00 00  00 00 00 00 94 b0 a4 3a  |...............:|
00000280  03 83 ff ff f4 b1 a4 3a  03 83 ff ff 00 00 00 00  |.......:........|
00000290  00 00 00 00 b8 a8 21 40  d0 82 ff ff 08 e0 00 00  |......!@........|
000002a0  00 00 00 00 02 00 00 00  00 00 00 00 30 af a5 3a  |............0..:|
000002b0  03 83 ff ff 10 e0 00 00  00 00 00 00 00 00 00 00  |................|
000002c0  00 00 00 00 00 00 c0 67  82 88 ff ff 00 00 00 00  |.......g........|
000002d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000002f0  00 00 00 00 04 00 00 00  20 00 00 00 02 00 00 01  |........ .......|
00000300  58 65 6e 00 33 00 05 80  00 00 00 00 00 60 ab 72  |Xen.3........`.r|
00000310  13 56 00 00 00 a0 a0 2e  03 00 00 00 60 26 16 00  |.V..........`&..|
00000320  00 00 00 00 04 00 00 00  50 00 00 00 01 00 00 01  |........P.......|
00000330  58 65 6e 00 04 00 00 00  00 00 00 00 10 00 00 00  |Xen.............|
00000340  00 00 00 00 c9 1c bf bf  00 00 00 00 b0 65 bd bf  |.............e..|
00000350  00 00 00 00 70 65 bd bf  00 00 00 00 6d 1c bf bf  |....pe......m...|
00000360  00 00 00 00 8a 1c bf bf  00 00 00 00 00 00 00 00  |................|
00000370  00 00 00 00 00 00 80 bf  00 00 00 00 e7 ac 24 00  |..............$.|
00000380  00 00 00 00 05 00 00 00  50 01 00 00 01 00 00 00  |........P.......|
00000390  43 4f 52 45 00 00 00 00  00 00 00 00 00 00 00 00  |CORE............|
000003a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

...

00000810  04 00 00 00 20 00 00 00  02 00 00 01 58 65 6e 00  |.... .......Xen.|
00000820  33 00 05 80 00 00 00 00  70 a4 61 db 7f 55 00 00  |3.......p.a..U..|
00000830  00 20 5e 96 01 00 00 00  60 26 16 00 00 00 00 00  |. ^.....`&......|
00000840  0f 00 00 00 e4 0f 00 00  00 00 00 00 56 4d 43 4f  |............VMCO|
00000850  52 45 49 4e 46 4f 5f 58  45 4e 00 00 50 41 47 45  |REINFO_XEN..PAGE|
00000860  53 49 5a 45 3d 34 30 39  36 0a 53 59 4d 42 4f 4c  |SIZE=4096.SYMBOL|
00000870  28 64 6f 6d 61 69 6e 5f  6c 69 73 74 29 3d 66 66  |(domain_list)=ff|
00000880  66 66 38 32 64 30 34 30  35 63 34 34 38 30 0a 53  |ff82d0405c4480.S|
00000890  59 4d 42 4f 4c 28 66 72  61 6d 65 5f 74 61 62 6c  |YMBOL(frame_tabl|
000008a0  65 29 3d 66 66 66 66 38  32 64 30 34 30 33 39 39  |e)=ffff82d040399|
000008b0  30 37 38 0a 53 59 4d 42  4f 4c 28 6d 61 78 5f 70  |078.SYMBOL(max_p|
000008c0  61 67 65 29 3d 66 66 66  66 38 32 64 30 34 30 35  |age)=ffff82d0405|
000008d0  64 63 35 34 38 0a 53 49  5a 45 28 70 61 67 65 5f  |dc548.SIZE(page_|
000008e0  69 6e 66 6f 29 3d 33 32  0a 53 49 5a 45 28 64 6f  |info)=32.SIZE(do|
000008f0  6d 61 69 6e 29 3d 33 30  37 32 0a 4f 46 46 53 45  |main)=3072.OFFSE|
00000900  54 28 70 61 67 65 5f 69  6e 66 6f 2e 63 6f 75 6e  |T(page_info.coun|
00000910  74 5f 69 6e 66 6f 29 3d  38 0a 4f 46 46 53 45 54  |t_info)=8.OFFSET|
00000920  28 70 61 67 65 5f 69 6e  66 6f 2e 5f 64 6f 6d 61  |(page_info._doma|
00000930  69 6e 29 3d 32 34 0a 4f  46 46 53 45 54 28 64 6f  |in)=24.OFFSET(do|
00000940  6d 61 69 6e 2e 64 6f 6d  61 69 6e 5f 69 64 29 3d  |main.domain_id)=|
00000950  30 0a 4f 46 46 53 45 54  28 64 6f 6d 61 69 6e 2e  |0.OFFSET(domain.|
00000960  6e 65 78 74 5f 69 6e 5f  6c 69 73 74 29 3d 31 34  |next_in_list)=14|
00000970  34 0a 53 59 4d 42 4f 4c  28 64 6f 6d 5f 78 65 6e  |4.SYMBOL(dom_xen|
00000980  29 3d 66 66 66 66 38 32  64 30 34 30 34 61 31 39  |)=ffff82d0404a19|
00000990  32 38 0a 53 59 4d 42 4f  4c 28 64 6f 6d 5f 69 6f  |28.SYMBOL(dom_io|
000009a0  29 3d 66 66 66 66 38 32  64 30 34 30 34 61 31 39  |)=ffff82d0404a19|
000009b0  32 30 0a 53 59 4d 42 4f  4c 28 70 67 64 5f 6c 34  |20.SYMBOL(pgd_l4|
000009c0  29 3d 66 66 66 66 38 32  64 30 34 30 34 62 37 30  |)=ffff82d0404b70|
000009d0  30 30 0a 00 00 00 00 00  00 00 00 00 00 00 00 00  |00..............|
000009e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

• Related commit: https://github.com/crash-utility/crash/commit/c6f0db666191df2342f536945f89cfcad88d265a

I've seen that there are some issues with Xen support in crash (https://github.com/crash-utility/crash/issues/21#issuecomment-330847410). Though I hope that there are somebody who could help me.

Thanks.

[skvl]

The error result from TSS parsing:

MYDEBUG xen_hyper.c:365:xen_hyper_x86_pcpu_init per_cpu__tss_page:0xffff82d0405c2000
GETBUF(104 -> 0)
MYDEBUG xen_hyper.c:377:xen_hyper_x86_pcpu_init flags=0x400, per_cpu_offset=0, init_tss=0xffff82d0405c2000
<readmem: ffff82d0405c2000, KVADDR, "init_tss", 104, (ROE), 5639e797ce80>
<read_kdump: addr: ffff82d0405c2000 paddr: bfdc2000 cnt: 104>
read_netdump: addr: ffff82d0405c2000 paddr: bfdc2000 cnt: 104 offset: bfd62840
MYDEBUG xen_hyper.c:324:dump_buf Dump buffer of size 104:
        +0x00 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        +0x10 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        +0x20 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        +0x30 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        +0x40 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        +0x50 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        +0x60 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

MYDEBUG xen_hyper.c:392:xen_hyper_x86_pcpu_init machine type x86_64
sp=0, cpu_info=1ef8
MYDEBUG xen_hyper.c:1835:xen_hyper_fill_pcpu_struct
<readmem: 1ef8, KVADDR, "fill_pcpu_struct", 264, (ROE), 5639e7f28760>
crash: invalid kernel virtual address: 1ef8  type: "fill_pcpu_struct"
MYDEBUG xen_hyper.c:1839:xen_hyper_fill_pcpu_struct
WARNING: cannot fill pcpu_struct.

crash: cannot read cpu_info.

[k-hagio]

There may be Xen users on the crash-utility mailing list [1] as said in the linked comment, I'd suggest to post your query to the list. [1] https://www.redhat.com/mailman/listinfo/crash-utility

[k-hagio]

A good news, a Xen developer posted a patchset that fixes several Xen issues, probably related to this. Could you try it? https://listman.redhat.com/archives/crash-utility/2023-March/010544.html

There is a branch that has the patchset: https://github.com/k-hagio/crash/tree/xen-fixes.v1

[k-hagio]

Last week, the latest crash got the patchset for Xen issues, please try it.