pkg/report: include information about USB buses for USB bugs

[a-nogikh]

Requested in a mailing list.

Hi,

here I understand what is happening, but not why it can happen. Usbnet
checks the endpoint type.

May I request an addition to syzbot? Could you include the output of
"lsusb -v" at the time
of the error condition for USB bugs?

    Regards
        Oliver

[a-nogikh]

Not entirely obvious how to best implement that. We definitely cannot query that information at the time of a crash because the VM is no longer running.

Can syzkaller affect the output of lsusb -v during fuzzing? If not, we could probably query that information as a part of collectMachineInfos and then attach that info to the report if it's a seemingly USB bug.

Cc @xairy

[xairy]

lsusb -v shows connected USB devices. Syzkaller's USB pseudo-syscalls can and do connect new devices, so collecting lsusb -v once during VM startup won't work.

[dvyukov]

And most bugs abort the kernel. Also not clear what are "USB bugs". If kernel could detect "USB bugs", it could dump the necessary info on panics.

[xairy]

Thinking about this more: to be of any use, lsusb -v must be collected exactly at the point when a USB crash occurs. A moment later the buggy USB device might already be disconnected.

Dumping the same info from the kernel is a viable approach. USB bugs can be identified as the ones that "have USB pseudo-syscalls in the reproducer if there's one". However, I don't think upstream will be happy about adding USB-related things to panic reports.

A better solution is collecting usbmon 0u trace when running a reproducer for USB bugs. It's not as visual as lsusb -v, but it provides a more accurate information about what's happening on the USB bus.

[dvyukov]

Would it work if kernel would log attach/detach events? This would be useful for any kernel testing system.

[xairy]

That's what usbmon is for, kind of.

Attaching a USB device is not a single event. It's a process that involves multiple messages between the device and the host. lsusb -v is the way to know the result of that process, and usbmon allows to follow it message-by-message.

Maybe there's a way to make usbmon print all communications to the kernel log.

[dvyukov]

Is it guaranteed to print the info before kernel crashes?

[dvyukov]

lsusb comes from usbutils package and buildroot has BR2_PACKAGE_USBUTILS. I don't have usbmon installed and apt-cache search usbmon does not give me anything.

[xairy]

Is it guaranteed to print the info before kernel crashes?

Not sure. The kernel probably puts packet data into some global queue before processing it. However, a userspace reader might only get to reading this data after the kernel has crashed.

lsusb comes from usbutils package and buildroot has BR2_PACKAGE_USBUTILS. I don't have usbmon installed and apt-cache search usbmon does not give me anything.

You don't need any package, just enable CONFIG_USB_MON and read from /sys/kernel/debug/usb/usbmon/0u.

[a-nogikh]

Another possibility is to enable KDB for our Linux builds. It will start right after a kernel panic and should let us dump the contents of some variables during DiagnoseLinux - hopefully those variables that contain USB bus-related info are also reachable this way.

[dvyukov]

Interesting. What does it take to enable KDB? How can it be communicated? Does it have any other side-effects?

[a-nogikh]

What does it take to enable KDB?

As I understand, we just need to enable some build-time config options.

How can it be communicated?

It should be accessible via the serial port. Actually, we already do a similar thing for OpenBSD and FreeBSD: https://github.com/google/syzkaller/blob/44d1319aab39b23be41fbf75d9d37ef9aaa665f2/vm/vmimpl/openbsd.go#L11-L23

[a-nogikh]

FTR managed to configure a Linux instance on qemu to enter kdb mode on crash.

I took https://github.com/google/syzkaller/blob/master/dashboard/config/linux/upstream-apparmor-kasan.config and the latest Linux kernel source (v5.16).

Add to .config file

CONFIG_CONSOLE_POLL=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y
CONFIG_PANIC_TIMEOUT=0

(CONFIG_PANIC_TIMEOUT=0 seems to be quite important!)

We'll probably also need to connect to it via gdb in this case. It must be possible. Then we'll have access to all the debug info and it'll be easy to evaluate expressions.

[a-nogikh]

The following approach works:

• Enable kdb and kgdb.
• After a panic, kdb starts automatically.
• I type kgdb command, it starts expecting gdb packets (though it seems that this step is unnecessary).
• I run gdb ./vmlinux and then target remote /* serial port */
• It becomes possible to evaluate expressions containing global and per-cpu variables.

(gdb) p *(struct xa_node *)(usb_bus_idr.idr_rt.xa_head)
$4 = {shift = 41 ')', offset = 0 '\000', count = 0 '\000', nr_values = 0 '\000', parent = 0x59e0000000000000, array = 0xdd8ffffffff8ca5, {private_list = {next = 0xdd8ffff88804326, prev = 0xffff88804326}, callback_head = {
      next = 0xdd8ffff88804326, func = 0xffff88804326}}, slots = {0xe000000000000000, 0x3000ffff888017f3, 0x3000ffff888045f2, 0x3000ffff888045f0, 0xb000ffff88801782, 0x3000ffff88804529, 0xb000ffff8880419b, 
    0xb000ffff8880409e, 0xb000ffff8880457b, 0x4000ffff8880178f, 0xffff88801288, 0x4000ffff8880409e, 0x8000ffff8880409e, 0xc000ffff8880161d, 0xc000ffff8880161d, 0xffff888041f5, 0x4000ffff88804579, 0x8000ffff88804579, 
    0xc000ffff888041d0, 0x8000ffff888041d0, 0x8000ffff888041f5, 0xc000ffff8880460a, 0xffff8880460a, 0xffff8880405d, 0x4000ffff88801796, 0x8000ffff88801796, 0xc000ffff8880463f, 0xffff8880463f, 0x4000ffff8880177f, 
    0x8000ffff8880177f, 0x4000ffff88801756, 0x8000ffff8880405d, 0xc000ffff888041a5, 0xc000ffff888041a5, 0x8000ffff88801756, 0xc000ffff88804688, 0x8000ffff88804688, 0xc000ffff8880468f, 0x8000ffff8880468f, 
    0xc000ffff8880169d, 0x8000ffff8880169d, 0xffff88801294, 0x0 <fixed_percpu_data> <repeats 21 times>, 0x1000000000000}, {tags = {{281474909601792}, {0}, {1188950301625810944}}, marks = {{281474909601792}, {0}, {
        1188950301625810944}}}}

If I understand it correctly, all the necessary data is accessible via the usb_bus_idr global variable. Though it would be definitely not easy to traverse it..

One time there was a radix tree-parsing gdb script, but later it was reverted and does not seem to be returned back.

[dvyukov]

There is something called crash, which is special debugger for kernel: crash - Analyze Linux crash dump data or a live system https://linux.die.net/man/8/crash https://www.linuxjournal.com/content/oops-debugging-kernel-panics-0

Also a scriptable drgn debugger: https://github.com/osandov/drgn

Maybe they can simplify extraction of info from dumps (but I did not look at them in detail).