Make it easier to integrate dfuzzer into CI

[evverx]

It was briefly discussed in https://github.com/matusmarhefka/dfuzzer/pull/13#issuecomment-1092258508

I think in general dfuzzer isn't ready to be integrated into CI due to some limitations like https://github.com/matusmarhefka/dfuzzer/issues/19 but given that it's capable of discovering crashes like

#0  0x00007f5de18fe88c in __pthread_kill_implementation () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install avahi-0.8-14.fc35.x86_64
(gdb) bt
#0  0x00007f5de18fe88c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007f5de18b16a6 in raise () from /lib64/libc.so.6
#2  0x00007f5de189b7d3 in abort () from /lib64/libc.so.6
#3  0x00007f5de1a8bb16 in _dbus_abort.cold () from /lib64/libdbus-1.so.3
#4  0x00007f5de1ab2250 in _dbus_warn_check_failed () from /lib64/libdbus-1.so.3
#5  0x00007f5de1aa14ef in dbus_message_iter_append_basic () from /lib64/libdbus-1.so.3
#6  0x00007f5de1aa29c6 in dbus_message_append_args_valist () from /lib64/libdbus-1.so.3
#7  0x00007f5de1aa2c1d in dbus_message_append_args () from /lib64/libdbus-1.so.3
#8  0x0000561af879063f in dbus_select_common_methods.part ()
#9  0x0000561af8791e53 in msg_server_impl ()
#10 0x00007f5de1a964a9 in dbus_connection_dispatch () from /lib64/libdbus-1.so.3
#11 0x0000561af8796b44 in dispatch_timeout_callback.lto_priv ()
#12 0x00007f5de1b54d28 in avahi_simple_poll_dispatch () from /lib64/libavahi-common.so.3
#13 0x0000561af8788dff in main ()

in less than 5 seconds even without ASan or Valgrind I think it should be possible to utilize it in its current form.

[evverx]

I reported that bug in https://github.com/lathiat/avahi/issues/375

[evverx]

@mrc0mmand FWIW dfuzzer managed to mess up dbus-broker somehow:

dbus-broker-launch[2660]: dbus-broker: ../src/util/user.c:195: user_free: Assertion `c_assert_result && "c_rbtree_is_empty(&user->usage_tree)"' failed

Sat 2022-04-30 19:45:04 UTC 81338   81   81 SIGABRT none     /usr/bin/dbus-broker                   n/a
Sat 2022-04-30 19:49:33 UTC 81669   81   81 SIGABRT none     /usr/bin/dbus-broker                   n/a
Sat 2022-04-30 19:57:28 UTC 81995   81   81 SIGABRT none     /usr/bin/dbus-broker                   n/a

It wasn't triggered by any dbus method in particular. It just crashed when it was stopped/reloaded by systemd. The backtraces aren't available unfortunately.

[evverx]

@mrc0mmand FYI Looks like dfuzzer was integrated into os-autoinst-distri-opensuse in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14669 a month ago

[evverx]

FWIW dfuzzer managed to mess up dbus-broker somehow

Looks like it was reported yesterday in https://github.com/bus1/dbus-broker/issues/288 and fixed in https://github.com/bus1/dbus-broker/commit/608b259e25ef1348b9e4a8e022c35b5c68d3df98

[evverx]

@mrc0mmand I'm not sure if you're planning to extend dfuzzer but it would probably be interesting to try to figure out how to get it to discover stuff like https://github.com/lathiat/avahi/issues/455, https://github.com/lathiat/avahi/issues/454, https://github.com/lathiat/avahi/issues/453 and https://github.com/lathiat/avahi/issues/451.

https://github.com/lathiat/avahi/issues/452 is a bit different in the sense that that particular method is somewhat hidden and can't be discovered by dfuzzer. (BTW systemd hides its methods too sometimes).

[mrc0mmand]

@mrc0mmand I'm not sure if you're planning to extend dfuzzer but it would probably be interesting to try to figure out how to get it to discover stuff like lathiat/avahi#455, lathiat/avahi#454, lathiat/avahi#453 and lathiat/avahi#451.

lathiat/avahi#452 is a bit different in the sense that that particular method is somewhat hidden and can't be discovered by dfuzzer. (BTW systemd hides its methods too sometimes).

Hah, that's indeed interesting, I'll definitely look into that, thanks for the links.

[evverx]

FWIW Sending big-endian messages on little-endian machines (and vice versa) is interesting too: https://gitlab.freedesktop.org/dbus/dbus/-/issues/417 :-) That would be involved though.

[mrc0mmand]

Oh well, this reminds me of a pitfall I ran into when playing around with a potential "vfuzzer" (https://github.com/systemd/systemd/issues/23785#issuecomment-1170261059) which is the (sad) reality that the quality of the introspection data depends on the bus' willingness to put the necessary stuff together.

In Avahi's case, however, this is a bit more complicated, since the objects seem to be tied to a specific peer, or at least that's what I got from a dbus capture.

For example, creating a new EntryGroup

# busctl call org.freedesktop.Avahi / org.freedesktop.Avahi.Server EntryGroupNew
o "/Client15/EntryGroup1"

emits

‣ Type=method_call  Endian=l  Flags=4  Version=1 Cookie=2  Timestamp="Tue 2023-04-25 21:15:06.302187 UTC"
  Sender=:1.2197  Destination=org.freedesktop.Avahi  Path=/  Interface=org.freedesktop.Avahi.Server  Member=EntryGroupNew
  UniqueName=:1.2197
  MESSAGE "" {
  };

‣ Type=method_return  Endian=l  Flags=1  Version=1 Cookie=2837  ReplyCookie=2  Timestamp="Tue 2023-04-25 21:15:06.302227 UTC"
  Sender=:1.1067  Destination=:1.2197
  UniqueName=:1.1067
  MESSAGE "o" {
          OBJECT_PATH "/Client15/EntryGroup1";
  };

where :1.2197 is "us". Since calling busctl again creates a new session, trying to call "IsEmpty()" on the newly created group object fails:

# busctl call org.freedesktop.Avahi /Client14/EntryGroup1 org.freedesktop.Avahi.EntryGroup IsEmpty
Call failed: Method "IsEmpty" with signature "" on interface "org.freedesktop.Avahi.EntryGroup" doesn't exist

since we have a different ID:

‣ Type=method_call  Endian=l  Flags=4  Version=1 Cookie=2  Timestamp="Tue 2023-04-25 21:17:04.432179 UTC"
  Sender=:1.2198  Destination=org.freedesktop.Avahi  Path=/Client14/EntryGroup1  Interface=org.freedesktop.Avahi.EntryGroup  Member=GetState
  UniqueName=:1.2198
  MESSAGE "" {
  };

‣ Type=error  Endian=l  Flags=1  Version=1 Cookie=2838  ReplyCookie=2  Timestamp="Tue 2023-04-25 21:17:04.432217 UTC"
  Sender=:1.1067  Destination=:1.2198
  ErrorName=org.freedesktop.DBus.Error.UnknownObject  ErrorMessage="Method "GetState" with signature "" on interface "org.freedesktop.Avahi.EntryGroup" doesn't exist
"
  UniqueName=:1.1067
  MESSAGE "s" {
          STRING "Method "GetState" with signature "" on interface "org.freedesktop.Avahi.EntryGroup" doesn't exist
";
  };

i.e. now "we" are :1.2198.

However, if I create a simple python script, that uses one bus connection for both calls, I get the expected result:

#!/bin/python3

from pydbus import SystemBus

bus = SystemBus()
avahi = bus.get("org.freedesktop.Avahi", "/")
obj_name = avahi.EntryGroupNew()
print(obj_name)
obj = bus.get("org.freedesktop.Avahi", obj_name)
print(obj.IsEmpty())

# ./repro.py 
/Client16/EntryGroup1
True

So, uh, yeah, this makes things slightly more complicated :)

[mrc0mmand]

And to expand on the introspection stuff - for example, in systemd, we append a list of child nodes to the introspection data when generating the response [0]. However, Avahi serves a static XML for introspection requests [1][2][...] so even if we somehow managed to deal with the peer-related stuff above, the object still wouldn't be "automagically" discoverable.

[0] https://github.com/systemd/systemd/blob/main/src/libsystemd/sd-bus/bus-introspect.c#L113 [1] https://github.com/lathiat/avahi/blob/d1e71b320d96d0f213ecb0885c8313039a09f693/avahi-daemon/dbus-protocol.c#L1003-L1004 [2] https://github.com/lathiat/avahi/blob/6395c45bcd1210cbbb23fc92294fd61e968daaf2/avahi-daemon/dbus-entry-group.c#L104-L106

[evverx]

systemd, we append a list of child nodes to the introspection data

Unless systemd tries to hide something: https://github.com/systemd/systemd/commit/fb22861da1866f7fd47c5d9c3744d527a44e2e06 :-)

Seriously though I think the problem is that for those nodes to be generated dfuzzer has to create them first (for example dfuzzer can't easily catch issues like https://github.com/systemd/systemd/issues/24114) so those nodes are effectively hidden. I agree that it's different from avahi though. With avahi it should probably be possible to infer methods by capturing messages its clients normally send but it can't be automated easily.

[mrc0mmand]

systemd, we append a list of child nodes to the introspection data

Unless systemd tries to hide something: systemd/systemd@fb22861 :-)

Hah, that one slipped past me. And I can't even run dfuzzer on it, since we always do an introspection, which, of course, doesn't work here:

# dfuzzer -n org.freedesktop.systemd1 -o /org/freedesktop/MemoryAllocation1 -i org.freedesktop.MemoryAllocation1 -t GetMallocInfo
[SESSION BUS]
[PROCESS: /usr/lib/systemd/systemd]
[CONNECTED TO PID: 698]
Object: /org/freedesktop/MemoryAllocation1
 Interface: org.freedesktop.MemoryAllocation1
Error while calling method 'org.freedesktop.DBus.Introspectable.Introspect': GDBus.Error:org.freedesktop.DBus.Error.UnknownObject: Unknown object '/org/freedesktop/MemoryAllocation1'.
[SYSTEM BUS]
[PROCESS: /usr/lib/systemd/systemd]
[CONNECTED TO PID: 1]
Object: /org/freedesktop/MemoryAllocation1
 Interface: org.freedesktop.MemoryAllocation1
Error while calling method 'org.freedesktop.DBus.Introspectable.Introspect': GDBus.Error:org.freedesktop.DBus.Error.UnknownObject: Unknown object '/org/freedesktop/MemoryAllocation1'.
Exit status: 1

Maybe we should just run in a "one-shot" mode if we get a full bus quadruplet (i.e. bus, object, interface, method).

Seriously though I think the problem is that for those nodes to be generated dfuzzer has to create them first (for example dfuzzer can't easily catch issues like systemd/systemd#24114) so those nodes are effectively hidden. I agree that it's different from avahi though. With avahi it should probably be possible to infer methods by capturing messages its clients normally send but it can't be automated easily.

Given the systemd case I though about adding something like "user-guided" mode to dfuzzer, where you would use a file containing quadruplets (bus, object, interface, method) to guide dfuzzer which methods to fuzz, to possibly cover these corner cases. But apart from it being way too manual, it still wouldn't work for Avahi, since even though we would use one bus for all calls, the object name is still dynamic, sigh.

[evverx]

Maybe we should just run in a "one-shot" mode if we get a full bus quadruplet (i.e. bus, object, interface, method)

I think it makes sense in the sense that it shouldn't be necessary to call Introspect in this particular case but on the other hand it's probably easier to just use busctl call/gdbus call for that thanks to their bash autocompletion and so on.

Given the systemd case I though about adding something like "user-guided" mode to dfuzzer, where you would use a file containing quadruplets (bus, object, interface, method) to guide dfuzzer which methods to fuzz

I think it should help but it seems that there should be a way to go further than that by making it possible to lead dfuzzer to certain points (by, say, replaying more or less valid messages) and then throwing gibberish. It doesn't work with services like networkd where actual leases should be created outside of its D-Bus API though.

But apart from it being way too manual, it still wouldn't work for Avahi, since even though we would use one bus for all calls, the object name is still dynamic

Agreed.

[evverx]

To judge from https://bugs.launchpad.net/ubuntu/+source/dbus-broker/+bug/2015538/comments/6 the Ubuntu folks took it to the next level. I guess at some point it should be documented that to test stuff on Ubuntu the AppArmor stuff should be relaxed a bit to actually fuzz stuff there (instead of testing whether AppArmor confines things properly).