Write a "Debugging Hipcheck" RFD

[alilleybrinker]

We really ought to care about the debuggability of Hipcheck, especially considering the transition to a plugin architecture will mean parts of Hipcheck's execution will be out of our immediate control. The things we likely care about include, but are not limited to:

• Structured logging for Hipcheck Core (the non-plugin parts of Hipcheck)
  • Providing a logging API to plugins
• Tracing support
  • We probably ought to try to support both eBPF-based tracing for Linux systems, and dtrace-based tracing for Macs, some BSDs, and Solaris-derived operating systems. The phrase we'd be looking for to identify the right existing libraries to hook into would be "USDT" (Userland Statically Define Tracing). I know there's a good Rust crate called usdt to support dtrace.

The things we probably care about being able to debug include:

• Data flow between Hipcheck core and plugins.
• Interactions with the filesystem API provided by core to plugins.
• Network activity from core and plugins.
• Operation of the incremental computation engine in Hipcheck core.

Recognizing that support for debugging within plugins may be limited by plugins own use of any APIs Hipcheck core exposes to support debugging.

Debugging should not only incorporate failure cases, but also cases of slowness, as a slow tool isn't a usable tool.

There's likely more that I've missed here. The basic idea though is this: we as Hipcheck maintainers, and Hipcheck users as well, should be empowered to understand what Hipcheck is doing so we can reduce the time to track down and address bugs.