[meta] bun feature scope: CI only or fuzzing-orchestration helper?

[gasche]

ocaml-bun is currently a tool designed for running fuzzing processes for a CI. Many of its default settings are inadequate for manual invocation or long-running fuzzing and some features that would be useful in those spaces are not provided. Is there a well-defined "feature scope" for bun that includes CI tasks, but not those other concerns, or is it desirable to extend the role of the tool?

I'm interested in having a convenient tool to manually invoke a fuzzing test on my development machine, or to start long-running fuzzer processes on a server. Some of the features in ocaml-bun already (CPU count detection, easy interface for parallel fuzzing) are very convenient for these purposes, so it seemed natural to start by trying to use it.

For example, here is a list of things that I could see myself needing/wishing and implementing in the future:

• not passing to the AFL_NO_UI variable to one of the invoked thread to provide a manual invoker with the fancy continuous output
• getting a command-line option to decide a time to stop the fuzzing where it is easy to ask "fuzz for one day" for example (instead of just a number of seconds as I would assume CI tasks would require)
• adding AFL_SKIP_CPUFREQ by default if necessary to start afl-fuzz successfully on a user machine, showing only a warning
• creating fresh input and output directories implictly if these options are not passed on the command-line (we could even think of supporting fuzz-restarting if an output directory is already found, by storing a hash of the binary in the output directory)

It is of course perfectly fine if bun restricts its scope to the CI case. (Projects with a well-defined scope are easier to maintaing.) There could be a separate tool for end-user invocation of crowbar fuzzer -- with possibly some code-sharing in the orchestration/spawning logic, and the afl-gotcpu usage logic, for example in a shared afl-utils repository.

[yomimono]

As a quick note, bun already creates input and output directories for you. (It should probably pull from /dev/urandom instead of using the constant seed it currently does for input, which will bias the initial test generation toward quotes about rabbits.)

It's funny you mention a shared afl-utils repository - there is https://github.com/rc0r/afl-utils , which is a collection of (non-OCaml, non-Crowbar) tools for launching and managing afl-fuzz in a variety of environments. Some of them are solving some of the same problems as bun (like CPU detection and parallel fuzzing), but none (as far as I could tell) were aimed at easy CI integration, so that's where I aimed bun.

I'm hesitant to re-invent those wheels and would like to keep bun aimed at the novel space combining fuzzing and CI that OCaml and Crowbar have given us.

[trefis]

Some of them are solving some of the same problems as bun (like CPU detection and parallel fuzzing)

Although, you should mention that trying to use these tools will result in a variety of python backtraces until one gets frustrated enough to give up.

[yomimono]

Do you think these failures are specific to fuzzing OCaml or Crowbar programs, or are the programs in question just hard to use/unmaintained/etc?

[trefis]

I think it's totally unrelated to ocaml. Some of the programs require configuration files, and then fail to parse configuration files that are constructed as described in their README. :)