Have subcommand for running multiple/all fuzz targets, one at a time

[5225225]

This comes from the https://github.com/rust-fuzz/targets crate, which has a CLI to make running cargo-fuzz on many different targets easier.

Maybe cargo fuzz autorun to run all targets, and cargo fuzz autorun <expression> to only fuzz the targets matching that expression (Maybe use a substring search, or regex).

Have some way to easily run all but a few known broken targets, --exclude <target name/expression>?

They would be run one at a time, and stopped after some timeout and then the next one is moved on to. Maybe allow continuing for the rest of the targets once a bug is found in one, but the default should be to stop after any fuzzer stops.

[Manishearth]

I wonder if we should expose "--run-for-seconds=n" and "--run-for-iterations=n" (better names perhaps) and require one of those be passed in

[fitzgen]

Right now cargo fuzz is essentially a tool for running a single libFuzzer process. This is a relatively simple task.

Changing that to running multiple subprocesses and addressing the fallout from that change will be a significant undertaking that adds a lot of complexity, and essentially requires rewriting this tool.

For example, what do we do with each subprocess's stdout/stderr output? Let them interleave as they will in the parent cargo fuzz's output? Multiplex them line by line? Only show a summary and some stats, like AFL's output? Depending how far we go in copying AFL's interface, that would eventually require building a TUI, which in turn would require testing when we aren't hooked up to a terminal and having a fall back plain text interface.

Or what happens when one subprocess finds a crash? Do kill the other libFuzzer subprocesses? What if another subprocess finds a crash in the time between the first crash we detected and when we are shutting down the other subprocesses? Or, if we don't shut down the other processes when one finds a crash, do we respawn the process that crashed so it can look for other crashes? Do we have to sort and classify crashes now, so that we can tell which crashes are the "same crash" or different ones?

And this is just the stuff off the top of my head.

Now maybe we can avoid most of these questions by being strict about running a single subprocess at a time, round-robin style as the OP describes. But if we support that, I think it is very logical from a user point of view to ask "I am running 8 fuzz targets, one at a time, but I have 8 cores on this machine. Why am I not running them all at once? cargo fuzz should support this." I don't want to be in the position of explaining over and over to each new user who has this idea that doing that is hard and not something the tool was designed for.

So I don't think this feature, as requested, is something we should add support for.

However, I think that if we really want to fully support multiple, concurrent libFuzzer subprocesses, then we should do it right. We (the maintainers of cargo fuzz) should talk about whether that is worth it, and how we would address those questions and anything else we think up. And if after that we decide we want to pursue this further, then we would ideally prototype in another branch or something, so that we can still make regular cargo fuzz releases until we are sure we want to release this new, reimagined version of cargo fuzz.

[ia0]

As a workaround, here's a simple run.sh script that will round-robin through all targets listed in targets.txt (it is created with the list of all targets if it does not exist).

#!/bin/sh
[ -e targets.txt ] || cargo fuzz list > targets.txt
i=1; N="$(cat targets.txt | wc -l)"
while cargo fuzz run "$(head -n$i targets.txt | tail -n1)" "$@"; do
  i=$(( i % N + 1 ))
done

You may call it with ./run.sh -- -max_total_time=600 to run each target for 10 minutes before going to the next one.

One issue with this method is that each time a new target is fuzzed, the fuzzer will go over the corpus which shouldn't produce a different outcome than the last time it did. I could not find a libFuzzer option to skip the corpus. This means you want to set the timeout high enough that running the corpus is negligible compared to the actual fuzzing time.