fuzzilli

fuzzilli copied to clipboard

If Fuzzilli's --resume is interrupted, whether it by a crash, or CTRL-C, it seems to wipe the corpus that has not yet been imported.

Example:

• Initial coverage before stopping Fuzzilli: 22.80%

ls -l ~/corpus | wc -l
  11086

• Launch Fuzzilli with --resume.

Crash happens or perform CTRL-C mid way through import:

ls -l ~/corpus | wc -l
   9823

• New coverage: 22.29%

The coverage attrition actually also seems to happen even when restarting Fuzzilli after the corpus import.

Initial Coverage: 22.80 Coverage after restart 1 22.29 Coverage after restart 2 21.19

@Zon8Research, on the second issue, are you running Fuzzilli with deterministic mode enabled?

My experience has been that a moderate number of edges found are not deterministic, and will be lost on reimport.

@WilliamParks no I'm not running with deterministic mode enabled.

Heres another example that just happened. First we have a full import, I ctrl-C after the successful import.

[Fuzzer] Imported 0 of 5959
[Fuzzer] Imported 500 of 5959
[Fuzzer] Imported 1000 of 5959
[Fuzzer] Imported 1500 of 5959
[Fuzzer] Imported 2000 of 5959
[Fuzzer] Imported 2500 of 5959
[Fuzzer] Imported 3000 of 5959
[Fuzzer] Imported 3500 of 5959
[Fuzzer] Imported 4000 of 5959
[Fuzzer] Imported 4500 of 5959
[Fuzzer] Imported 5000 of 5959
[Fuzzer] Imported 5500 of 5959
[Cli] Successfully resumed previous state. Corpus now contains 5335 elements

Next time we import, there are 615 less items in the corpus:

[Fuzzer] Imported 0 of 5344

Next, during this import, we actually get a crash in fuzzilli, the corpus is completely wiped out:

[Fuzzer] Imported 0 of 261

There is not even the old_corpus directory or anywhere that I can recover the corpus from.

So there appears to be two issues here:

• The natural corpus attrition everytime you restart fuzzilli and resume/import. I'm not sure why this happens.
• The fact that if a crash in Fuzzilli occurs during the corpus import, it wipes the corpus completely. I believe Fuzzilli's corpus import should be re-engineered as to not delete the corpus at any point, otherwise unexpected crashes will wipe out the time and resources spent building a corpus.

Thanks for the report!

From what I can tell, the 2nd issue (less coverage after import) is most likely working as intended. When resuming, Fuzzilli executes each sample again and only keeps those that trigger new coverage. Samples that trigger edges non-deterministically will thus likely be discarded, resulting in less coverage after the import. using --deterministicCorpus should mitigate that to some degree, but in any case, the discarded samples are probably ones that aren't important anyway. Maybe we should make --deterministicCorpus the default, I'll open a new issue for that.

For the first issue, I agree that we should improve that, although it's probably not a high priority thing? If you want to send a PR, that'd be appreciated though. Maybe the easiest way to "fix" this is to simply leave the old_corpus directory lying around and only delete it if it already exists. So basically, this line would need to move here probably?

Alternatively, I guess the line deleting old_corpus/ could just move a couple lines down until after the import was successful (here) and the code moving corpus/ to /old_corpus should then check if that directory already exists and abort with some useful message if it does?

Closing this issue now, but feel free to reopen (or send a PR) if still relevant!