cppcheck icon indicating copy to clipboard operation
cppcheck copied to clipboard

refs #122442 - oss-fuzz: removed bogus code generation / adjusted settings

Open firewave opened this issue 1 year ago • 4 comments

It seems the way how the fuzzing works was misunderstood.

The existing code used what the fuzzer was feeding us and tried to incorporate into a generated framework of valid code and passed that to the analysis. It seems it was assumed that the fuzzer is just throwing random data at the code to provoke errors. That is not was is happening. It is collecting coverage data and mutates the input to reach new code paths based on that data. If a new path is detected that mutation is being stored.

With the previous code we were giving data to the analysis which the fuzzer does not know about so the coverage associated with the given data is not correct. Also that data was never stored anywhere so the fuzzer was not actually doing much mutating but we were doing it essentially cancelling out the actual fuzzing. I have not looked at that code but it is possible that it wasn't even deterministically generating data based on the input so that made things even worse.

The code for generating data should probably been used to generate the actual corpus/input data but I do not think taht is necessary. Simply feeding it a few small source files is already enough to generate a lot of data. Only from the 18 files within the samples folder it has generated a corpus of more than 35,000 files (after removing duplicates) so basically just a single file might be enough just for the most basic fuzzing.

firewave avatar Mar 12 '24 14:03 firewave

I am not even trying to run that job with the modified in a PR yet because I have no idea what actually happens if that finds something. That might cause havoc since we essentially never properly fuzzed. Also I want to have fixes in for all known issues I encountered locally so we have a proper baseline.

FYI we already have a small corpus in OSS-Fuzz:

INFO:     1286 files found in /github/workspace/cifuzz-corpus/oss-fuzz-client

firewave avatar Mar 12 '24 14:03 firewave