jonathanmetzman

>To compare you should not look at the time but the number of executions. As there is always randomness in scheduling the processes that is what must be matched. Good...

> Sorry to chime in here, but do you have to provide a value for the `-s` param, or will it fall back to a predefined fixed seed? > >...

We continued some of this discussion in [discord](https://discord.com/channels/736989425770168422/736989426298912813/743206729306603572). To summarize what was learned there, @gamozolabs tested the hypothesis that it's non-determinism in fuzz targets that is causing AFL++ to behave...

> btw is there any other fuzzer that can be configured to run determinstic? > because afl + variants can't, honggfuzz cant.... > maybe libfuzzer? not if they select seeds...

> using afl++ 2.66c or newer with llvm 9 or newer and afl-clang-fast/afl-clang-lto with the default instrumentation options produce deterministic results if the target has 100% stability. > However: using...

> yes I know it does. though this is not the only point of entropy. for example afl* and hongfuzz select seeds based on their runtime, where shorter is better....

> The complete list of 100% stability targets: > jsoncpp_jsoncpp_fuzzer, lcms-2017-03-21, libpng-1.2.56, mbedtls_fuzz_dtlsclient, vorbis-2017-12-11, woff2-2016-05-06, zlib_zlib_uncompress_fuzzer > > That is 1/3 which is enough IMHO for a proper analysis Funny,...

> OK I am done with my analysis and you will not like the results. > > I took the data from the google storage bucket with gsutil from `gs://fuzzbench-data/2020-08-14/experiment-folders/lcms-2017-03-21-aflplusplus_same*/trial-*/corpus/corpus-archive-0093.tar.gz`...

Yes I'm aware that the difference is ~20% in the worst cases. As I said: >There are some cases like bloaty where the difference is closer to 20% and I...

I assume the clang cov one isn't finished yet?