clusterfuzz
clusterfuzz copied to clipboard
google
[honggfuzz] add netdriver support
This is a bit of a guess from my side, in that I'm not entirely sure on what is needed to make this complete. The current state is I can run honggfuzz fuzzers with/without netdrivers locally by way of clusterfuzz.
The requirements written by @jonathanmetzman here https://github.com/google/oss-fuzz/issues/7347 lists some features e.g. (1) testcase and corpus pruning in the cases where libFuzzer is absent and (4) optional part of coverage reports. Are these relevant here?
Thanks for adding this!
This is a bit of a guess from my side, in that I'm not entirely sure on what is needed to make this complete. The current state is I can run honggfuzz fuzzers with/without netdrivers locally by way of clusterfuzz.
The requirements written by @jonathanmetzman here google/oss-fuzz#7347 lists some features e.g. (1) testcase and corpus pruning in the cases where libFuzzer is absent and (4) optional part of coverage reports. Are these relevant here?
Indeed, we don't have corpus minimization support using honggfuzz. We'd ideally want this implemented as part of the minimize_corpus method in the Engine interface.
am happy to integrated the corpus minimization (looks straightforward https://github.com/google/honggfuzz/blob/master/docs/USAGE.md#corpus-minimization--m) -- do you know at this point if there will be anything more missing once that's done?
Added a PoC for corpus minimisation, the logs show:
2022-07-25 14:48:40,387 - run_bot - INFO - Running merge...
2022-07-25 14:48:40,387 - run_bot - INFO - Minimizing honggfuzz corpus - step 1
2022-07-25 14:48:40,387 - run_bot - INFO - Minimizing honggfuzz corpus - step 2
2022-07-25 14:48:40,387 - run_bot - INFO - copying input dir /home/dav/code/bots/bot12/clusterfuzz/bot/inputs/fuzzer-testcases-disk/fuzz_initial_corpus
2022-07-25 14:48:40,481 - run_bot - INFO - Minimizing honggfuzz corpus - step 3
2022-07-25 14:48:40,483 - run_bot - INFO - Minimizing honggfuzz corpus - step 4
2022-07-25 14:48:41,498 - run_bot - INFO - Minimizing honggfuzz corpus - step 5
2022-07-25 14:48:41,530 - run_bot - INFO - Corpus merge finished successfully.
2022-07-25 14:48:45,870 - run_bot - INFO - 436 corpus files uploaded for fuzz.
2022-07-25 14:48:45,916 - run_bot - INFO - Corpus pruned from 1062 to 436 units.
2022-07-25 14:48:45,917 - run_bot - INFO - Processing bad units.
2022-07-25 14:48:45,917 - run_bot - INFO - Found 0 bad units, 0 unique crashes.
2022-07-25 14:48:47,529 - run_bot - INFO - 0 corpus files uploaded for fuzz.
2022-07-25 14:48:47,530 - run_bot - INFO - No files found in shared corpus, skip merge.
2022-07-25 14:48:48,829 - run_bot - INFO - 436 corpus files uploaded for fuzz.
a bit of cleanup needed but besides that WDYT?
Lint should work now, locally I get:
Running: pylint src/clusterfuzz/_internal/bot/fuzzers/honggfuzz/engine.py
|
| --------------------------------------------------------------------
| Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
|
Running: yapf -d src/clusterfuzz/_internal/bot/fuzzers/honggfuzz/engine.py
Running: isort --dont-order-by-type --force-single-line-imports --force-sort-within-sections --line-length=80 -p handlers -p libs -p clusterfuzz -c src/clusterfuzz/_internal/bot/fuzzers/honggfuzz/engine.py
Running: pylint src/clusterfuzz/_internal/bot/fuzzers/libFuzzer/engine.py
thanks for the reviews @jonathanmetzman @oliverchang -- all should be addressed!
One last request: could we add some integration test to https://github.com/google/clusterfuzz/blob/e9ff4b99f82cfdba0d370a06db5c882af9a94313/src/clusterfuzz/_internal/tests/core/bot/fuzzers/honggfuzz/honggfuzz_engine_test.py to help us make sure this doesn't break in the future?
One last request: could we add some integration test to https://github.com/google/clusterfuzz/blob/e9ff4b99f82cfdba0d370a06db5c882af9a94313/src/clusterfuzz/_internal/tests/core/bot/fuzzers/honggfuzz/honggfuzz_engine_test.py to help us make sure this doesn't break in the future?
Yes! Added a crashing test case following a pattern similar to existing crashing test. Integration tests are passing:
clusterfuzz$ python butler.py integration_tests
Running: python butler.py run setup --non-dry-run --local --config-dir=configs/test
| None
| afl fuzzer already exists
| Updating stats metrics.
| libFuzzer fuzzer already exists
| Updating stats metrics.
| honggfuzz fuzzer already exists
| Updating stats metrics.
| googlefuzztest fuzzer already exists
| Updating stats metrics.
| syzkaller fuzzer already exists
| Updating stats metrics.
| Template with name afl already exists.
| Template with name engine_asan already exists.
| Template with name engine_msan already exists.
| Template with name engine_ubsan already exists.
| Template with name honggfuzz already exists. | Template with name googlefuzztest already exists.
| Template with name libfuzzer already exists.
| Template with name syzkaller already exists.
| Template with name prune already exists.
| Done
All end-to-end integration tests passed.
Will confirm once lint passes
One last request: could we add some integration test to https://github.com/google/clusterfuzz/blob/e9ff4b99f82cfdba0d370a06db5c882af9a94313/src/clusterfuzz/_internal/tests/core/bot/fuzzers/honggfuzz/honggfuzz_engine_test.py to help us make sure this doesn't break in the future?
Yes! Added a crashing test case following a pattern similar to existing crashing test. Integration tests are passing:
clusterfuzz$ python butler.py integration_tests Running: python butler.py run setup --non-dry-run --local --config-dir=configs/test | None | afl fuzzer already exists | Updating stats metrics. | libFuzzer fuzzer already exists | Updating stats metrics. | honggfuzz fuzzer already exists | Updating stats metrics. | googlefuzztest fuzzer already exists | Updating stats metrics. | syzkaller fuzzer already exists | Updating stats metrics. | Template with name afl already exists. | Template with name engine_asan already exists. | Template with name engine_msan already exists. | Template with name engine_ubsan already exists. | Template with name honggfuzz already exists. | Template with name googlefuzztest already exists. | Template with name libfuzzer already exists. | Template with name syzkaller already exists. | Template with name prune already exists. | Done All end-to-end integration tests passed.Will confirm once lint passes
Note you need to set INTEGRATION=1 to actually run integration tests.
Note you need to set
INTEGRATION=1to actually run integration tests.
Ah, thanks. The CI is probably failing because of the integration test then. Is it possible to see the failed CI log?
I see:
======================================================================
FAIL: test_fuzz_netdriver_crash (tests.core.bot.fuzzers.honggfuzz.honggfuzz_engine_test.IntegrationTest)
Test netdriver fuzzing that results in a crash.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/workspace/src/clusterfuzz/_internal/tests/core/bot/fuzzers/honggfuzz/honggfuzz_engine_test.py", line 208, in test_fuzz_netdriver_crash
crash.stacktrace)
AssertionError: 'ERROR: AddressSanitizer: heap-use-after-free' not found in '==6892==LeakSanitizer has encountered a fatal error.\n==6892==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1\n==6892==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)\n'
======================================================================
FAIL: test_fuzz_netdriver_crash (tests.core.bot.fuzzers.honggfuzz.honggfuzz_engine_test.UnshareIntegrationTest)
Test netdriver fuzzing that results in a crash.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/workspace/src/clusterfuzz/_internal/tests/core/bot/fuzzers/honggfuzz/honggfuzz_engine_test.py", line 208, in test_fuzz_netdriver_crash
crash.stacktrace)
AssertionError: 'ERROR: AddressSanitizer: heap-use-after-free' not found in '==8735==LeakSanitizer has encountered a fatal error.\n==8735==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1\n==8735==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)\n'
Are you able to reproduce this locally?
Are you able to reproduce this locally?
I was able to reproduce. Am not entirely sure what the underlying issue was but it may be due to an old version of Honggfuzz. After I updated the honggfuzz binary in the test_data directory then it worked. As this worked I also updated the two other binaries in the directory compiled with a fresh honggfuzz version from OSS-Fuzz. I added a new hongfuzz binary which is extracted from the oss-fuzz environment.