scapy icon indicating copy to clipboard operation
scapy copied to clipboard

ci: run the fuzz target on PRs

Open evverx opened this issue 9 months ago • 1 comments

using https://google.github.io/oss-fuzz/getting-started/continuous-integration/

It downloads the corpus OSS-Fuzz has accumulated so far (including the test cases that triggered issues in the past) and runs the fuzz target with it. It should help to catch most regressions when PRs are opened.

Prompted by https://github.com/secdev/scapy/pull/4373.

It's a draft because to make it more useful in terms of testing ~~the dissectors should probably be covered first~~ (The dissectors were (partly) covered in https://github.com/google/oss-fuzz/pull/11912).

As expected it triggered

 === Uncaught Python exception: ===
error: unpack requires a buffer of 2 bytes
Traceback (most recent call last):
  File "pcap_fuzzer.py", line 29, in TestOneInput
  File "scapy/utils.py", line 1259, in rdpcap
  File "scapy/utils.py", line 1319, in __call__
  File "scapy/utils.py", line 1883, in __init__
  File "scapy/utils.py", line 1584, in __init__
  File "scapy/utils.py", line 1648, in _read_block_shb
error: unpack requires a buffer of 2 bytes

==38== ERROR: libFuzzer: fuzz target exited
    #0 0x7f1941d7e7f1 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3
    #1 0x7f1941c816e8 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5
    #2 0x7f1941c644cc in fuzzer::Fuzzer::ExitCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:248:3
    #3 0x7f1941a318a6  (/lib/x86_64-linux-gnu/libc.so.6+0x468a6) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce)
    #4 0x7f1941a31a5f in exit (/lib/x86_64-linux-gnu/libc.so.6+0x46a5f) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce)
    #5 0x7f193fa4eaa8 in Py_Exit /tmp/Python-3.8.3/Python/pylifecycle.c:2299:5
    #6 0x7f193fa534b1 in handle_system_exit /tmp/Python-3.8.3/Python/pythonrun.c:658:9
    #7 0x7f193fa534b1 in _PyErr_PrintEx /tmp/Python-3.8.3/Python/pythonrun.c:668:5
    #8 0x55ec2c838b73  (build-out/pcap_fuzzer.pkg+0x3b73)
    #9 0x55ec2c838f10  (build-out/pcap_fuzzer.pkg+0x3f10)
    #10 0x7f1941a0f082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce)
    #11 0x55ec2c8374ad  (build-out/pcap_fuzzer.pkg+0x24ad)

https://github.com/secdev/scapy/actions/runs/8941500042/job/24561958924?pr=4378

evverx avatar May 03 '24 15:05 evverx

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.46%. Comparing base (8ed8647) to head (84867e7).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4378      +/-   ##
==========================================
- Coverage   81.46%   81.46%   -0.01%     
==========================================
  Files         353      353              
  Lines       84477    84477              
==========================================
- Hits        68821    68818       -3     
- Misses      15656    15659       +3     
Files Coverage Δ
scapy/layers/dcerpc.py 90.99% <100.00%> (ø)

... and 4 files with indirect coverage changes

codecov[bot] avatar May 03 '24 15:05 codecov[bot]

Currently CIFuzz triggers

AttributeError: 'NoneType' object has no attribute 'val'
Traceback (most recent call last):
  File "pcap_fuzzer.py", line 31, in TestOneInput
  File "scapy/packet.py", line 1688, in summary
  File "scapy/packet.py", line 1662, in _do_summary
  File "scapy/packet.py", line 1662, in _do_summary
  File "scapy/packet.py", line 1662, in _do_summary
  File "scapy/packet.py", line 1665, in _do_summary
  File "scapy/layers/ldap.py", line 736, in mysummary
AttributeError: 'NoneType' object has no attribute 'val'

but since it's reproducible with the master branch it's ignored and the CI is green here as expected because it isn't introduced in this PR:

2024-07-01 07:43:55,579 - root - INFO - Done downloading latest build.
2024-07-01 07:43:55,580 - root - INFO - Trying to reproduce crash using: /tmp/tmpmj6t60wr/crash-32c922e403e265577a7a671701c4a984f455c808.
2024-07-01 07:44:44,733 - root - INFO - Reproduce command returned: 1. Reproducible on /github/workspace/cifuzz-prev-build/pcap_fuzzer.
2024-07-01 07:44:44,734 - root - INFO - The crash is reproducible on previous build. Code change (pr/commit) did not introduce crash.

All in all I think in its current form it can catch shallow issues when PRs are opened.

(Also https://github.com/google/oss-fuzz/pull/12074 was merged so CIFuzz should be more or less in sync with the scapy repository)

evverx avatar Jul 01 '24 07:07 evverx