image icon indicating copy to clipboard operation
image copied to clipboard

Cache Cargo dependencies for fuzzing targets in CI

Open astral4 opened this issue 1 year ago • 6 comments

Addresses #2091.

astral4 avatar Jan 14 '24 04:01 astral4

I don't think this quite works as-is.

I tried triggering a re-run of the afl target to see if it would use the cache, and it seems to have both downloaded/re-compiled all the dependencies and then failed saying:

    Checking image-fuzz-afl v0.0.1 (/home/runner/work/image/image/fuzz-afl)
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
AFL LLVM runtime was not built for Rust rustc-1.77.0-nightly-d78329b; run `cargo afl config --build` to build it.
Error: Process completed with exit code 1.

fintelia avatar Jan 14 '24 06:01 fintelia

~~Seems like cargo-fuzz re-compiles dependencies as well, even though there is no error.~~

EDIT: I think caching dependencies does work with cargo-fuzz. It's just that compiling the final crates (i.e. the contents of fuzz/fuzzers/) takes a long time.

astral4 avatar Jan 14 '24 15:01 astral4

I'm not sure how to get caching to work for cargo-afl. It seems that they include the source of afl++ at build time (see here), which doesn't work nicely with caches in GitHub Actions like Swatinem/rust-cache@v2 or actions/cache@v3.

The current build times for cargo-afl might be acceptable, since that job doesn't seem to be the bottleneck (i.e. the job that finishes last) for CI times.

astral4 avatar Jan 14 '24 21:01 astral4

Other people seem to experience long cargo-fuzz compile times as well - see https://github.com/rust-lang/rust/issues/95240.

Changing this line seems to help a little:

- cargo fuzz run "$format" -- -runs=0;
+ RUSTFLAGS="-C codegen-units=16" cargo fuzz run "$format" -- -runs=0;

(I picked codegen-units=16 because it's the default value for non-incremental builds, according to the rustc docs.)

In the end, if the goal is to reduce CI time, I am no longer sure that caching dependencies is the way to go. Since the cargo-afl and cargo-fuzz jobs use the latest nightly toolchain, dependencies will usually have to be re-compiled between runs anyway.

astral4 avatar Jan 14 '24 21:01 astral4

Since the cargo-afl and cargo-fuzz jobs use the latest nightly toolchain, dependencies will usually have to be re-compiled between runs anyway.

We could switch to a specific, pinned nightly version to mitigate that.

Shnatsel avatar Jan 16 '24 01:01 Shnatsel

I imagine we'd then have to periodically bump the pinned version when it got too old and CI started failing. Not sure that's worth it

fintelia avatar Jan 16 '24 03:01 fintelia