LibAFL
LibAFL copied to clipboard
Published
20 hours ago •
AFLplusplus
AFLplusplus
SIGSEGV when using asan-cores in qemu_launcher fuzzer
IMPORTANT
- You have verified that the issue to be present in the current
mainbranch yes
% git log | head -n 1
commit a212d66afef398cefb5ab7d1844e935efb702d23
Describe the bug
I get a SIGSEGV when trying to run the qemu_launcher example with asan
To Reproduce Steps to reproduce the behavior:
add cargo-make to the docker file
% echo -e "RUN cargo install cargo-make" >> Dockerfile
build the docker container image
% docker build --platform linux/amd64 -t libafl .
run the docker container, clone LibAFL and run qemu_launcher with asan
% docker run --rm --platform linux/amd64 -it libafl
root@f072d778bf3f:/libafl# cd /tmp
root@f072d778bf3f:/tmp# git clone https://github.com/AFLplusplus/LibAFL.git && cd LibAFL/fuzzers/binary_only/qemu_launcher && cargo make asan
Cloning into 'LibAFL'...
remote: Enumerating objects: 49623, done.
remote: Counting objects: 100% (10040/10040), done.
remote: Compressing objects: 100% (2608/2608), done.
remote: Total 49623 (delta 8480), reused 8273 (delta 7371), pack-reused 39583 (from 1)
Receiving objects: 100% (49623/49623), 22.69 MiB | 5.70 MiB/s, done.
Resolving deltas: 100% (34798/34798), done.
[cargo-make] INFO - cargo make 0.37.18
[cargo-make] INFO - Calling cargo metadata to extract project info
[cargo-make] INFO - Cargo metadata done
[cargo-make] INFO - Project: qemu_launcher
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: asan
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: target_dir
[cargo-make] INFO - Running Task: arch_target_dir
[cargo-make] INFO - Running Task: deps_dir
[cargo-make] INFO - Running Task: zlib_unix_wget
SNIP
[cargo-make] INFO - Running Task: harness
[cargo-make] INFO - Execute Command: "cargo" "build" "--profile" "release" "--features" "x86_64" "--target-dir" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64"
Compiling proc-macro2 v1.0.86
Compiling unicode-ident v1.0.13
Compiling rustversion v1.0.17
SNIP
Compiling qemu_launcher v0.13.2 (/tmp/LibAFL/fuzzers/binary_only/qemu_launcher)
Compiling getset v0.1.3
Compiling clap_derive v4.5.18
Compiling thread_local v1.1.8
Compiling num-derive v0.4.2
Compiling syscall-numbers v4.0.0
Compiling clap v4.5.19
Compiling readonly v0.2.12
Compiling capstone v0.12.0
warning: [email protected]: Qemu not found, cloning with git (d6637939526f453c69f4c6bfe4635feb5dc5c0be)...
warning: [email protected]: Version of llvm-config is 18 but needs to be at least rustc's version (19)! We will (try to) continue to build. Continue at your own risk, or rebuild with a set LLVM_CONFIG_PATH env variable, pointing to a newer version.
warning: [email protected]: wrapper.h has been regenerated.
warning: field `page_filter` is never read
--> /tmp/LibAFL/libafl_qemu/src/modules/edges.rs:360:5
|
357 | pub struct EdgeCoverageModule<AF, PF, V> {
| ------------------ field in this struct
...
360 | page_filter: PF,
| ^^^^^^^^^^^
|
= note: `EdgeCoverageModule` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis
= note: `#[warn(dead_code)]` on by default
warning: `libafl_qemu` (lib) generated 1 warning
Finished `release` profile [optimized + debuginfo] target(s) in 6m 18s
[cargo-make] INFO - Running Task: fuzzer
[cargo-make] INFO - Execute Command: "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development" "--input" "./corpus" "--output" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/" "--log" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt" "--cores" "0" "--asan-cores" "0" "--" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development"
Fuzzing stopped by user. Good bye.
[cargo-make] INFO - Build Done in 456.63 seconds.
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher#
Try run the fuzzer with the --verbose flag
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x200071506a0b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.
Add the debug / log environment variables
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# export QASAN_LOG=1
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# export QASAN_DEBUG=1
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
==13629== QEMU-AddressSanitizer (v0.3)
==13629== Copyright (C) 2019-2021 Andrea Fioraldi <[email protected]>
==13629==
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x200071c12c0b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.
Environment info
root@f3d6e2040375:/tmp/LibAFL/fuzzers/binary_only/qemu_launcher# lsb_release -a && \
arch && \
llvm-config --version && \
rustup toolchain list && \
rustc -V
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm
x86_64
18.1.8
nightly-x86_64-unknown-linux-gnu (default)
1.76.0-x86_64-unknown-linux-gnu
rustc 1.83.0-nightly (06bb8364a 2024-10-01)
Expected behavior
when using asan cores not to cause a SIGSEGV
Additional context
I've also tried llvm-19 because of that warning during the compilation of libafl_qemu_sys
first remove all docker images/container
% docker rmi -f $(docker images -aq) ; docker rm -f $(docker ps -aq); docker system prune -f ; docker rmi $(docker images -aq); docker rm $(docker images -aq)
Then change LLVM version from 18 to 19
% sed -i s/18/19/g ./Dockerfile
% cat Dockerfile | grep 19
# Install clang 19, common build tools
ENV LLVM_VERSION=19
build the container image again
docker build --platform linux/amd64 -t libafl .
clone LibAFL and attempt the qemu_launcher example with asan again
docker run --rm --platform linux/amd64 -it libafl
cd /tmp
git clone https://github.com/AFLplusplus/LibAFL.git && cd LibAFL/fuzzers/binary_only/qemu_launcher && cargo make asan
No llvm version warning, however asan cores still crashes.
export QASAN_LOG=1
export QASAN_DEBUG=1
/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
==13784== QEMU-AddressSanitizer (v0.3)
==13784== Copyright (C) 2019-2021 Andrea Fioraldi <[email protected]>
==13784==
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x20007532860b}
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 0)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.
I'd like to add I did try adding the --privileged flag when running the docker container, which did not appear to change any of the results.
Hello, I don't think so.
docker build --platform linux/amd64 -t libafl .
In the container
cargo install cargo-make
cd /tmp && git clone https://github.com/AFLplusplus/LibAFL.git && cd LibAFL/fuzzers/binary_only/qemu_launcher && cargo make asan
I did get an error about an unresolved import which I fixed by adding the prelude feature to libafl_bolts in Cargo.toml. I then ran cargo make asan again
[cargo-make] INFO - Execute Command: "cargo" "build" "--profile" "release" "--features" "x86_64" "--target-dir" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64"
warning: unused manifest key: profile.release-abort.abort
Compiling libafl_bolts v0.13.2 (/tmp/LibAFL/libafl_bolts)
Compiling libafl_qemu_sys v0.13.2 (/tmp/LibAFL/libafl_qemu/libafl_qemu_sys)
warning: [email protected]: Version of llvm-config is 18 but needs to be at least rustc's version (19)! We will (try to) continue to build. Continue at your own risk, or rebuild with a set LLVM_CONFIG_PATH env variable, pointing to a newer version.
Compiling libafl_qemu v0.13.2 (/tmp/LibAFL/libafl_qemu)
Compiling libafl v0.13.2 (/tmp/LibAFL/libafl)
Compiling libafl_targets v0.13.2 (/tmp/LibAFL/libafl_targets)
Compiling qemu_launcher v0.13.2 (/tmp/LibAFL/fuzzers/binary_only/qemu_launcher)
Finished `release` profile [optimized + debuginfo] target(s) in 2m 49s
[cargo-make] INFO - Running Task: fuzzer
[cargo-make] INFO - Execute Command: "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development" "--input" "./corpus" "--output" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/" "--log" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt" "--cores" "0" "--asan-cores" "0" "--" "/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development"
[Objective #1] (GLOBAL) run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 1, executions: 0, exec/sec: 0.000
(CLIENT) corpus: 0, objectives: 1, executions: 0, exec/sec: 0.000
[Objective #1] (GLOBAL) run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 2, executions: 0, exec/sec: 0.000
(CLIENT) corpus: 0, objectives: 2, executions: 0, exec/sec: 0.000
[Objective #1] (GLOBAL) run time: 0h-0m-1s, clients: 1, corpus: 0, objectives: 3, executions: 0, exec/sec: 0.000
(CLIENT) corpus: 0, objectives: 3, executions: 0, exec/sec: 0.000
[Objective #1] (GLOBAL) run time: 0h-0m-1s, clients: 1, corpus: 0, objectives: 4, executions: 0, exec/sec: 0.000
(CLIENT) corpus: 0, objectives: 4, executions: 0, exec/sec: 0.000
Fuzzing stopped by user. Good bye.
[cargo-make] INFO - Build Done in 174.00 seconds.
If I add the --verbose flag, we see the SIGSEGV as before.
/tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/release/qemu_launcher-development --input ./corpus/ --output /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/ --log /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/output/log.txt --cores 0 --asan-cores 0 --verbose -- /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/target/x86_64/libpng-harness-development
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x2000709a7c0b}
Crashed with signal 11
[Objective #1] (GLOBAL) run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 1, executions: 0, exec/sec: 0.000
(CLIENT) corpus: 0, objectives: 1, executions: 0, exec/sec: 0.000
Context:
Rax: 0x00000000000178 Rbx: 0x007ffffe07f760 Rcx: 0x00000000000178 Rdx: 0x00000000000000
Rsi: 0x00000000000178 Rdi: 0x00000000000178 Rbp: 0x007ffffe07f610 Rsp: 0x007ffffe07f5e8
R8: 0x00000000000000 R9: 0x007ffffd84f680 R10: 0x007ffffd84b878 R11: 0x007ffffd861940
R12: 0x00000000000000 R13: 0x007ffffe07f948 R14: 0x00ffff84d7dc38 R15: 0x007ffffd87d020
Rip: 0x00ffff84d270e0 Rflags: 0x00000000000202
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x2000709a7c0b}
Crashed with signal 11
[Objective #1] (GLOBAL) run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 2, executions: 0, exec/sec: 0.000
(CLIENT) corpus: 0, objectives: 2, executions: 0, exec/sec: 0.000
Context:
Rax: 0x000000000000da Rbx: 0x007ffffe07f760 Rcx: 0x000000000000da Rdx: 0x00000000000000
Rsi: 0x000000000000da Rdi: 0x000000000000da Rbp: 0x007ffffe07f610 Rsp: 0x007ffffe07f5e8
R8: 0x00000000000000 R9: 0x007ffffd84f680 R10: 0x007ffffd84b878 R11: 0x007ffffd861940
R12: 0x00000000000000 R13: 0x007ffffe07f948 R14: 0x00ffff84d7dc38 R15: 0x007ffffd87d020
Rip: 0x00ffff84d270e0 Rflags: 0x00000000000202
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x2000709a7c0b}
Crashed with signal 11
[Objective #1] (GLOBAL) run time: 0h-0m-1s, clients: 1, corpus: 0, objectives: 3, executions: 0, exec/sec: 0.000
(CLIENT) corpus: 0, objectives: 3, executions: 0, exec/sec: 0.000
Context:
Rax: 0x000000000001ab Rbx: 0x007ffffe07f760 Rcx: 0x000000000001ab Rdx: 0x00000000000000
Rsi: 0x000000000001ab Rdi: 0x000000000001ab Rbp: 0x007ffffe07f610 Rsp: 0x007ffffe07f5e8
R8: 0x00000000000000 R9: 0x007ffffd84f680 R10: 0x007ffffd84b878 R11: 0x007ffffd861940
R12: 0x00000000000000 R13: 0x007ffffe07f948 R14: 0x00ffff84d7dc38 R15: 0x007ffffd87d020
Rip: 0x00ffff84d270e0 Rflags: 0x00000000000202
qemu_launcher-development: QEMU internal SIGSEGV {code=MAPERR, addr=0x2000709a7c0b}
Crashed with signal 11
[Objective #1] (GLOBAL) run time: 0h-0m-1s, clients: 1, corpus: 0, objectives: 4, executions: 0, exec/sec: 0.000
(CLIENT) corpus: 0, objectives: 4, executions: 0, exec/sec: 0.000
Context:
Rax: 0x000000000000e4 Rbx: 0x007ffffe07f760 Rcx: 0x000000000000e4 Rdx: 0x00000000000000
Rsi: 0x000000000000e4 Rdi: 0x000000000000e4 Rbp: 0x007ffffe07f610 Rsp: 0x007ffffe07f5e8
R8: 0x00000000000000 R9: 0x007ffffd84f680 R10: 0x007ffffd84b878 R11: 0x007ffffd861940
R12: 0x00000000000000 R13: 0x007ffffe07f948 R14: 0x00ffff84d7dc38 R15: 0x007ffffd87d020
Rip: 0x00ffff84d270e0 Rflags: 0x00000000000202
We imported 0 inputs from disk.
thread 'main' panicked at src/main.rs:20:26:
called `Result::unwrap()` on an `Err` value: Empty("No entries in corpus. This often implies the target is not properly instrumented.", 0: libafl_bolts::Error::empty
at /tmp/LibAFL/libafl_bolts/src/lib.rs:371:34
<libafl::schedulers::powersched::PowerQueueScheduler<C,O> as libafl::schedulers::Scheduler<I,S>>::next
at /tmp/LibAFL/libafl/src/schedulers/powersched.rs:348:17
1: <libafl::schedulers::minimizer::MinimizerScheduler<CS,F,M,O> as libafl::schedulers::Scheduler<<<S as libafl::state::HasCorpus>::Corpus as libafl::corpus::Corpus>::Input,S>>::next
at /tmp/LibAFL/libafl/src/schedulers/minimizer.rs:220:22
2: <libafl::fuzzer::StdFuzzer<CS,F,OF,S> as libafl::fuzzer::Fuzzer<E,EM,ST>>::fuzz_one
at /tmp/LibAFL/libafl/src/fuzzer/mod.rs:777:22
3: libafl::fuzzer::Fuzzer::fuzz_loop
at /tmp/LibAFL/libafl/src/fuzzer/mod.rs:247:13
qemu_launcher::instance::Instance<M>::fuzz
at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/instance.rs:327:13
4: qemu_launcher::instance::Instance<M>::run
at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/instance.rs:290:13
5: qemu_launcher::client::Client::run
at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/client.rs:183:17
6: qemu_launcher::fuzzer::Fuzzer::launch::{{closure}}
at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/fuzzer.rs:145:35
core::ops::function::FnOnce::call_once
at /rustc/27861c429af736ea3a6bb015956c7286071b286d/library/core/src/ops/function.rs:250:5
libafl::events::launcher::Launcher<CF,MT,SP>::launch_with_hooks
at /tmp/LibAFL/libafl/src/events/launcher.rs:289:32
libafl::events::launcher::Launcher<CF,MT,SP>::launch
at /tmp/LibAFL/libafl/src/events/launcher.rs:181:9
qemu_launcher::fuzzer::Fuzzer::launch
at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/fuzzer.rs:150:14
qemu_launcher::fuzzer::Fuzzer::fuzz
at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/fuzzer.rs:81:13
7: qemu_launcher::main
at /tmp/LibAFL/fuzzers/binary_only/qemu_launcher/src/main.rs:20:5
8: core::ops::function::FnOnce::call_once
at /rustc/27861c429af736ea3a6bb015956c7286071b286d/library/core/src/ops/function.rs:250:5
std::sys::backtrace::__rust_begin_short_backtrace
at /rustc/27861c429af736ea3a6bb015956c7286071b286d/library/std/src/sys/backtrace.rs:154:18
9: main
10: __libc_start_call_main
at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
11: __libc_start_main_impl
at ./csu/../csu/libc-start.c:360:3
12: _start
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Fuzzing stopped by user. Good bye.
thread 'main' panicked at /tmp/LibAFL/libafl/src/events/llmp/restarting.rs:637:21:
Fuzzer-respawner: Storing state in crashed fuzzer instance did not work, no point to spawn the next client! This can happen if the child calls `exit()`, in that case make sure it uses `abort()`, if it got killed unrecoverable (OOM), or if there is a bug in the fuzzer itself. (Child exited with: 101)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
@jthorpe6 do you mind trying to run the fuzzer with gdb and checking where the segfault happens on your side? just to check if this the same problem as in #2617
Hey @rmalmain
I have just noticed that I do not see a SIGSEGV if the host OS that I'm running the docker container on is Linux. My main host OS is macOS, so I suspect that Rosetta is doing something when mapping memory.
Using the container on a macOS host, I am unable to run the fuzzer with gdb. This seems to be the problem I'm seeing when trying to debug.
So anyway running an emulator in an emulator sounds really bad for performance, why not just run it on aarch?
oh I don't disagree with the performance hit. The fuzzer I'm working on is x86, so I'm only using an emulator inside of an emulator for development.
Ideally the emulator should cross compile to aarch64 just fine - so you don't necessarily have to develop on the same architecture.
That being said, of course it shouldn't really segfault on emulated x86, either.