Document how to symbolicate fuzzer stack traces

[MatejKastak]

Hi, first of all cargo-fuzz is awesome :+1:.

I have been playing a little with it. And I found out that resolving memory leaks is not as easy as I thought.

An example:

Searching online for the documentation, I wasn't able to find what this trace means and how to correlate it with the original source code. Not even in LLVM sanitizer docs. After some playing with objdump I found a manually intensive way of retrieving the information one by one.

To simplify this, I have created a PoC tool in Python to do this automatically.

It is located at https://github.com/MatejKastak/cargo-fuzz-sourcer if anyone is interested.

The question :question:

• What do you think the right way to correlate leaks with the code is?
• Is this approach using objdump reasonable?
• Is this project interested in adopting functionality like this? I can help with it.
• Should we extend the documentation?

I hope I am not missing something obvious and reinventing the wheel.

[fitzgen]

Those stacks are the stack at the time that the leaked memory was allocated.

There are some build configurations where you can see symbolicated stack traces instead of just addresses. This makes the information much more actionable and easier to process without using objdump by hand. Unfortunately, I don't remember what the build flags for this config is. It is worth trying non-optimized builds and seeing if that has any affect, as well as using RUSTFLAGS=-g and maybe even CFLAGS=-g. Not totally sure here.

Yes, we should definitely document this once we figure it out.

I don't think we want to adopt a post-processing tool into this repo / tool. IME, post-processing tools, especially of ad-hoc/human readable formats rather than something like JSON, are a pain to maintain across upstream upgrades.

[fitzgen]

https://clang.llvm.org/docs/AddressSanitizer.html#symbolizing-the-reports seems relevant.

Unfortunately, it looks like the llvm-tools rustup component doesn't contain llvm-symbolize, so we can't just recommend rustup component add llvm-tools-preview as a one-size-fits-all solution for users.

[fitzgen]

I had llvm-symbolizer-11 on my $PATH, but not llvm-symbolizer. So I did this:

$ sudo ln -s $(which llvm-symbolizer-11) $(dirname $(which llvm-symbolizer-11))/llvm-symbolizer 

and after that, I get properly symbolicated stack traces for your example leak:

==292699==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x55e43d4529ed in malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:145:3
    rust-fuzz/cargo-fuzz#1 0x55e43d4dcf68 in leak::leak::hdae5dce9ea1bcf19 (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0x132f68)
    rust-fuzz/cargo-fuzz#2 0x55e43d4dd3f5 in leak::eval::h12e51d43de723d01 (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0x1333f5)
    rust-fuzz/cargo-fuzz#3 0x55e43d489187 in rust_fuzzer_test_input (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xdf187)
    rust-fuzz/cargo-fuzz#4 0x55e43d48d9e0 in __rust_try (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xe39e0)
    rust-fuzz/cargo-fuzz#5 0x55e43d48d16f in LLVMFuzzerTestOneInput (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xe316f)
    rust-fuzz/cargo-fuzz#6 0x55e43d494208 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xea208)
    rust-fuzz/cargo-fuzz#7 0x55e43d4998a3 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xef8a3)
    rust-fuzz/cargo-fuzz#8 0x55e43d49a684 in fuzzer::Fuzzer::MutateAndTestOne() (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xf0684)
    rust-fuzz/cargo-fuzz#9 0x55e43d49c847 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xf2847)
    rust-fuzz/cargo-fuzz#10 0x55e43d4befcf in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0x114fcf)
    rust-fuzz/cargo-fuzz#11 0x55e43d3d6de6 in main (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0x2cde6)
    rust-fuzz/cargo-fuzz#12 0x7ff54fbdacb1 in __libc_start_main csu/../csu/libc-start.c:314:16

Indirect leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x55e43d4529ed in malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:145:3
    rust-fuzz/cargo-fuzz#1 0x55e43d4dd023 in leak::leak::hdae5dce9ea1bcf19 (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0x133023)
    rust-fuzz/cargo-fuzz#2 0x55e43d4dd3f5 in leak::eval::h12e51d43de723d01 (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0x1333f5)
    rust-fuzz/cargo-fuzz#3 0x55e43d489187 in rust_fuzzer_test_input (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xdf187)
    rust-fuzz/cargo-fuzz#4 0x55e43d48d9e0 in __rust_try (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xe39e0)
    rust-fuzz/cargo-fuzz#5 0x55e43d48d16f in LLVMFuzzerTestOneInput (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xe316f)
    rust-fuzz/cargo-fuzz#6 0x55e43d494208 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xea208)
    rust-fuzz/cargo-fuzz#7 0x55e43d4998a3 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xef8a3)
    rust-fuzz/cargo-fuzz#8 0x55e43d49a684 in fuzzer::Fuzzer::MutateAndTestOne() (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xf0684)
    rust-fuzz/cargo-fuzz#9 0x55e43d49c847 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0xf2847)
    rust-fuzz/cargo-fuzz#10 0x55e43d4befcf in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0x114fcf)
    rust-fuzz/cargo-fuzz#11 0x55e43d3d6de6 in main (/home/nick/scratch/cargo-fuzz-sourcer/leak/fuzz/target/x86_64-unknown-linux-gnu/release/eval+0x2cde6)
    rust-fuzz/cargo-fuzz#12 0x7ff54fbdacb1 in __libc_start_main csu/../csu/libc-start.c:314:16

SUMMARY: AddressSanitizer: 52 byte(s) leaked in 2 allocation(s).
INFO: to ignore leaks on libFuzzer side use -detect_leaks=0.

[MatejKastak]

Thanks for your research!

I was able to replicate this using llvm-symbolizer. I had no idea such tool existed. Also interesting note is that it accepts command line arguments in LLVM_SYMBOLIZER_OPTS env variable. However, an option like --pretty-print, -p is crashing on me.

Feel free to close this question.

[fitzgen]

I think we should keep this open to track documenting this stuff. I'll also move it to the book repo's issue tracker.