fuzzbench

fuzzbench copied to clipboard

I have been trying to identify new interesting benchmarks to evaluate a fuzzer. For this I wrote some scripts that integrate all OSS-Fuzz experiments to fuzzbench. It basically runs the oss_fuzz_benchmark_integration.py and tests that integration by running make test-run-afl-{benchmark_name}.

After integrating all, only about 40 out of >300 projects succeed.

I'm not quite sure, whether this is because of wrong integration on my part or broken experiments.

Thank you very much.

Thank you so much for reporting this. I'll take a look soon. It would be very helpful if you could give me an example of a failing project (I suppose if I try two I'm pretty likely to get a failure). I assume the reason for the failure is related to the change in OSS-Fuzz's builder image to Ubuntu 20.04 while FuzzBench still only supports 16.04. Do your scripts try to do the integration at a specific time?

Yeah I tested things out and I'm pretty sure the timestamps you've picked won't work because they point to a version of OSS-Fuzz that uses a different Ubuntu version than FuzzBench. I'll try to add a warning about this. This is blocked on https://github.com/google/fuzzbench/issues/1275

Thanks for the quick answer. I'm currently running the benchmarks on Ubuntu 20.04, so I guess the problem was on my part. I did a test run, where for each experiment I use 3 different commits (the newest, the oldest and one in the middle). In this test run I didn't see much more success in older commits but I guess the problem then is that I'm currently using Ubuntu 20.04.

I can rerun my scripts on Ubuntu 16.04 and only use commits older than 25/8/2021 and update you on the issue in the next days.

Can you clarify what you mean by "I can rerun my scripts on Ubuntu 16.04", it doesn't matter what your machine is, it matters what docker images are being used. Commits after August 25 2021 use 20.04 and therefore won't work. You said you didn't see more success with older commits, can you please provide the data on this so I can investigate. Maybe there's a problem similar to the one above, but at the end of the day the integration script is best effort and devs may need to do some work on their own to get benchmarks working.

By "FuzzBench still only supports 16.04", I thought Fuzzbench needs to be run on 16.04 and did not think about the docker images. Thus I spend some time on getting this to run and didn't check github. (didn't get an email notification for some reason) ¯_( '-' )_/¯ So please excuse the late answer.

Concerning the older commits: I added the log files of the test run to a repository. For this test run, before integrating an oss-fuzz project, I first checked out to a certain commit in the oss-fuzz submodule. Then after integrating, I again did a test run with afl.

If you need more infos or another test run just notify me. (Although, the test runs will take some time.)

Edit: I think, the issue still exists for older commits before August 25 2021.

Sorry for closing this issue. There's too much in that repo for me to look at. Can you go through the repo find some patterns and then ask me about those?

I looked through some and frankly there's nothing that can be done on FuzzBench's side about some of these, bintray doesn't exist anymore for example so the integration script wont work for builds relying on it.

Yes sure, I can do that. Will take some time though.

I will do a clean rerun with the newest fuzzbench version and use only experiments that have been available before August, while still being available today. I can group them by errors if you want and do a quick explanation of what I've found.

If I haven't sent you results by the end of next week, just ping me again.

Alright, I forgot to prepare this before I went to holiday, so here are the promised results. I reran the scripts and only used commits before August. I added a readme to this repo and interesting projects that currently fail. If I can help you in any way just ping me :)

Hello @jonathanmetzman, is there any update for upgrading Base builder image of fuzzbench as ubuntu 20.04?

I just tried to forcefully integrate wolfssl from oss-fuzz into fuzzbench. Sadly this caused a compilation error during Run fuzzer_build to build the target. wolfssl was not available before August 2021. So there seems to way to easily integrate that benchmark, right?

Oke I got it working by using the following in the benchmark Dockerfile: FROM gcr.io/oss-fuzz-base/base-builder@sha256:1b6a6993690fa947df74ceabbf6a1f89a46d7e4277492addcd45a8525e34be5a

It seems like that image is still based on ubuntu 16.