fuzzbench

fuzzbench copied to clipboard

I have successfully built images for quickjs, libarchive, etc, and run local experiment using

PYTHONPATH=. python3 experiment/run_experiment.py \
                                          --experiment-config run.config \
                                          --benchmarks quickjs_eval-2020-01-05 libarchive_libarchive_fuzzer \
                                          --experiment-name test--0 \
                                          --fuzzers aflplusplus \
                                          -mc 4 -rc 4

The run.config file:

# The number of trials of a fuzzer-benchmark pair.
trials: 10

# The amount of time in seconds that each trial is run for.
# 1 day = 24 * 60 * 60 = 86400
max_total_time: 600

# The location of the docker registry.
# FIXME: Support custom docker registry.
# See https://github.com/google/fuzzbench/issues/777
docker_registry: gcr.io/fuzzbench

# The local experiment folder that will store most of the experiment data.
# Please use an absolute path.
experiment_filestore: /tmp/experiment-data

# The local report folder where HTML reports and summary data will be stored.
# Please use an absolute path.
report_filestore: /tmp/report-data

# Flag that indicates this is a local experiment.
local_experiment: true

Then the output just repeats

INFO:root:Reading experiment data from db.                                                                                                                    
INFO:root:Done reading experiment data from db.                                                                                                               
WARNING:root:No snapshot data.                                                                                                                                
INFO:root:Finding trials to schedule.                                          
INFO:root:Starting trials.                                                                                                                                    
INFO:root:Started 0 trials.                                                    
INFO:root:Reading experiment data from db.                                                                                                                    
INFO:root:Done reading experiment data from db.                                                                                                               
WARNING:root:No snapshot data.                                                                                                                                
INFO:root:Finding trials to schedule.                                                                                                                         
INFO:root:Starting trials.                                                                                                                                    
INFO:root:Started 0 trials.
INFO:root:Measuring all trials.
INFO:root:Measuring all trials.
INFO:root:Reading experiment data from db.
INFO:root:Done reading experiment data from db.
WARNING:root:No snapshot data.

I pressed Ctrl+C and inspected the dirs in the container, experiment-folders, measurement-folders and reports are all empty.

On the host, I have

/tmp/experiment-data/                                                          
├── local.db                                                                   
├── test--0  
│   ├── coverage-binaries
│   │   ├── coverage-build-libarchive_libarchive_fuzzer.tar.gz
│   │   └── coverage-build-quickjs_eval-2020-01-05.tar.gz
│   └── input
│       ├── config      
│       │   └── experiment.yaml    
│       └── src.tar.gz

However, there is nothing under /tmp/report-data/. I wonder where went wrong.

Same issue here, have you fix it?

Same issue here, was wondering why this is occurring

same problem, after running the experiment. No data in /tmp/report-data. This directory is empty.

Hmmm...I wonder if the builds succeeded. What does docker image ls | grep fuzzbench output?

same problem

same problem

You mean it outputs nothing?

The /tmp/report-data is empty

After I reboot the machine, the directory /tmp/report-data is disappeared

What does What does docker image ls | grep fuzzbench output? I'm trying to figure out if the builds succeeded. A reboot shouldn't affect this.

root@localhost:~# docker image ls | grep fuzzbench gcr.io/fuzzbench/dispatcher-image latest e62d7c8a2404 10 months ago 3.35GB

root@localhost:~/fuzzbench# ls /tmp snap-private-tmp systemd-private-2eb06cea678542a9b6ba52e9729b49b5-haveged.service-KefDYd systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-logind.service-vFX6TL systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-resolved.service-kP4BSO systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-timesyncd.service-m4auLz systemd-private-2eb06cea678542a9b6ba52e9729b49b5-upower.service-b8wkBq

Interesting, that sounds a lot like the builds failed. We probably should log better. Do you remember what the output from running the experiment was? Can you run it again (with a new name) so you can share with me?

root@localhost:~/fuzzbench# cat start.sh export EXPERIMENT_NAME=afltest

PYTHONPATH=. python3 experiment/run_experiment.py
--experiment-config experiment-config.yaml
--benchmarks libpng_libpng_read_fuzzer bloaty_fuzz_target
--experiment-name afltest
--fuzzers afl aflplusplus

....... src_analysis/ src_analysis/pycache/ src_analysis/benchmark_dependencies.py src_analysis/change_utils.py src_analysis/diff_utils.py src_analysis/experiment_changes.py src_analysis/fuzzer_dependencies.py src_analysis/test_benchmark_dependencies.py src_analysis/test_change_utils.py src_analysis/test_experiment_changes.py src_analysis/test_fuzzer_dependencies.py start.sh test_libs/ test_libs/pycache/ test_libs/test_data/ test_libs/utils.py third_party/ third_party/sancov.py INFO:root:Starting experiment.

If you have a computer now you can directly login into server ssh [email protected] password: unioncity2021.NJ

INFO:root:Starting experiment. /bin/bash: line 1: 13 Killed PYTHONPATH=${WORK}/src python3 ${WORK}/src/experiment/dispatcher.py

Eh I'm not so sure I want to SSH into random boxes. This is a nice attempt to help me repro though.

Now, output looks like this: /bin/bash: line 1: 13 Killed PYTHONPATH=${WORK}/src python3 ${WORK}/src/experiment/dispatcher.py root@e070325ae5c4:/work# start.sh: line 7: 6883 Killed PYTHONPATH=. python3 experiment/run_experiment.py --experiment-config experiment-config.yaml --benchmarks libpng_libpng_read_fuzzer bloaty_fuzz_target --experiment-name afltest --fuzzers afl aflplusplus (.venv) root@localhost:~/fuzzbench#

The server running very slow now.......

Can you do ps aux | grep docker

Maybe try again with a new experiment name and when you call run-experiment, pass a low value to concurrent-builds

root@localhost:~# ls /tmp experiment-data systemd-private-2eb06cea678542a9b6ba52e9729b49b5-haveged.service-KefDYd systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-timesyncd.service-m4auLz report-data systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-logind.service-vFX6TL systemd-private-2eb06cea678542a9b6ba52e9729b49b5-upower.service-b8wkBq snap-private-tmp systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-resolved.service-kP4BSO root@localhost:~# ls /tmp/report-data root@localhost:~# ps aux | grep docker root 799 0.0 0.2 1401876 5272 ? Ssl 01:11 0:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 6890 0.0 0.0 1180700 0 pts/0 Sl 20:10 0:00 docker run -ti --rm -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/experiment-data:/tmp/experiment-data -v /tmp/report-data:/tmp/report-data -e LOCAL_EXPERIMENT=True -e INSTANCE_NAME=d-afltest -e EXPERIMENT=afltest -e SQL_DATABASE_URL=sqlite:////tmp/experiment-data/local.db?check_same_thread=False -e EXPERIMENT_FILESTORE=/tmp/experiment-data -e SNAPSHOT_PERIOD=900 -e REPORT_FILESTORE=/tmp/report-data -e DOCKER_REGISTRY=gcr.io/fuzzbench -e CONCURRENT_BUILDS=30 -e WORKER_POOL_NAME= --shm-size=2g --cap-add=SYS_PTRACE --cap-add=SYS_NICE --name=dispatcher-container gcr.io/fuzzbench/dispatcher-image /bin/bash -c rsync -r "${EXPERIMENT_FILESTORE}/${EXPERIMENT}/input/" ${WORK} && mkdir ${WORK}/src && tar -xvzf ${WORK}/src.tar.gz -C ${WORK}/src && PYTHONPATH=${WORK}/src python3 ${WORK}/src/experiment/dispatcher.py || /bin/bash root 7149 1.8 0.2 45084 4536 pts/0 Sl+ 20:12 0:07 docker build --tag gcr.io/fuzzbench/base-image --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from gcr.io/fuzzbench/base-image --file docker/base-image/Dockerfile . root 7173 0.0 0.0 1302776 0 ? Sl 20:12 0:00 runc --log /var/lib/docker/buildkit/executor/runc-log.json --log-format json run --bundle /var/lib/docker/buildkit/executor/tthpdtgj3g2kt5tct0vy9qe8f tthpdtgj3g2kt5tct0vy9qe8f root 16262 1.5 0.0 7008 140 pts/1 S+ 20:18 0:00 grep --color=auto docker

root@localhost:~# ls /tmp experiment-data systemd-private-2eb06cea678542a9b6ba52e9729b49b5-haveged.service-KefDYd systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-timesyncd.service-m4auLz report-data systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-logind.service-vFX6TL systemd-private-2eb06cea678542a9b6ba52e9729b49b5-upower.service-b8wkBq snap-private-tmp systemd-private-2eb06cea678542a9b6ba52e9729b49b5-systemd-resolved.service-kP4BSO root@localhost:~# ls /tmp/report-data root@localhost:~# ps aux | grep docker root 799 0.0 0.2 1401876 5272 ? Ssl 01:11 0:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 6890 0.0 0.0 1180700 0 pts/0 Sl 20:10 0:00 docker run -ti --rm -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/experiment-data:/tmp/experiment-data -v /tmp/report-data:/tmp/report-data -e LOCAL_EXPERIMENT=True -e INSTANCE_NAME=d-afltest -e EXPERIMENT=afltest -e SQL_DATABASE_URL=sqlite:////tmp/experiment-data/local.db?check_same_thread=False -e EXPERIMENT_FILESTORE=/tmp/experiment-data -e SNAPSHOT_PERIOD=900 -e REPORT_FILESTORE=/tmp/report-data -e DOCKER_REGISTRY=gcr.io/fuzzbench -e CONCURRENT_BUILDS=30 -e WORKER_POOL_NAME= --shm-size=2g --cap-add=SYS_PTRACE --cap-add=SYS_NICE --name=dispatcher-container gcr.io/fuzzbench/dispatcher-image /bin/bash -c rsync -r "${EXPERIMENT_FILESTORE}/${EXPERIMENT}/input/" ${WORK} && mkdir ${WORK}/src && tar -xvzf ${WORK}/src.tar.gz -C ${WORK}/src && PYTHONPATH=${WORK}/src python3 ${WORK}/src/experiment/dispatcher.py || /bin/bash root 7149 1.8 0.2 45084 4536 pts/0 Sl+ 20:12 0:07 docker build --tag gcr.io/fuzzbench/base-image --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from gcr.io/fuzzbench/base-image --file docker/base-image/Dockerfile . root 7173 0.0 0.0 1302776 0 ? Sl 20:12 0:00 runc --log /var/lib/docker/buildkit/executor/runc-log.json --log-format json run --bundle /var/lib/docker/buildkit/executor/tthpdtgj3g2kt5tct0vy9qe8f tthpdtgj3g2kt5tct0vy9qe8f root 16262 1.5 0.0 7008 140 pts/1 S+ 20:18 0:00 grep --color=auto docker

I run it again with a new name

Probably worth killing the old building process that was left over from last experiment.

How can I pass a low value to concurrent-builds? I fuzz them about 2ms. Not very long time

I most will reboot the mechine to run it again

` export EXPERIMENT_NAME=afltest2

PYTHONPATH=. python3 experiment/run_experiment.py
--experiment-config experiment-config.yaml
--benchmarks libpng_libpng_read_fuzzer bloaty_fuzz_target
--experiment-name afltest
--fuzzers afl aflplusplus ~
~ `

How can I pass a low value to concurrent-builds? I fuzz them about 2ms. Not very long time

Look at run_experiment.py --help, I don't have it handy. Then do the number of cores your machine has or less.

2 minutes is too short and you will have other problems. need to do at least 30 minutes