syzkaller icon indicating copy to clipboard operation
syzkaller copied to clipboard

Syzkaller slows down dramatically after 45 minutes

Open chop0 opened this issue 3 years ago • 1 comments

I'm running syzkaller in a Docker container with Debian Bullseye as the guest and host OS and Linux 5.10. Go 1.18 is being used to build syzkaller, with QEMU for virtualisation. This is the syz-manager config:

{
    "suppressions": ["lost connection to test machine", "no output from test", "panic"],
"image": "/syzkaller/image/debian/buster.img",
    "target": "linux/amd64",
    "workdir": "/syzkaller/workdir",
    "kernel_obj": "/syzkaller/image/linux",
    "sshkey": "/syzkaller/image/debian/buster.id_rsa",
    "syzkaller": "/syzkaller/syzkaller",
    "procs": 32,
"reproduce": false,
"http": "0.0.0.0:80",
    "type": "qemu",
    "vm": {
        "count": 16,
        "kernel": "/syzkaller/image/linux/bzImage",
"cmdline": "net.ifnames=0 mitigations=off",
        "cpu": 4,
        "mem": 2048
    }
}

Syzkaller starts off at about 2-3k executions per second (even with a persistent working directory), but after 45 minutes the speed has decreased to about 300-500 executions per second.

chop0 avatar Jan 29 '22 00:01 chop0

  • Do you have any other metrics that might be of help here? We have a prometheus exporter that helps me see system performance while syzkaller is running.
  • Does the count/procs count make a difference in the throughput?
  • Have you tried wiping the corpus.db and does this issue recur?

oswalpalash avatar Feb 24 '22 18:02 oswalpalash

https://github.com/google/syzkaller/commit/6753db5cdc04330ec9d1a5116b890c19481d69b3 could have been one of the reasons - once syzkaller has to perform a lot of exec fuzz, the required computation resources grew 2-3x.

You also use "procs": 32, with only 2GB of RAM, which is definitely not enough to accomodate that many procs.

Please reopen the issue if the problem persists.

a-nogikh avatar Mar 21 '24 13:03 a-nogikh