Remove fuzzbench local experiment limitations?

[Microsvuln]

Hi!

Is it possible to remove fuzzbench local experiment limitations of the number of fuzzers and benchmarks?

I remember I had experiments before with about 18 benchmarks and 5-8 fuzzers. Since I have access to enough hardware resources to do local experiments that would help me a lot. Now I run with a very limited number of benchmarks / fuzzers & after that, I should repeat this for other benchmarks / fuzzers which needs a lot more time.

Any solution to remove these limitations on the local machine?

Thanks!

[jonathanmetzman]

Is it possible to remove fuzzbench local experiment limitations of the number of fuzzers and benchmarks?

Short of implementing proper local support (which probably won't come before January) I think we might be able to solve some of the issues you reported by reducing parallelism in local builds. It may end up taking longer to do the experiment but at least it won't fail. WDYT?

[Microsvuln]

Yes, I think this would be better to have an experiment even if it takes longer to do.

Also, I think this helps till that proper implementation comes to master.

[andreafioraldi]

What's the state of this issue @jonathanmetzman? On a 64 gb ram and 64 cores VM I cannot start a local experiment with 4 fuzzers and 3 benchs, I'm stuck to 2 benchs at time. I got "ERRO[…] error waiting for container: EOF".

Are there some memory limits in the local experiments too? I'm running some expensive compiler transformations at build time, is it possible that I'm hitting a memory limit of the builder image?

[inferno-chromium]

What's the state of this issue @jonathanmetzman? On a 64 gb ram and 64 cores VM I cannot start a local experiment with 4 fuzzers and 3 benchs, I'm stuck to 2 benchs at time. I got "ERRO[…] error waiting for container: EOF".

Are there some memory limits in the local experiments too? I'm running some expensive compiler transformations at build time, is it possible that I'm hitting a memory limit of the builder image?

We havent got a chance, try adding memory limits in these two places.

Try enforcing some memory limits in two places

1. builder here - https://github.com/google/fuzzbench/blob/master/docker/generate_makefile.py#L109
2. runner here - https://github.com/google/fuzzbench/blob/master/experiment/resources/runner-startup-script-template.sh#L38

[andreafioraldi]

Ok so currently there isn't a memory limit?

[inferno-chromium]

Ok so currently there isn't a memory limit?

Yes right!

[jonathanmetzman]

What's the state of this issue @jonathanmetzman? On a 64 gb ram and 64 cores VM I cannot start a local experiment with 4 fuzzers and 3 benchs, I'm stuck to 2 benchs at time. I got "ERRO[…] error waiting for container: EOF".

Are there some memory limits in the local experiments too? I'm running some expensive compiler transformations at build time, is it possible that I'm hitting a memory limit of the builder image?

Do you know when this happens? Is it during building of images or running fuzzers? I think what's really the underlying problem is locally we assume there enough resources to run everything (even when there almost certainly wont be in large experiments).

Also, if you want to make some changes to fuzzbench infra and are having trouble running it locally, we can run it for you or maybe give you some cloud credit that you might be able to run it yourself.