foundry icon indicating copy to clipboard operation
foundry copied to clipboard

Reconsider default fuzz and invariant settings

Open mds1 opened this issue 9 months ago • 0 comments

Component

Forge

Describe the feature you would like

As the fuzzer has improved and we've learned more about how to catch various kinds of bugs, we've realized the current default parameters may not be optimal. For example, see comments from @mds1 and @grandizzy starting at https://github.com/foundry-rs/foundry/issues/5868#issuecomment-2091531418:

From my testing, trying to flesh out depth vs. runs relationship a bit:

  • Default fuzz settings do not catch it, regardless of number of runs (I only tried up to 10k runs)
  • Setting depth=500 catches it consistently with the default number of runs (256)
  • Setting depth=500 with runs=100 occasionally catches it
  • Setting depth=10000 with runs=1 occasionally catches it, but seems to be a bit less often than prior bullet

So:

  • I think we can close this as it's now catchable with the proper config
  • But we probably need a higher default depth than 15, as the depth seems very import, and should reconsider our default invariant config

Agree, we should revisit all defaults and update them to more relevant values, like

  • depth to 500 (For example echidna uses a default seqLen: 100 and testLimit: 50000 which is like 500 runs with depth of 100 in foundry terms)
  • shrink_run_limit defaults now to 2^18 which is not realistic anymore with our new shrinking mechanism (echidna uses shrinkLimit: 5000). From tests done in https://github.com/foundry-rs/foundry/pull/7756 we can shrink a sequence of 5000 calls in 215 seconds, so I think that default is acceptable
  • maybe default senders to 3 known addresses as echidna does with 0x1, 0x2, 0x3 addresses
  • others

Worth mentioning that this kind of failure will be identified must faster when we implement per type fuzzing from state, so new owner address will be exercised right away.

Let's revisit these and update the defaults.

Additional context

No response

mds1 avatar May 03 '24 15:05 mds1