foundry
foundry copied to clipboard
Reconsider default fuzz and invariant settings
Component
Forge
Describe the feature you would like
As the fuzzer has improved and we've learned more about how to catch various kinds of bugs, we've realized the current default parameters may not be optimal. For example, see comments from @mds1 and @grandizzy starting at https://github.com/foundry-rs/foundry/issues/5868#issuecomment-2091531418:
From my testing, trying to flesh out depth vs. runs relationship a bit:
- Default fuzz settings do not catch it, regardless of number of runs (I only tried up to 10k runs)
- Setting
depth=500
catches it consistently with the default number of runs (256)- Setting
depth=500
withruns=100
occasionally catches it- Setting
depth=10000
withruns=1
occasionally catches it, but seems to be a bit less often than prior bulletSo:
- I think we can close this as it's now catchable with the proper config
- But we probably need a higher default depth than 15, as the depth seems very import, and should reconsider our default invariant config
Agree, we should revisit all defaults and update them to more relevant values, like
depth
to 500 (For example echidna uses a defaultseqLen: 100
andtestLimit: 50000
which is like 500 runs with depth of 100 in foundry terms)shrink_run_limit
defaults now to 2^18 which is not realistic anymore with our new shrinking mechanism (echidna usesshrinkLimit: 5000
). From tests done in https://github.com/foundry-rs/foundry/pull/7756 we can shrink a sequence of 5000 calls in 215 seconds, so I think that default is acceptable- maybe default senders to 3 known addresses as echidna does with
0x1
,0x2
,0x3
addresses- others
Worth mentioning that this kind of failure will be identified must faster when we implement per type fuzzing from state, so new owner address will be exercised right away.
Let's revisit these and update the defaults.
Additional context
No response