foundry icon indicating copy to clipboard operation
foundry copied to clipboard

feat(fuzz) - add test progress

Open grandizzy opened this issue 1 year ago • 1 comments

Motivation

Closes #585 Closes #3277 Closes #4452

Solution

  • make use of indicatif crate
  • add --show-progress arg (to give more time for feedback / OS tests, then make it default and add a --stream-results flag for legacy output)
  • long running tasks (fuzz and invariant tests) get their own entry / progress bar in test suite progress. Fuzz tests shows [current / total] runs; Invariants show [current / total] runs and then the [current shrink / max shrink] runs (if case)
  • when a test suite is finished it is moved from running tasks to summary section showing the result summary
  • when all test suits are completed then detailed results are streamed

test-out

TBD: rn the fail fast flag doesn't stop test case execution (rayon par iter continue to run until all tasks are completed), need to find a way to exit early when such

grandizzy avatar May 13 '24 09:05 grandizzy

rn the fail fast flag doesn't stop test case execution (rayon par iter continue to run until all tasks are completed), need to find a way to exit early when such

I think we could add a termination check with something like an Arc<AtomicBool> to exit early

mattsse avatar May 13 '24 15:05 mattsse

@DaniPopes please take a look when you have time, would love to get this merged and take feedback from community. Thank you 🙏

grandizzy avatar Jun 06 '24 17:06 grandizzy

needs rebase + clippy

DaniPopes avatar Jun 07 '24 13:06 DaniPopes