Hypothesis-like test case database

[tjltjl]

🚀 Feature Request

Storing failed cases (not seeds) in a database for future runs. When running tests, first run the previous known failed cases, then some failed (and fixed) cases and then new cases.

Motivation

This is one of the features of hypothesis I like the most.

If you have an error that fast-check finds 10% of the time (rare enough to hit the right values), running fast-check again to see if it has been fixed requires you to manually move seeds etc.

Hypothesis has figured out a really nice way to do this (like magic): it stores all failing examples in the .hypothesis directory, see [https://hypothesis.readthedocs.io/en/latest/database.html].

This way, if you hit an error, running hypothesis again right away with the same params will always hit the same error even though finding it originally was unlikely. See the implementation.

(Another thing they appear to use the database for is to try to increase test coverage, by storing those examples that exercise unusual paths).

Example

The process is very smooth:

$ npm run test
< failure >
$ vim source.ts
<...edit...>
$ npm run test
< failure or success, *deterministically* redoing the failing case from before. That
   case will be retried every time until it passes and after that occasionally >

(or a test runner in a different window, watching the files, etc).

[dubzzz]

Hey, While the idea looks great, a version not based on a seed would be hardly feasible as fast-check can generate a wide range of items going from simple integers to complete functions and even stateful runners or async schedulers. I'm flagging it as 'accepted feature' as it may be worthy to think about it for future iterations, but I don't know how far we will be able to go. Thanks a lot for the suggestion

[gruhn]

Could this be problematic in combination with in-browser testing , since there is no direct file system access from the browser?

[dubzzz]

I was seeing that as some kind of plugin external to the core library. But so far I don't really have a plugin system ready