loom icon indicating copy to clipboard operation
loom copied to clipboard

per-test checkpoint files

Open hawkw opened this issue 3 years ago • 1 comments

Currently, when running with checkpointing enabled, loom will generate a single checkpoint file. When a test fails, cargo test will continue running tests, and may overwrite that checkpoint file. This means that checkpointing can really only be used when running one test at a time.

This is unfortunate, as it prevents workflows where all the tests are run with checkpoints enabled to generate a checkpoint file for each failing test, which can then be re-run.

We should find some way to generate separate checkpoint files for each test, so that running additional tests doesn't clobber a checkpoint file.

hawkw avatar May 10 '22 17:05 hawkw

The tricky part here is generating a unique, durable identifier for the current test.

If cargo test is not run with --test-threads=1, the current thread name should be the name of the test. But, this won't work with --test-threads=1. Another approach might be to have loom::model and loom::model::Buiilder::check have #[track_caller] attributes and capture the source location where check is called from. But this falls down in a case where the user has their own function that wraps Builder::check or model that doesn't also have a #[track_caller] on it.

If we could ensure that loom controls the invocation of these functions, such as by having a special #[loom::test] proc macro, we could ensure that those locations are correct. This is what the proptest crate does for its regression checkpointing system. But, currently, loom works nicely with the standard #[test] attribute, and it would be a shame to give that up...

hawkw avatar May 10 '22 17:05 hawkw