Ability to re-run a shrunk case

[vlsi]

Testing Problem

Stateful testing starts with a sequence of 31 actions, and then shrinking takes noticeable time and it fails with an example of 3 or so actions. It would be nice if there was a way to start with a shrunk example right away.

Suggested Solution

I'm not sure how jqwik handles shrinking, however, shrinkSeed might be a base64 bit sequence like: skip 5 shrink suggestions, accept, skip 3 suggestions, accept, ...

For instance:

@Property(shrinking = ShrinkingMode.FULL, seed = "5012573069204752443", shrinkSeed = "XrseKUk6kzcwjYr/o7NPVQ==")

Then jqwik could skip irrelevant shrink attempts, and it could reproduce the failures faster.

[jlink]

Rerunning just the shrunk sample is covered by ˋAfterFailureMode.SAMPLE_ONLYˋ. Won’t that suffice?

The problem with exposing the full shrinking information is that it can be quite lengthy; it’s actually a list of shrink results with potentially a lot of entries.

[vlsi]

`SAMPLE_ONLYˋ would be helpful for local runs.

Would it work for reproducing CI failures?

[jlink]

Would it work for reproducing CI failures?

If you keep the jqwik database around across runs, then yes.

It may be useful, though, to have an explicit way of rerunning a shrunk sample. Given the actual shrinking data, your example would more look like:

@Property(seed = "5012573069204752443", shrinking = {"TFFFFT", "TTTF"})

with the Strings being theoretically very long, in practice usually below 50 chars.

[vlsi]

Just in case, SAMPLE_ONLY fails without providing the reason:

февр. 16, 2022 9:17:29 PM net.jqwik.engine.execution.AfterFailureParametersGenerator logAfterFailureHandling
INFO: After Failure Handling: SAMPLE_FIRST, Previous Generation: <GenerationInfo(5012573069204752443, 18, [size=103])>
февр. 16, 2022 9:17:29 PM net.jqwik.engine.execution.AfterFailureParametersGenerator logFailingOfPreviousSampleGeneration
WARNING: Cannot generate previous falsified sample <GenerationInfo(5012573069204752443, 18, [size=103])>.
	Using previous seed instead.

[jlink]

Is it in the context of stateful generation?

[vlsi]

It is

[jlink]

To be frank, I’ve never considered that case: sample recreation for action sequences. Now that I do, it seems obvious that it cannot work as is. Since building up the actual sequence requires running the sequence. Maybe I’ll find a work-around for the new approach to cover that.

[vlsi]

I am inclined to implement a special Action#toSourceCode so the test prints a semi-copy-pastable fragment that could be added to a test class.

Of course, it does not replace re-runner, however, it is worth baking the discovered bugs into small non-property tests so they do not change as generators improve.

[jlink]

If that can bridge the shortcomings of the current stateful generation it's probably worthwhile.

[jlink]

The capability to regenerate a sequence of stateful changes is a core feature of jqwik 2, the work on which will hopefully start in summer or earlier. So I'm closing this issue.