jqwik-team
Allow Sharing of Generated Values Across Arbitraries
Testing Problem
Sometimes a value generated by an arbitrary should be the same one for all usages of this arbitrary in the same try. Currently, this can only be accomplished by flat mapping over this value and handing the value down to all other arbitraries.
It would simplify some properties and arbitrary definitions if there was a simpler way to accomplish the same effect
Suggested Solution
There are at least two situations in which you might want to use this feature:
-
In Arbitrary Creation API. E.g.:
Arbitrary.share(String key)This could create an arbitrary that will only generate a value the first time it is used with the same key! Look at a similar Hypothesis feature
-
When using as property parameters, e.g.:
@Property void myProperty(@ForAll("fullNames") String aString, @ForAll(sharedBy="firstName") String firstName) { } @Provide Arbitrary<String> fullNames() { Arbitrary<String> firstName = Arbitraries.strings().share("firstName"); Arbitrary<String> lastName = Arbitraries.strings().share("lastName"); return Combinators.combine(firstName, lastName).as( (f, l) -> f + " " + l); }
Discussion
Implementation has a few complications:
- One has to make sure that the same sharing Id is only used once per try
- Shrinking may be a problem unless there will be flat mapping under the hood. This has not been fully researched yet, though.
Sketched some half-working code: https://github.com/jlink/jqwik/blob/main/engine/src/test/java/experiments/SharedArbitraryExperiments.java
Frankly speaking, I would love to hide combine and flatMap into jqwik implementation.
Of course, there will always be cases when flatMap would be useful, however, I would like to write code like
@Property
void myProperty(@ForAll("fullNames") String aString, @ForAll("firstNames") String firstName) { }
// It might come from extension or something like that
@Provide
Arbitrary<String> firstNames() {...}
@Provide
Arbitrary<String> lastNames() {...}
@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName) {
return firstName + lastName;
}
Would that feature be useful if shared values couldn't be shrunk?
Without shrinking this is rather easy to implement...
What is the "easy" implementation you have in mind?
I thought the implementation would be to arrange the needed combine behind the scenes, then it should support shrinking, shouldn't it?
The easy implementation is to just store the generated unshrinkable value on first access.
Going for a full flatmap and flat combine behind the scenes would IMO require a data flow analysis across all generated values. I don’t even know if that’s possible. There may be a more hacky solution making use of stores and try lifecycle, but I could go with a first unshrinkable solution, if it provides value.
I see.
@Provide
Arbitrary<String> fullNames() {
Arbitrary<String> firstName = Arbitraries.strings().share("firstName");
Arbitrary<String> lastName = Arbitraries.strings().share("lastName");
return Combinators.combine(firstName, lastName).as( (f, l) -> f + " " + l);
}
it does indeed need dataflow analysis.
However, is dataflow needed if all the "sharing" is done via method parameters and return values?
In other words, the definition like
// combine, works
@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName) {
return just(firstName + lastName);
}
// combine, works
@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName) {
ints(..)
...
vs
@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName, @ForAll int ints) {
means fullNames is a combination of firstNames and lastNames. There's no need to analyze the code, and there's no need to execute the code.
@Provide Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName) { return firstName + lastName; }
That code doesn’t compile though. Jqwik needs a clear criterion to decide if it has to flatMap/combine over parameters or if plain map/combine suffices. A return type String would also work but then you could never map/combine to an Arbitrary object, since this would signal a flatMap/combine.
A new annotation would do the trick, but there are already too many for my taste.
Did some experimentation in https://github.com/jlink/jqwik/blob/main/engine/src/test/java/experiments/SharedArbitraryExperiments.java
One more idea. What if @ForAll could also resolve Arbitrary types. Like
@Provide
Arbitrary<String> aString(@ForAll Arbitrary<Integer> ints) {
return ints.map(i -> i.toString());
}
This would not hide map/flatMap/combine but it could be used directly in provider methods.
What if @ForAll could also resolve Arbitrary types. Like
That would be useful as well.
That was so easy to implement that I just did it. Now this works:
@Property
void test(@ForAll("chessSquares") String square) {
System.out.println(square);
}
@Provide
Arbitrary<String> chessSquares(
@ForAll Arbitrary<@CharRange(from = 'a', to = 'h') Character> rows,
@ForAll Arbitrary<@IntRange(min = 1, max = 8) Integer> columns
) {
return Combinators.combine(rows, columns).as((r, c) -> r.toString() + c);
}
Released and available in "1.6.3-SNAPSHOT"
@vlsi Would @InjectArbitrary be more understandable than @ForAll in this case?
@Inject might make sense if there is more than one type of injectable item (e.g. Arbitrary, some sort of jqwik-related service, something related to test feedback, etc). I don't know if that is the case.