jqwik-team
Use constrained domain models as generation input
Testing Problem
It is currently not possible to use domain models with JSR 303/380 validation annotation as input for the generation process. Duplicating this constraint information in the form of hand-crafted Arbitraries violates the DRY principle and can be a lot of effort for larger models.
Suggested Solution
Use Jakarta Bean Validation as basis for generating constrained domain models.
Derive generic 'companion' Arbitraries from said annotations within jquik. In many cases this should be straightforward. Add helpers to facilitate the end-user development of Arbitraries for custom validation annotations.
If possible such an approach would much enhance testing of total functions.
Discussion
This might be difficult or impossible for some classes of annotations, e.g. regexp-based ones.
If private attributes are annotated - instead of construtor or setter parameters - it's hard to get at this information at runtime. It's even harder to make a connection from annotated parameters to ctor params or setter params. It might be easier to "just" offer automatic validation of generated objects and filter invalid ones out, but rely on standard generation to come up with the initial instances.
@mmerdes Can you give a link to a good reference or overview for JSR 303/380?
An example to show what you mean would be helpful to the viewer who does not know about this JSRs yet.
These annotation go by the offical name of 'Jakarta Bean Validation'. Specs and articles can be found here: https://beanvalidation.org/
Here is a concrete example:
Given a domain class Person with 'Jakarta Bean Validation' annotations like so:
import javax.validation.constraints.Pattern;
import javax.validation.constraints.Size;
class Person {
@Size(min = 2, max = 128)
String name;
@Pattern(regexp = "^(\\+\\d{1,3}( )?)?((\\(\\d{3}\\))|\\d{3})[- .]?\\d{3}[- .]?\\d{4}$")
String phoneNumber;
}
The goal would then be to derive Arbitraries from the @Size and @Pattern annotations.
This would enable us to write such a test:
@Property
void someTestWithValidPersons(@ForAll @Valid Person person) {
//...
}
where a new @Valid annotation would trigger the generation of Person instances satisfying the respective constraints on its fields.
@mmerdes This makes sense I'll have a look at what Jakarta Bean Validation has in store and identify the low hanging fruits. How useful do you think would a partial support for just some annotations be?
I would consider it rather useful - especially if there was a way to combine manual Arbitraries with such annotation-derived ones.
This whole approach could become very powerful - especially if progress could be made regarding the regex-issue: #68
Property-based testing of total functions on constrained types :)
@mmerdes Would you expect the constraints to always apply or would an annotation like @ApplyValidations suite your needs?
I think I would prefer the flexibility of an additional annotation (e.g. @Valid in the Java example above). There might well be cases where a user would not want to trigger the constraints automatically.
To be precise: I think it would make sense to use the 'official' @Valid annotation:
javax.validation.Validfor this purpose - similar to how it is done Spring MVC.
Maybe an extension point would be helpful for this problem:
Instead of @Domain something like @RegisterArbitraryFactory(Function<Object, Arbitrary>).
This function could act as a strategy to map arguments of @Property-methods to their respective Arbitrary in a generic way.
For the problem at hand such a strategy could create a suitable Arbitrary for every supported bean-validation annotation.
(This would replace the usage of @Valid and might be useful for other extensions as well.)
What do you think, @jlink?
The suggested annotation does not work, because annotation attributes cannot be functions, all you can have is a class that implements a function. In that sense @Domain is as good as you can get, because it accepts ˋDomainContextˋ as argument which is a collection of arbitrary providers (and configurators). You can use it to register an arbitrary provider that accepts any object and computes the resulting arbitrary, which is what you want right?
But maybe I’m missing your point?
Here's a simple example:
class Experiments {
@Property
@Domain(GenericDomainContext.class)
void test(@ForAll String aString) {
System.out.println(aString);
}
}
class GenericDomainContext extends DomainContextBase {
@Provide
Arbitrary<?> provideAnything(TypeUsage targetType) {
if (targetType.isAssignableFrom(String.class)) {
return Arbitraries.just("My Value");
}
return null;
}
}
Could be a bit simple if a domain context implementation that also implements ArbitraryProvider would register itself. Then it would look like:
class GenericDomainContext2 extends DomainContextBase implements ArbitraryProvider {
@Override
public boolean canProvideFor(TypeUsage targetType) {
return true;
}
@Override
public Set<Arbitrary<?>> provideFor(TypeUsage targetType, SubtypeProvider subtypeProvider) {
if (targetType.isAssignableFrom(String.class)) {
return Collections.singleton(Arbitraries.just("My Value"));
}
return Collections.emptySet();
}
}
The suggested annotation does not work, because annotation attributes cannot be functions, all you can have is a class that implements a function. In that sense @Domain is as good as you can get, because it accepts ˋDomainContextˋ as argument which is a collection of arbitrary providers (and configurators). You can use it to register an arbitrary provider that accepts any object and computes the resulting arbitrary, which is what you want right?
But maybe I’m missing your point?
Yes, I know that plain Java can not have a function/lambda in the annotation ;)
what I meant was a class implementing the functional interface Function<Object, Arbitrary>.
Sorry for the sloppy syntax.
@jlink Thank you for your explicit examples. Maybe the existing API is powerful enough, after all. Will have a closer look.
With the current snapshot, the solution in GenericDomainContext2 is now possible.
Short introduction to bean validation: https://www.baeldung.com/javax-validation
@mmerdes Now that I think of it, your best option (and the way it's done with other annotation based arbitraries) is to globally register an arbitrary provider that looks approximately like this:
public class ValidatedArbitraryProvider implements ArbitraryProvider {
@Override
public boolean canProvideFor(TypeUsage targetType) {
return targetType.findAnnotation(Validated.class).isPresent();
}
@Override
public Set<Arbitrary<?>> provideFor(TypeUsage targetType, SubtypeProvider subtypeProvider) {
Optional<Validated> optionalValidated = targetType.findAnnotation(Validated.class);
return optionalValidated.map(validated -> {
ValidatedArbitrary<?> arbitrary = ... // Whatever has to be done here
return Collections.<Arbitrary<?>>singleton(arbitrary);
}).orElse(Collections.emptySet());
}
@Override
public int priority() {
// To override standard arbitrary providers
return 5;
}
}
A general background question:
say I wanted to provide arbitraries for a number of Objects, e.g. instances of classes A, B, or C annotated with certain annotations. Now let A have fields of type B which in turn has fields of type C.
Does it suffice to just create arbitrary-providers for A, B, C separately, or would I have to traverse the containment tree myself?
The best way to do that depends on a few things:
- Are B and C generic (specified through type variable) or concrete?
- have instances of B and C additional constraints?
- How do B and C instances get into their containing objects? I think that’s best discussed on concrete examples.
One general thing, though: Fields are never filled automatically by jqwik. Constructor parameters or factory method parameters, however, can be filled.
Maybe this feature should be an addition to @UseType as described in https://jqwik.net/docs/snapshot/user-guide.html#generation-from-a-types-interface
@mmerdes I generalized Arbitraries.forType(..) into Arbitraries.traverse(..) which might be exactly what you need:
Arbitraries.traverse(Class<T> targetType, Function<TypeUsage, Optional<Arbitrary<Object>> parameterResolver): TraverseArbitrary<T>
It's not deployed to snapshot yet. I will probably do it tomorrow.
public interface TraverseArbitrary<T> extends Arbitrary<T> {
/**
* Add another creator (function or constructor) to be used
* for generating values of type {@code T}
*
* @param creator The static function or constructor
* @return new arbitrary instance
*/
TraverseArbitrary<T> use(Executable creator);
/**
* Add public constructors of class {@code T} to be used
* for generating values of type {@code T}
*
* @return new arbitrary instance
*/
TraverseArbitrary<T> usePublicConstructors();
/**
* Add all constructors (public, private or package scope) of class {@code T} to be used
* for generating values of type {@code T}
*
* @return new arbitrary instance
*/
TraverseArbitrary<T> useAllConstructors();
/**
* Add all constructors (public, private or package scope) of class {@code T} to be used
* for generating values of type {@code T}
*
* @param filter Predicate to add only those constructors for which the predicate returns true
* @return the same arbitrary instance
*/
TraverseArbitrary<T> useConstructors(Predicate<? super Constructor<?>> filter);
/**
* Add public factory methods (static methods with return type {@code T})
* of class {@code T} to be used for generating values of type {@code T}
*
* @return new arbitrary instance
*/
TraverseArbitrary<T> usePublicFactoryMethods();
/**
* Add all factory methods (static methods with return type {@code T})
* of class {@code T} to be used for generating values of type {@code T}
*
* @return new arbitrary instance
*/
TraverseArbitrary<T> useAllFactoryMethods();
/**
* Add all factory methods (static methods with return type {@code T})
* of class {@code T} to be used for generating values of type {@code T}
*
* @param filter Predicate to add only those factory methods for which the predicate returns true
* @return new arbitrary instance
*/
TraverseArbitrary<T> useFactoryMethods(Predicate<Method> filter);
/**
* Enable recursive use of traversal:
* If a parameter of a creator function cannot be resolved,
* jqwik will also traverse this parameter's type.
*
* @return new arbitrary instance
*/
TraverseArbitrary<T> enableRecursion(boolean enabled);
}
The shape of TraverseArbitrary has changed:
public interface TraverseArbitrary<T> extends Arbitrary<T> {
interface Traverser {
/**
* Create an arbitrary for a creator parameter.
* Only implement if you do not want to resolve the parameter through its default arbitrary (if there is one)
*
* @param parameterType The typeUsage of the parameter, including annotations
* @return New Arbitrary or {@code Optional.empty()}
*/
default Optional<Arbitrary<Object>> resolveParameter(TypeUsage parameterType) {
return Optional.empty();
}
/**
* Return all creators (constructors or static factory methods) for a type to traverse.
*
* <p>
* If you return an empty set, the attempt to generate a type will be stopped by throwing an exception.
* </p>
*
* @param targetType The target type for which to find creators (factory methods or constructors)
* @return A set of at least one creator.
*/
Set<Executable> findCreators(TypeUsage targetType);
}
/**
* Enable recursive use of traversal:
* If a parameter of a creator function cannot be resolved,
* jqwik will also traverse this parameter's type.
*
* @return new arbitrary instance
*/
TraverseArbitrary<T> enableRecursion();
}
Usage is supposed to look like:
Traverser traverser = .... // Define how to resolve parameters and how to create instances
Arbitrary<MyType> myTypeArbitrary = Arbitraries.traverse(MyType, traverser).enableRecursion();
@mmerdes You can use Fixture Monkey that supports JSR 303/380 validation annotation. It use jqwik for arbitrary object generation. https://github.com/naver/fixture-monkey#example https://naver.github.io/fixture-monkey/docs/v0.3.x/getting-started/#try-it-out
@Property
@Domain(SampleTestSpecs.class)
void giveMeRegisteredWrapper(@ForAll Sample sample) {
...
}
class SampleTestSpecs extends AbstractDomainContextBase {
public static final FixtureMonkey FIXTURE_MONKEY = FixtureMonkey.create();
SampleTestSpecs() {
registerArbitrary(Sample.class, sample());
}
Arbitrary<Sample> sample() {
return FIXTURE_MONKEY.giveMeArbitrary(Sample.class);
}
}
@jlink A team(Naver) I'm working on uses jqwik to write and open source code to help create complex objects. Fixture Monkey depends on jqwik. Thank you.
https://naver.github.io/fixture-monkey/docs/v0.3.x/getting-started/#prerequisites
[Arbitraries.traverse(..)](https://jqwik.net/docs/snapshot/javadoc/net/jqwik/api/Arbitraries.html#traverse(java.lang.Class,net.jqwik.api.arbitraries.TraverseArbitrary.Traverser)
is now available in 1.6.1-SNAPSHOT.
@mmerdes It probably requires some more explanation to be used for your purpose.
Thanks @mhyeon-lee for the hint - will check it out. And thanks @jlink for your continuous support!