Annotation for initializing members per try

[jlink]

Testing Problem

Unlike example-based tests properties often require to freshly initialize a member variable per try - instead of per property. For example:

class MyTests {

   private MyService service;

   @BeforeTry
   void intializeService() {
      service = new Service();
   }

   @Property
   void myProperty(@ForAll String aString) {
      service.call(aString);
      ... // proceed with property code
   }
}

I found myself a few times with initializing the member variable in the declaration:

private MyService service = new Service();

And then wondering why the property did not work as expected.

Suggested Solution

I suggest a new annotation @InitializeBeforeTry for annotating member variables:

@InitializeBeforeTry
private MyService service = new Service();

Alternatively, annotation @BeforeTry could be allowed for member variables.

Discussion

It's just syntactic sugar. Will it really help mitigating the problem of forgetting about the real lifecycle behaviour? Unlike other member variables with initialization in declaration, those variables could not be final.

[vlsi]

I wonder if that is a relevant or different question. I use jqwik to test database, and it looks like it might help to batch "data population" in the database.

In other words:

1. Generate a random "record length". For instance, 42 bytes.
2. Insert the record in the db.
3. Generate a sequence of queries (e.g. 100 queries) and verify if their result is sane. For instance substring(row, begin, end) should be sane (at the end of the day, we know which record was in the DB).
4. Goto 1

This should allow faster iterations (no need to remove and re-insert test data) at cost of "sticking to a single record length".

It tried to model it via @Group, however, it is not clear how to generate inner arbitraries based on the outer ones.

WDYT?

[jlink]

@vlsi I currently cannot see the connection of the problem you describe to @InitializeBeforeTry. But I think I haven't really understood your use case.

Some clarifying questions:

• What is a single try in your scenario? One iteration from 1. to 4. or a single query of the 100 queries mentioned in 3.?
• In what way would you like to generate an inner arbitrary depending on an outer one?

Sorry for being slow in the uptake here.

[vlsi]

In what way would you like to generate an inner arbitrary depending on an outer one?

For instance:

1. Add a row of 42 bytes <-- this is "the first-level arbitrary
2. Then make several attempts to falsify the property. For instance, call read(byte[], int offset, int length). Note that the length of the buffer should probably be related to the record size. In other words, it makes no sense to use 100MiB buffers to read the record of 42 bytes, and it is unlikely to trigger bugs. I am inclined that it is worth trying buffers in the range of 0.142...1042. The same goes for skip(int) API. It makes no sense to randomize across skip(100MiB).

[vlsi]

Sorry for being slow in the uptake here.

No rush, thank you.

I just realized that what I need to test for pgjdbc is not just an InputStream, but it is more like a SeekableByteChannel. In other words, the implementation should be able to handle both reads, writes, seeks, truncates in random order 🙀

[jlink]

Are you thinking of something like that:

@Property
class OuterProperty {

	@ForAll("bufferSize")
	int sizeOfBuffer;

	@Provide
	Arbitrary<Integer> bufferSize() {
		return Arbitraries.integers().between(1, 100);
	}

	@Check
	void aCheckThatUsesSizeOfBuffer() {
		//...
	}

	@Property
	class NestedProperty {

		@ForAll("queries")
		String query;

		@Provide
		Arbitrary<String> queries() {
			return Arbitraries.strings().ofLength(sizeOfBuffer)...
		}

		@Check
		void aCheckThatUsesQueryAndBufferSize() {
			// You can be sure that bufferSize has the same value here as it was used in the queries() method
		}
	}
}

I'd consider that a new kind of class-based property. The nesting may not even be necessary.

[vlsi]

Are you thinking of something like that:

It looks great!

The nesting may not even be necessary

Maybe. Frankly speaking, this is the first time I apply jqwik to pgjdbc, so "nesting" was be my immediate reaction to segregate "database population" from "property falsification attempts". I might have better suggestions once I get more familiar with jqwik

[jlink]

My best guess about how you could solve the problem with jqwik's current capabilities is to generate a pair of bufferSize and whatever depends on it:

@Provide
fun sizeAndQuery() : Arbitrary<Pair<Int, String>> {
    val bufferSize = Int.any(1..100)
    return bufferSize.flatMap { size -> query(size).map { query -> Pair(size, query) }}
}

fun query(int size): Arbitrary<String> = ... // however you generate a query arbitrary for a given size

Most of the time when generated values depend on other generated values, the answer is flatMap or flatCombine.

Then the generated pair (or triple or quadruple or...) can be used to initialize and set up your database. What's not possible, and I think that was your initial thought, is to use a generated bufferSize to set up the database once and then generate new queries for each try that depend on this random size. To allow for that would probably require tweaking the lifecycle of generation and shrinking quite a bit.

[vlsi]

For now, I ended up with

    @Provide
    fun data() =
        Combinators.combine(
            integers().between(0, 20000),
            integers().between(1, 60000),
            longs().greaterOrEqual(1L)
        ).flatAs { size, bufferSize, limit ->
            sequences(
                oneOf(
                    read(),
                    readOffsetLength(size),
                    skip(size.toLong()),
                    mark(size),
                    reset()
                )
            )
                .map { InputStreamTestData(size, bufferSize, limit, it) }
        }

---

By the way, what do you think of the following API?

@Provide
fun bufferSize(): Arbitrary<Int> = ...

@Provide
fun query(@CurentOf("bufferSize") bufferSize: Int) : Arbitrary<String> = ...

@Property
fun queryRuns(@ForAll("bufferSize") bufferSize: Int, @ForAll("query") query: String) {
    ...
} 

In other words, jqwik could infer that queryRuns property depends on bufferSize and query. query depends on bufferSize, so it could infer that query has to be flatMapped from bufferSize.

The edge case would be when user requires multiple "instances" of the buffer size. For instance

@Property
fun queryRuns(
    @ForAll("bufferSize") @ArbitraryId("read buffer") readBufferSize: Int,
    @ForAll("bufferSize") @ArbitraryId("write buffer") writeBufferSize: Int,
     ...
) {

Then it could infer that read and write share the same arbitrary configuration, however, they should be use instances (e.g. different seeds).

What's not possible, and I think that was your initial thought, is to use a generated bufferSize to set up the database once and then generate new queries for each try that depend on this random size

Thank you for the confirmation. For now, the trivial approach of re-initializing the db many times is good enough to capture the bug in the current implementation, so the split is not that pressing.

[jlink]

By the way, what do you think of the following API?

Part of what you sketch here, already works. One can use ForAll parameters in provider methods:

@Property
void fullNames(@ForAll("fullNames") String aName) {
	System.out.println(aName);
}

@Provide
Arbitrary<String> fullNames(@ForAll @AlphaChars @StringLength(5) String firstName, @ForAll @AlphaChars @StringLength(5) String lastName) {
	return Arbitraries.just(firstName + " " + lastName);
}

The other part is sharing values across arbitraries. That's a feature that's been a while in jqwik's (somewhat hidden) backlog. I opened an issue for it: https://github.com/jlink/jqwik/issues/294

[adam-waldenberg]

Just a thought here, but why not expand on this concept and simply allow @BeforeTry @BeforeProperty and @BeforeContainer to be settable also on properties rather than just methods ? I could see how it could be useful not only for tries. As you said, it's syntatic sugar, but it makes it looks slightly nicer.

[jlink]

Maybe you’re right. @BeforeProperty would be redundant though, since standard lifecycle creates a new instance of the test class anyway. Und less it’s allowed for static variables as well…

[jlink]

@adam-waldenberg @BeforeContainer MyType myValue = new MyType() would be the same as a static variable, right? Or would there be a value in having a static value injected as member variable?

[jlink]

Released in 1.7.3-SNAPSHOT