Unexpected property failure when testing a mutating method

[lucasmdjl]

Hi there, I'm exploring jqwik and encountered some behavior that I found surprising. I've created a minimal test case to demonstrate it.

The goal of the test would be to test HashSet.add(element), which should return true if the set did not already contain the specified element. For the test, I created a provider of Tuple2 with a Set and an Integer that is guaranteed not to be in that set.

import net.jqwik.api.*;
import net.jqwik.api.Tuple.Tuple2;
import java.util.*;
import static org.assertj.core.api.Assertions.*;

public class JqwikTest {

    Arbitrary<Set<Integer>> integerSet() {
        return Arbitraries.integers().list().map(HashSet::new);
    }

    @Provide
    Arbitrary<Tuple2<Set<Integer>, Integer>> setsWithNotContainedElement() {
        return integerSet().flatMap(set ->
            Arbitraries.integers()
                .filter(e -> !set.contains(e))
                .map(e -> Tuple.of(set, e))
        );
    }

    @Property(shrinking = ShrinkingMode.OFF)
    void when_not_in_set_contains_should_be_false(
        @ForAll("setsWithNotContainedElement") Tuple2<Set<Integer>, Integer> setAndElement
    ) {
        var set = setAndElement.get1();
        var element = setAndElement.get2();
        assertThat(set).doesNotContain(element);
    }

    @Property(shrinking = ShrinkingMode.OFF)
    void when_not_in_set_add_should_be_true(
        @ForAll("setsWithNotContainedElement") Tuple2<Set<Integer>, Integer> setAndElement
    ) {
        var set = setAndElement.get1();
        var element = setAndElement.get2();
        assertThat(set.add(element)).isTrue();
    }

}

I would expect both tests to pass. However, the second fails with the following output:

INFO: After Failure Handling: SAMPLE_FIRST, Previous Generation: <GenerationInfo(-4808330830343592445, 72, [])>
timestamp = 2025-07-26T15:43:33.077617, JqwikTest:when not in set add should be true = 
  org.opentest4j.AssertionFailedError:
    Expecting value to be true but was false

                              |-----------------------jqwik-----------------------
tries = 158                   | # of calls to property
checks = 158                  | # of not rejected calls
generation = RANDOMIZED       | parameters are randomly generated
after-failure = SAMPLE_FIRST  | try previously failed sample, then previous seed
when-fixed-seed = ALLOW       | fixing the random seed is allowed
edge-cases#mode = MIXIN       | edge cases are mixed in
edge-cases#total = 81         | # of all combined edge cases
edge-cases#tried = 15         | # of edge cases tried in current run
seed = -4808330830343592445   | random seed to reproduce generated values

Sample
------
  setAndElement: ([0, 2147483647, -1, -2147483647], -2147483647)

The fact that the first test passes demonstrates that the setsWithNotContainedElement provider is working correctly.

Since the failure only occurs in the second test, which mutates the set, my guess is that jqwik might be reusing the set across property tries and generating a batch of not-contained integers before running said tries. Could you clarify if this is the intended behavior? If so, what would be the correct way to test the add method?

Thanks for your time and for the great framework!

[jlink]

@lucasmdjl Thanks for reporting that. The behaviour looks indeed strange and I can replicate it. It has to do with mutating a generated object. I'll dive deeper into it when I find some time.

In case you're running into this problem in a real property, you can circumvent it by "cloning" the generated set before you mutate it:

Set<Integer> set = new HashSet<>(setAndElement.get1());

This seems to resolve the issue in your example.

Moreover, a more powerful approach to testing stateful objects than just mutate them in properties are stateful properties, as described here: https://jqwik.net/docs/current/user-guide.html#stateful-testing

[lucasmdjl]

Thanks for the quick reply and for looking into this. I'll check the workarounds and stay tuned for updates

[jlink]

I dove a bit into the problem and it probably has to do with how flat-mapping handles the re-generation of dependent objects. This could likely be fixed but I currently don't have the time for deep analysis and debugging.

My suggested fix is to wrap the generated set in another HashSet during generation, like that:

@Provide
Arbitrary<Tuple2<Set<Integer>, Integer>> setsWithNotContainedElement() {
	return integerSet().flatMap(set ->
		Arbitraries.integers()
			.filter(e -> !set.contains(e))
			.map(e -> Tuple.of(new HashSet(set), e))
	);
}

This makes sure that the original set is not tampered with in the property method and will not be affected on regeneration.