`@Spy` fields initalized by `closable = MockitoAnnotations.openMocks(this)` is not cleared by `closable.close()` for Jqwik

[rsampaths16]

Testing Problem

The @Spy annotated fields initalized by closable = MockitoAnnotations.openMocks(this) is not cleared by closable.close()

ex:

package example.package;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.ArrayList;
import net.jqwik.api.ForAll;
import net.jqwik.api.Property;
import net.jqwik.api.constraints.AlphaChars;
import net.jqwik.api.constraints.StringLength;
import net.jqwik.api.lifecycle.AfterTry;
import net.jqwik.api.lifecycle.BeforeTry;
import org.mockito.MockitoAnnotations;
import org.mockito.Spy;

public class ExampleTest {
  @Spy private ArrayList<String> list;
  private AutoCloseable closeable;

  @BeforeTry
  void setUp() {
    closeable = MockitoAnnotations.openMocks(this);
  }

  @AfterTry
  void tearDown() throws Exception {
    closeable.close();
  }

  @Property
  void testExample(@ForAll @AlphaChars @StringLength(min = 5, max = 10) String data) {
    assertThat(list).isEmpty();
    list.add(data);
  }
}

Gives errors

  java.lang.AssertionError:
    Expecting empty but was: ["AAALE"]

                              |-----------------------jqwik-----------------------
tries = 2                     | # of calls to property
checks = 2                    | # of not rejected calls
generation = RANDOMIZED       | parameters are randomly generated
after-failure = SAMPLE_FIRST  | try previously failed sample, then previous seed
when-fixed-seed = ALLOW       | fixing the random seed is allowed
edge-cases#mode = MIXIN       | edge cases are mixed in
edge-cases#total = 4          | # of all combined edge cases
edge-cases#tried = 0          | # of edge cases tried in current run
seed = -8776742494156535285   | random seed to reproduce generated values

Shrunk Sample (9 steps)
-----------------------
  data: "AAAAA"

Original Sample
---------------
  data: "vDraLEb"

  Original Error
  --------------
  java.lang.AssertionError:
    Expecting empty but was: ["AAALE"]

[!NOTE] I've followed https://github.com/jqwik-team/jqwik/issues/261#issuecomment-978912205 to use mocks with Jqwik

Expectation

The expectation was that if a spy class was initialised by openMocks then it'd be torn down explicitly too. i.e., marking null

The same however works in Junit5 ( not sure why? )

[!IMPORTANT] This works as expected in Junit5, for both regular tests & parametrized tests

package example.package;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.ArrayList;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.MockitoAnnotations;
import org.mockito.Spy;

public class ExampleTest {
  @Spy private ArrayList<String> list;
  private AutoCloseable closeable;

  @BeforeEach
  void setUp() {
    closeable = MockitoAnnotations.openMocks(this);
  }

  @AfterEach
  void tearDown() throws Exception {
    closeable.close();
  }

  @Test
  void testA() {
    assertThat(list).isEmpty();
    list.add("A");
  }

  @Test
  void testB() {
    assertThat(list).isEmpty();
    list.add("B");
  }

  @Test
  void testC() {
    assertThat(list).isEmpty();
    list.add("C");
  }

  @Test
  void testD() {
    assertThat(list).isEmpty();
    list.add("D");
  }

  @Test
  void testE() {
    assertThat(list).isEmpty();
    list.add("E");
  }
}

Even for huge number of parametrised tests

package example.package;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;
import org.mockito.MockitoAnnotations;
import org.mockito.Spy;

public class ExampleTest {
  @Spy private ArrayList<String> list;
  private AutoCloseable closeable;

  private static Stream<Arguments> testCases() {
    List<String> args = new ArrayList<>();
    for (int i = 0; i < 100_000; i++) {
      args.add(String.valueOf(i));
    }

    return args.stream().map(Arguments::of);
  }

  @BeforeEach
  void setUp() {
    closeable = MockitoAnnotations.openMocks(this);
  }

  @AfterEach
  void tearDown() throws Exception {
    closeable.close();
  }

  @ParameterizedTest
  @MethodSource("testCases")
  void testExample(String value) {
    assertThat(list).isEmpty();
    list.add(value);
  }
}

Work Around

This can be solved my marking the object explicitly null in @BeforeTry or @AfterTry

package example.package;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.ArrayList;
import net.jqwik.api.ForAll;
import net.jqwik.api.Property;
import net.jqwik.api.constraints.AlphaChars;
import net.jqwik.api.constraints.StringLength;
import net.jqwik.api.lifecycle.AfterTry;
import net.jqwik.api.lifecycle.BeforeTry;
import org.mockito.MockitoAnnotations;
import org.mockito.Spy;

public class ExampleTest {
  @Spy private ArrayList<String> list;
  private AutoCloseable closeable;

  @BeforeTry
  void setUp() {
    // ---> CAN BE DONE HERE
    // list = null;
    closeable = MockitoAnnotations.openMocks(this);
  }

  @AfterTry
  void tearDown() throws Exception {
    closeable.close();
    // ---> OR HERE
    list = null;
  }

  @Property
  void testExample(@ForAll @AlphaChars @StringLength(min = 5, max = 10) String data) {
    assertThat(list).isEmpty();
    list.add(data);
  }
}

Discussion

Discuss advantages and disadvantages of your solution. Compare it to alternative suggestions if there are any.