Inherited nested tests do not trigger refresh

[Zincfox]

I believe that when nested tests are discovered through inheritance (abstract A has nested N, subclass S inherits nested N) the nested tests do not respect RefreshMode.BEFORE_EACH_METHOD.

The original project where I encountered this was a kotlin-spring project with Flyway and the R2DBC adapter from #121, though I could reproduce this issue with jdbc (see below). There it could be seen that as long as the nested tests do not modify the database, all nested tests can access the migrated and filled-with-data (via @Sql(scripts=...)) database as expected. But once one test is added that modifies the data, all tests after it operate only the migrated version of the database - until the test execution leaves the nested class, where refreshing resumes before each test as intended. It later turned out that the database only contains data when another test of the Nest-Hosting subclass is executed before the nested ones - if not, the nested classes receive only the migrated-but-not-filled database.

In our tests we passed references to the ApplicationContext between the nesting-levels instead of using spring-injection, which could be considered bypassing spring and as such might be an important detail in this issue.

The only workaround I could discover was making the nested classes (A.N) in the abstract host-class abstract as well and then explicitly subclass them in the subclass again (S extends A, A.N is nested in A, S.N extends A.N, S.N is nested in S), which however defeats the purpose of inheriting the nested classes. Adding @Sql annotations to the nested classes in the abstract class appear to be executed against the unmigrated database (our schema cannot be found and its tables do not exist in the default schema either), even when @AutoConfigureEmbeddedDatabase and @FlywayTest are added as well.

Sadly I have had a lot of trouble reducing this problem out of our project-code, as such the reproducer I wound up with is still a bit long:

public class NestedTestBaseExample {

    @SpringBootTest
    @FlywayTest
    @Sql(statements = {"INSERT INTO app.demo_table (id) VALUES (1),(2),(3);"})
    @AutoConfigureEmbeddedDatabase(refresh = AutoConfigureEmbeddedDatabase.RefreshMode.BEFORE_EACH_TEST_METHOD)
    static class TestContainerHost {
        private final DataSource hostSource;

        public TestContainerHost(@Autowired DataSource hostSource) {
            this.hostSource = hostSource;
        }

        @Nested
        class TestContainerImpl extends TestContainerBase {
            public TestContainerImpl() {
                super(hostSource);
            }

            @Test
            public void OutsideTest() throws SQLException{
                SubTest.assertSelectResults(
                        hostSource,
                        1,
                        2,
                        3
                );
            }
        }
    }

    static abstract class TestContainerBase {
        private final DataSource dataSource;

        public TestContainerBase(DataSource dataSource) {
            this.dataSource = dataSource;
        }

        @Nested
        @AutoConfigureEmbeddedDatabase(refresh = AutoConfigureEmbeddedDatabase.RefreshMode.BEFORE_EACH_TEST_METHOD)
        @TestMethodOrder(MethodOrderer.OrderAnnotation.class)
        class SubTest {

            public static void assertSelectResults(DataSource dataSource, int... expected) throws SQLException {
                IntStream.Builder builder = IntStream.builder();
                try (Statement statement = dataSource.getConnection().createStatement()) {
                    Assertions.assertTrue(statement.execute("SELECT (id) FROM app.demo_table;"), "SELECT reported no results");
                    ResultSet resultSet = statement.getResultSet();
                    SQLWarning warnings = resultSet.getWarnings();
                    if (warnings != null) {
                        Assertions.fail(warnings);
                    }
                    while (resultSet.next()) {
                        builder.add(resultSet.getInt(1));
                    }
                }
                int[] results = builder.build().toArray();
                Assertions.assertArrayEquals(expected, results, "Got unexpected results: " + Arrays.toString(results));
            }

            @Test
            @Order(-5)
            public void selectElementsBeforeAllInsertTests() throws SQLException {
                assertSelectResults(dataSource, 1, 2, 3);
            }

            @RepeatedTest(4)
            @Order(0)
            public void insertElementTest(RepetitionInfo info) throws SQLException {
                int insertValue = info.getCurrentRepetition() + 3;
                try (Statement insertStatement = dataSource.getConnection().createStatement()) {
                    Assertions.assertFalse(insertStatement.execute(
                                    "INSERT INTO app.demo_table VALUES (" + insertValue + ");"),
                            "INSERT reported results instead of update-count");
                    Assertions.assertEquals(1, insertStatement.getUpdateCount());
                }
                assertSelectResults(dataSource, 1, 2, 3, insertValue);
            }


            @Test
            @Order(5)
            public void selectResetElementsAfterAllInsertTests() throws SQLException {
                assertSelectResults(dataSource, 1, 2, 3);
            }
        }
    }
}

Which results in the following test-results:

Where repetition 2 retains the new element from repetition 1, repetition 3 those from rep. 2 and 1, rep. 4 of rep. 3, 2 and 1, and selectResetElementsAfterAllInsertTests operates on the data left by repetition 4.

I have created a gist where I added other files that could be relevant to this problem (build.gradle, application.properties etc.)

The behavior with "if no outside tests are executed before the nested tests" can be seen when deleting the NestedTestBaseExample.TestContainerHost.TestContainerImpl.OutsideTest() test - this results in not only all tests starting with the second iteration of insertElementTest failing, but those before that as well as the database contains no data.

There is one more 'gotcha' I would like to point out because it caused me a lot of headache while trying to reproduce this: It is important that the org.junit.jupiter.api.Test-Annotation is used because the org.junit.Test-Annotation appears to not support (inherited?) nested testing and as such will report that no tests could be found in this case.