[SUREFIRE-2194] JUnit 5: Implementing tests in test class and nested class results in incorrect reporting

[jira-importer]

Jochen Kraushaar opened SUREFIRE-2194 and commented

Example Test Class

import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertTrue;

class ExampleTest {

    @Test
    void outerTest() {
        assertTrue(true);
    }

    @Nested
    class InnerExampleTest {
        @Test
        void innerTest() {
            assertTrue(true);
        }
    }
}

Expected Result

One TEST-ExampleTest.xml file containing the results of both tests or two files ({}TEST-ExampleTest.xml{} and {}TEST-ExampleTest$InnerExampleTest.xml{}) containing the results of their tests.

Actual Result

• TEST-ExampleTest.xml does not contain any test case.
• TEST-ExampleTest$InnerExampleTest.xml contains results for both test cases.

Workaround

Do not put any tests into the parent class if nested classes are used.

Analysis

org.apache.maven.surefire.junitplatform.RunListenerAdapter does not distinguish between the parent and the nested classes. Both are reported as new test sets.

org.apache.maven.plugin.surefire.report.TestSetRunListener does not support nested test sets, so the test set of the nested class overrides the test set of the parent class.

---

Affects: 3.1.2

[jira-importer]

Michael Osipov commented

Why do you consider this a a bug and why is this a problem?

[jira-importer]

Jochen Kraushaar commented

The results of ExampleTest#outerTest are written to TEST-ExampleTest$InnerExampleTest.xml, which is misleading as the test is not part of ExampleTest$InnerExampleTest.

In our project this is a big problem, as QA will not approve our software if the test results are put into the wrong result set.

[jira-importer]

Michael Osipov commented

I see your point, it should be consistent.

[uhafner]

I hoped that https://github.com/apache/maven-surefire/pull/828 will fix this problem here as well, but this is not the case. The XML file of the nested class still has the results of all tests and the XML of the parent class is empty. This causes warnings in tools that read test files as they expect at least one test in such a file. As a workaround it would make sense to not write the parent XML file when nothing is written there...

[olamy]

@uhafner please provide some use cases similar to what has been done here https://github.com/apache/maven-surefire/issues/2601https://github.com/apache/maven-surefire/issues/2601

#828 has been implemented according to #2601 this seems to be a reasonnabe implementation which seems to fix cucumber and archunit use cases. If you have any other use cases, please provide a sample project to reproduce, at least this will help us to understand your case.

[uhafner]

In https://github.com/apache/maven-surefire/issues/2601#issuecomment-3034504077 I already mentioned that I think that the expectation is wrong (with the example of #2682). Nobody commented on this comment though. Is this scenario not sufficient to understand the problem? Or what additional use case is required? For me (and some tools) it does not make sense to produce empty files without tests.

If I don't miss anything, there are 4 options to handle the problem (the order is my preferred priority):

1. Test results of the parent are written to parent.xml, and tests of the nested test to parent$nested.xml.
2. Test results of the parent and nested are written to parent.xml, and the file parent$nested.xml is not created.
3. Test results of the parent and nested are written to parent$nested.xml, and the file parent$nested.xml is not created.
4. Test results of the parent and nested are written to parent$nested.xml, and the file parent$nested.xml is created with no tests. This is the current behavior which seems to be broken...

What I don't understand (option 4): Why should the nested test case file contain all results? Wouldn't it make sense to map the results to the matching XML files? And if not: why does a test file XML exist for the parent class at all? So my expectation (and the expectation of the original reporter) is, that nested test cases use either option 1 or 2. Option 3 would be strange but better than the current implementation of option 4. This creates empty files, which are picked up by tools (Jenkins Coverage Plugin, GitLab CI Quality Monitor) that normally issue warnings when they find test files that contain no tests.