With cucumber integration tests not counted in total test summary in surefire-plugin 3.5.3

[jamshidCo]

Affected version

3.5.3

Bug description

Hello, Using JDK:

• openjdk 21.0.5 2024-10-15 LTS
• Apache Maven 3.9.9
• maven surfire plugin

Issue

When running integration tests using cucumber-junit-platform-engine with maven-surefire-plugin version 3.5.3, the tests are executed and the logs are printed, but the final test summary says:

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

This behavior does not happen in version 3.5.2 — in that version, Cucumber integration tests are correctly counted in the final summary — even though Cucumber scenarios are technically not individual JUnit test methods.

But in version 3.5.3, this seems to have changed:

• Tests still run
• Output/logs appear
• But Tests run: 0 appears at the end ➤ That can mess with CI pipelines or give misleading reports.

Expected Behavior

The final summary should include the number of Cucumber scenarios executed (like in 3.5.2).

[pzygielo]

• SUREFIRE-2299
• https://github.com/apache/maven-surefire/pull/828#discussion_r2030755705

[juergencodes]

I experienced a similar problem: My maven build was successful though I had a test failure. Since it is in a setup of a custom framework based on JBehave it is interesting to see that others experience similar problems in cucumber (this issue) or e.g. Arch (following the links in SUREFIRE-2229).

I was drilling down a bit deeper and found out the following reproducer that just uses DynamicTest (which probably all the above mentioned frameworks use) and plays with providing test source.

    @TestFactory
    DynamicTest withSource() {
        return test("withSource", URI.create("pom.xml"));
    }

    @TestFactory
    DynamicTest noSource() {
        return test("noSource", null);
    }

    DynamicTest test(final String test, final URI source) {
        return DynamicTest.dynamicTest(test, source, () -> {
            System.out.println("Running '" + test + "'");
            org.junit.jupiter.api.Assertions.assertNull(source);
        });
    }

Running the test (also see attachment for "full" project) with surefire 3.5.2 fails as expected with

[INFO] Running SurefireReproducerTest
Running 'withSource'
Running 'noSource'
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.049 s <<< FAILURE! -- in SurefireReproducerTest
[ERROR] withSource().withSource -- Time elapsed: 0.012 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <null> but was: <pom.xml>
        at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        at org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:50)
        at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:35)
        at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:30)
        at org.junit.jupiter.api.Assertions.assertNull(Assertions.java:279)
        at SurefireReproducerTest.lambda$test$0(SurefireReproducerTest.java:20)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   expected: <null> but was: <pom.xml>
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0

Switching to 3.5.3 makes

[INFO] Running SurefireReproducerTest
Running 'withSource'
Running 'noSource'
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049 s -- in SurefireReproducerTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Mind that both tests are executed (see "Running..." output), but

• No error is reported
• The build does not fail
• The test count dropped from 2 to 1

(just in case it matters, i used Apache Maven 3.8.6)

Full reproducer, just make sure to bump 3.5.2 to 3.5.3 surefire-reproducer.zip

[juergencodes]

I forgot to mention that using a source or not isn't just something I was doing for fun. If you change the assertNull to assertNotNull, everythings works fine even with surefire 3.5.3

[INFO] Running SurefireReproducerTest
Running 'withSource'
Running 'noSource'
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.048 s <<< FAILURE! -- in SurefireReproducerTest
[ERROR] SurefireReproducerTest.noSource()[1] -- Time elapsed: 0.004 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: not <null>
        at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
        at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        at org.junit.jupiter.api.AssertNotNull.failNull(AssertNotNull.java:49)
        at org.junit.jupiter.api.AssertNotNull.assertNotNull(AssertNotNull.java:35)
        at org.junit.jupiter.api.AssertNotNull.assertNotNull(AssertNotNull.java:30)
        at org.junit.jupiter.api.Assertions.assertNotNull(Assertions.java:304)
        at SurefireReproducerTest.lambda$test$0(SurefireReproducerTest.java:20)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   SurefireReproducerTest.lambda$test$0:20 expected: not <null>
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

I.e. the problem ONLY happens if one provides a source.

[bramklg]

I'm also running into this problem where unit test reports are still properly reported (and failures result in a build failure) but the (cucumber) integration test results are not recorded properly any more, it always says 0 tests executed.

I ran into this after upgrading spring boot from 3.4.4 to 3.4.5 which includes an upgrade of the surefire plugin from 3.5.2 to 3.5.3.

[Yith1]

We have had this problem. But have also noticed that ArchUnit tests are affected as well.

[francoissey]

I confirm that if you downgrade to 3.5.2, it works fine. It is really 3.5.3 specific.

[ascopes]

Can also confirm 3.5.2 does not encounter the issue.

Is there any chance of prioritising this?

~This might be caused by #816~ edit: seems forcing the changed settings does not change this, so likely unrelated.

[claudemartin]

How is this still not fixed? I just had to rework weeks of PRs on a project because all our failing ArchUnit tests were not reported.

[ondrejfafejta-ext90099]

I've created a minimal project that demonstrates the bug: https://github.com/fafejtao/cucumberSurefireBug

[rturner-edjuster]

@olamy I'm just trying to understand the current status related to this issue and pull request https://github.com/apache/maven-surefire/pull/828. It seems you've done some work, and reverted some things (maybe), but it seems like work might be paused at the moment. Is that the correct analysis?

Is this something that help could be used on? I can try to assist (as this issue, and the related 3.5.2 issue about parallel output being mixed up, have been driving me nuts this week).

[ascopes]

Would it be worth for now reverting the corresponding change that introduced the regression and then a fix being produced concurrently?

Suggest this since I have seen this issue impact half a dozen different groups of people in the past few weeks, it seems to be causing a fair amount of noise. Especially since the issue has been active and causing build issues for over 4 months now.

[rturner-edjuster]

Would it be worth for now reverting the corresponding change that introduced the regression and then a fix being produced concurrently? @ascopes I'm not sure it's that simple as I believe the change that triggered this fault also fixes a pretty important data corruption issue for parallel JUnit 5 output. @olamy would be best to comment when they can as they are likely the most familiar with both issues.

Also, since this is OSS, "we" the community need to do the work and makes the required changes. I just don't want to start a new PR if Olivier is "almost there" with a solution, nor do I want to step on anyones toes in the process. (Not that I have gone that far yet digging into the problem or a structure for a solution...)

[olamy]

Would it be worth for now reverting the corresponding change that introduced the regression and then a fix being produced concurrently? @ascopes I'm not sure it's that simple as I believe the change that triggered this fault also fixes a pretty important data corruption issue for parallel JUnit 5 output. @olamy would be best to comment when they can as they are likely the most familiar with both issues.

Also, since this is OSS, "we" the community need to do the work and make the required changes. I just don't want to start a new PR if Olivier is "almost there" with a solution, nor do I want to step on anyone's toes in the process. (Not that I have gone that far yet, digging into the problem or a structure for a solution.)

long story short.

The change has been introduced to fix an issue for users who have enabled parallel test running (junit.jupiter.execution.parallel.enabled=true), as test result and stacktrace were all mixed up, reverting it means reintroducing the issue for those users. The issue here is with the provider junit-platform (code running within the forked jvm) need to communicate in a thread safe way to a listener (which is global so common code to all providers from junit 3 to junit 5 via junit 4.x and testng) and this code is probably more than 15 years old. This main listener is fairly complicated to adapt this to the changes needed in junit-platform multi thread and doing the changes obviously broke a lot of providers (junit3/4 and testng). (not sure we really need to keep supporting 3.x though) Fundamentally, this would need a massive refactoring and A LOT OF code simplification. But that's another topic (I only need to express my complaints and frustrations somewhere :) )

My first idea is to enable the change only for users who need this parallel working. But I wanted to have a better solution and tried something, couldn't find a quick easy change and then being distracted into other stuff :)

so "almost-there", I plan to spend more time on this one the next few days :)

[rturner-edjuster]

@olamy Thanks for the update. Please let me know if there is any assistance I can provide. I appreciate your work on this project (and others).

[olamy]

I've created a minimal project that demonstrates the bug: https://github.com/fafejtao/cucumberSurefireBug

@ondrejfafejta-ext90099 thanks a lot I have added your test as part of #828 PR looking good now