maven-surefire icon indicating copy to clipboard operation
maven-surefire copied to clipboard
verified publisher icon apache

surefire version 3.5.4 broke maven-surefire-junit5-tree-reporter

Open fabriciorby opened this issue 2 months ago • 6 comments

Affected version

3.5.4

Bug description

Hello,

I've developed this plugin extension some years ago and it's kind of totally dependant on the test "source" field from the ReportEntry to have the full nested classes paths

From version 3.5.4 on, there's no full path received by the ConsoleReporter.

Image

From version 3.5.3, I got it filled like this

Image

I suspect that it has to do with the new bug fix described in the Release Notes: [SUREFIRE-2298] - fix xml output with junit 5 nested classes (fix integration with Cucumber and Archunit) (https://github.com/apache/maven-surefire/pull/828) @olamy

This is absolutely necessary for the plugin to work, I took a quicklook in the MR that is linked in the bug fix but I couldn't relate it to the source field. If someone can point me towards where this info is being retrieved/consumed and if this can behavior can be rollbacked it would be helpful so I can open a PR myself.

Link to the plugin: https://github.com/fabriciorby/maven-surefire-junit5-tree-reporter

fabriciorby avatar Oct 11 '25 11:10 fabriciorby

It seems there are some broken tests that were not detected by the CI/CD pipeline btw

Image Image

When running for 3.5.3 it's all green

pipeline logs:

[INFO] --- surefire:3.5.2:test (default-test) @ surefire-junit-platform ---
[INFO] Surefire report directory: /home/runner/work/maven-surefire/maven-surefire/surefire-providers/surefire-junit-platform/target/surefire-reports
[INFO] Using configured provider org.apache.maven.shadefire.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

fabriciorby avatar Oct 13 '25 21:10 fabriciorby

please provide logs those screenshots are not usable.

olamy avatar Oct 13 '25 22:10 olamy

sure, now I know better (and have the project properly set up)

This is about the testing part, not sure why it does not work, I needed to screenshot because I managed to make it run on IntelliJ

╭╴󰀵 // …/maven-surefire/surefire-providers/surefire-junit-platform //  [master] 
mvn clean test

[INFO] Scanning for projects...
[INFO] 
[INFO] ---------< org.apache.maven.surefire:surefire-junit-platform >----------
[INFO] Building Surefire JUnit Platform Runner 3.5.5-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- clean:3.4.1:clean (default-clean) @ surefire-junit-platform ---
[INFO] Deleting /Users/fabriciorby/Desktop/projects/maven-surefire/surefire-providers/surefire-junit-platform/target
[INFO] 
[INFO] --- enforcer:3.5.0:enforce (enforce-bytecode-version) @ surefire-junit-platform ---
[INFO] Rule 0: org.codehaus.mojo.extraenforcer.dependencies.EnforceBytecodeVersion passed
[INFO] 
[INFO] --- enforcer:3.5.0:enforce (enforce-maven-version) @ surefire-junit-platform ---
[INFO] Rule 0: org.apache.maven.enforcer.rules.version.RequireMavenVersion passed
[INFO] 
[INFO] --- enforcer:3.5.0:enforce (enforce-java-version) @ surefire-junit-platform ---
[INFO] Rule 0: org.apache.maven.enforcer.rules.version.RequireJavaVersion passed
[INFO] 
[INFO] --- jacoco:0.8.13:prepare-agent (jacoco-agent) @ surefire-junit-platform ---
[INFO] jacoco.agent set to -javaagent:/Users/fabriciorby/.m2/repository/org/jacoco/org.jacoco.agent/0.8.13/org.jacoco.agent-0.8.13-runtime.jar=destfile=/Users/fabriciorby/Desktop/projects/maven-surefire/surefire-providers/surefire-junit-platform/target/jacoco.exec,includes=**/failsafe/*:**/failsafe/**/*:**/surefire/*:**/surefire/**/*,excludes=**/HelpMojo.class:**/shadefire/**/*:org/jacoco/**/*:com/vladium/emma/rt/*
[INFO] 
[INFO] --- checkstyle:3.6.0:check (checkstyle-check) @ surefire-junit-platform ---
[INFO] You have 0 Checkstyle violations.
[INFO] 
[INFO] --- spotless:2.44.3:apply (default) @ surefire-junit-platform ---
[INFO] Index file does not exist. Fallback to an empty index
[INFO] Spotless.Java is keeping 13 files clean - 0 were changed to be clean, 13 were already clean, 0 were skipped because caching determined they were already clean
[INFO] Spotless.Pom is keeping 1 files clean - 0 were changed to be clean, 1 were already clean, 0 were skipped because caching determined they were already clean
[INFO] 
[INFO] --- remote-resources:3.3.0:process (process-resource-bundles) @ surefire-junit-platform ---
[INFO] Preparing remote bundle org.apache.apache.resources:apache-jar-resource-bundle:1.7
[INFO] Copying 3 resources from 1 bundle.
[INFO] 
[INFO] --- resources:3.3.1:resources (default-resources) @ surefire-junit-platform ---
[INFO] Copying 1 resource from src/main/resources to target/classes
[INFO] Copying 3 resources from target/maven-shared-archive-resources to target/classes
[INFO] 
[INFO] --- apache-rat:0.16.1:check (rat-check) @ surefire-junit-platform ---
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 14 licenses.
[INFO] 
[INFO] --- compiler:3.14.0:compile (default-compile) @ surefire-junit-platform ---
[INFO] Recompiling the module because of changed source code.
[INFO] Compiling 8 source files with javac [forked debug deprecation release 8] to target/classes
[INFO] 
[INFO] --- resources:3.3.1:testResources (default-testResources) @ surefire-junit-platform ---
[INFO] skip non existing resourceDirectory /Users/fabriciorby/Desktop/projects/maven-surefire/surefire-providers/surefire-junit-platform/src/test/resources
[INFO] Copying 3 resources from target/maven-shared-archive-resources to target/test-classes
[INFO] 
[INFO] --- compiler:3.14.0:testCompile (default-testCompile) @ surefire-junit-platform ---
[INFO] Recompiling the module because of changed dependency.
[INFO] Compiling 5 source files with javac [forked debug deprecation release 8] to target/test-classes
[INFO] 
[INFO] --- surefire:3.5.2:test (default-test) @ surefire-junit-platform ---
[INFO] Using configured provider org.apache.maven.shadefire.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.865 s
[INFO] Finished at: 2025-10-14T00:57:59+02:00
[INFO] ------------------------------------------------------------------------

But about my main issue, is that there's no inner class path in the source of nested tests; this explicitly breaks one of the tests, which is this one (log from IntelliJ execution)

org.opentest4j.AssertionFailedError: expected: <org.apache.maven.surefire.junitplatform.JUnitPlatformProviderTest$NestingTest$Level1NestedTest> but was: <org.apache.maven.surefire.junitplatform.JUnitPlatformProviderTest$NestingTest>
Expected :org.apache.maven.surefire.junitplatform.JUnitPlatformProviderTest$NestingTest$Level1NestedTest
Actual   :org.apache.maven.surefire.junitplatform.JUnitPlatformProviderTest$NestingTest
<Click to see difference>


	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProviderTest.runNestingTest(JUnitPlatformProviderTest.java:442)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

test path: org.apache.maven.surefire.junitplatform.JUnitPlatformProviderTest.runNestingTest(JUnitPlatformProviderTest.java:442)

fabriciorby avatar Oct 13 '25 23:10 fabriciorby

and to be more precise, it happens here: https://github.com/apache/maven-surefire/blob/2b16f5dae6d79982e6c4fd53e2240230f2865d5b/surefire-providers/surefire-junit-platform/src/main/java/org/apache/maven/surefire/junitplatform/RunListenerAdapter.java#L225

It will always get the class name without the inner class

fabriciorby avatar Oct 13 '25 23:10 fabriciorby

correct this change is on purpose to fix issue with parallel test with junit5. the choice has been made to use the source class name only. Ideally, adding another field with this value could help. This will be easier once the big refactoring is finished from https://github.com/apache/maven-surefire/pull/3179 currently you only need to setup pluginManagement section to stick with previous version of surefire before the change

olamy avatar Oct 13 '25 23:10 olamy

that's indeed a big refactoring! if you could add this extra field in the SimpleReportEntry it would be very helpful.

meanwhile I'll keep recommending people to use surefire version 3.5.3, thank you for your reply

fabriciorby avatar Oct 13 '25 23:10 fabriciorby