jfrog
jf mvn doesn't ignore any failure that occurs during test execution
Describe the bug
JFrog CLI doesn't ignore the failed tests, which led the build became "unstable" and build failed.
Current behavior
When passing "-Dmaven.test.failure.ignore=true" in jf mvn deploy command, some test fail and user is getting these errors:
2024-06-25T21:18:57.7016614Z [ThreadedStreamConsumer] ERROR org.apache.maven.plugin.surefire.SurefirePlugin - Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.763 s <<< FAILURE! - in com.roche.upath.schedulers.unmark.service.impl.CaseUnmarkServiceImplTest
2024-06-25T21:18:57.7027306Z [ThreadedStreamConsumer] ERROR org.apache.maven.plugin.surefire.SurefirePlugin - testUnmarkBookmarkedCase Time elapsed: 1.715 s <<< FAILURE!
This makes the build becomes “unstable”
[main] WARNING org.jfrog.build.extractor.maven.BuildDeploymentHelper - Artifactory Build Info Recorder: unstable build, artifacts will not be deployed...
and leads to
{
"status": "failure",
"totals": {
"success": 0,
"failure": 19
},
"files": []
}
It looked like JFrog CLI would search for the failed tests under target/surefire-reports directory. If there are failed tests, the jars are not published.
Reproduction steps
I cloned the repo https://github.com/jfrog/project-examples/tree/master/maven-examples/maven-example and modified this line in /multi1/src/test/java/artifactory/test/AppTest.java
public void testApp()
{
assertTrue( false );
}
Add to pom.xml:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.9</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.8</version>
<scope>test</scope>
</dependency>
</dependencies>
.
.
.
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.2.2</version>
<configuration>
<includes>
<include>AppTest.java</include>
</includes>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
Then execute
jf mvn deploy -Dsurefire.skipAfterFailureCount=1 --build-name=test-mvn-deploy --build-number=2.0701 -Dmaven.test.failure.ignore=true
the test is failing
[main] ERROR org.apache.maven.plugin.surefire.SurefirePlugin - Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
.
.
.
{
"status": "failure",
"totals": {
"success": 0,
"failure": 9
}
}
The only way to overcome is setting -Dartifactory.publish.unstable=true .
Expected behavior
It should ignore all failed tests and complete the build without any failures.
JFrog CLI version
2.52.10.
Operating system type and version
macOS - 14.4.1 - 23E224
JFrog Artifactory version
No response
JFrog Xray version
No response
Any progress on this?
It's forcing CI changes across all of our repos that are using jf mvn.
This is a big down-side to using jFrog tool wrappers.
It would be a different story if the confirmed bugs were addressed in a timely fashion.
