junit-plugin
junit-plugin copied to clipboard
jenkinsci
Test trend not generated
As i understand test trend is not generated until there is at least one 100% test pass result? Is this really expected behavior?
If yes, it would be nice if trend chart showed up instantly after first pipeline build even if some tests are failing ...
The trend doesn't show until your second build I believe, nothing to do with pass rate
The trend doesn't show until your second build I believe, nothing to do with pass rate @timja
Well then this is clearly a bug. I'm experiencing this as same as described in these threads:
https://stackoverflow.com/questions/36767569/jenkins-not-showing-test-result-trend-graph-for-some-projects https://issues.jenkins.io/browse/JENKINS-40103
I have multiple pipeline builds now but still not seeing any trend generated. Those pipelines with successful builds are showing trend correctly ...
Currently i have Jenkins version: 2.263.4 Junit plugin version: 1.48
@timja any ideas how to proceed with this? maybe some workaround here or possible that issues could be from my side and more info needed?
I don't have time to look into it right now, I'm not sure why that may be happening.
@timja Nothing unusual from Jenkins logs when Junit report gets generated. I am executing junit step with following options:
junit allowEmptyResults: true, healthScaleFactor: 0.0, skipPublishingChecks: true, testResults: 'report.xml'
@timja
Looks like i found why this is happening.
Because of incorrect exit code handling for pytest execution. Exit code 1 is normal pytest exit code and should not cause shell step to throw exception in runtime.
If these return statuses are correctly handled then Job gets it's trend no matter of if some test are failing.
@timja Sorry for hassle and getting ahead of myself. After all I could not resolve this with just correct error handling.
I'm now running my pytests correctly by getting the status code from pytest execution command itself.
def ec = sh (script: 'pytest --junitxml=some/path /my/tests', returnStatus: true)
So no way there could be some unexpected sh exception.
But i still was not getting the trend to appear - only when ec = 0 and in this case i am setting currentBuild.result = 'SUCCESS'
So here is the deal I understood:
junit really does not care about if some tests are failing or not. Test trend is showing up only when I have my first SUCCESS or UNSTABLE build. All the time i was doing this:
if (ec > 0) currentBuild.result = 'FAILURE
So instantly when I changed up to this:
if (ec > 0) currentBuild.result = 'UNSTABLE
test trend showed up
I did some digging and found this issue: https://issues.jenkins.io/browse/JENKINS-28479 Maybe this should be revised. I really think that test trend should show up even if there are no non-FAILURE build statuses in job.
@timja I guess what we are missing here is this: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/tasks/SimpleBuildStep.java#L204