opendevstack
RM: Failures in test component (NOT failed tests) in QA/Prod do not generate DIL but make the build stop / fail
Describe the bug If a test component fails in production the pipeline fails and no DIL is generated. Reported with @clemensutschig
https://github.com/opendevstack/ods-jenkins-shared-library/blob/f2470541f2e9cd8d247f5924ce0f5246b2069644/src/org/ods/orchestration/util/MROPipelineUtil.groovy#L399 does not set failfast to false - which means if an error occurs it will be thrown immediately .. (ref: https://github.com/opendevstack/ods-jenkins-shared-library/blob/master/src/org/ods/orchestration/util/MROPipelineUtil.groovy#L127-L134)
To Reproduce Do a full deployment to WIP -> D -> QA -> PROD where a test component fails either in QA or PROD. For instance, if sonarqube within the test case fails in PROD, the pipeline fails and no DIL is generated.
Expected behavior The pipeline should create and upload documents such as DIL.
Screenshots If applicable, add screenshots to help explain your problem.
Affected version (please complete the following information):
- OpenShift: 3.9
- OpenDevStack 3.x, 4.x, master
Log Output (ensure to remove any confidential information like tokens, project names, etc.
Failing SQ in component
14:46:13.500 ERROR: Error during SonarQube Scanner execution
java.lang.IllegalStateException: No files nor directories matching 'build/classes'
at org.sonar.java.AbstractJavaClasspath.getFilesFromProperty(AbstractJavaClasspath.java:92)
at org.sonar.java.JavaClasspath.init(JavaClasspath.java:54)
at org.sonar.java.AbstractJavaClasspath.getElements(AbstractJavaClasspath.java:280)
....
[Pipeline] }
[Pipeline] // withSonarQubeEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] ech)[SonarQube Analysis-spock] **** ENDED stage 'SonarQube Analysis' for component 'spock' branch 'master' **** (took 19023 ms)
[Pipeline] }
[Pipeline] // wrap
[Pipeline] stag)
[Pipeline] { (odsPipeline error)
[Pipeline] ech)WARN: [spock] ***** Finished ODS Pipeline for spock (with error) ***** (took 96705 ms)
[Pipeline] ech)WARN: Error: hudson.AbortException: script returned exit code 1
[Pipeline] ech)DEBUG: [buildstatus-464fb7ca-FAILED] Setting Bitbucket build status to 'FAILED' on commit '464fb7cad4bc535b4e1a01f4fd6dda4aac60ca71' / 'https://..../job/edpst-cd/job/edpst-cd-releasemanager-master/40/'
[Pipeline] withCredential)Masking supported pattern matches of $USERNAME or $TOKEN
[Pipeline] )
[Pipeline] sh (Set bitbucket build status via API)+ curl --fail -sS --request POST --header 'Authorization: Bearer ****' --header 'Content-Type: application/json' --data '{"state":"FAILED","key":"464fb7ca","name":"464fb7ca","url":"https://..../job/edpst-cd/job/edpst-cd-releasemanager-master/40/"}' https://....s/1.0/commits/464fb7cad4bc535b4e1a01f4fd6dda4aac60ca71
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] ech)DEBUG: [buildstatus-464fb7ca-FAILED] (took 507 ms)
[Pipeline] }
[Pipeline] // stage
[Pipeline] ech)DEBUG: Collecting (applicable) testresults from location: 'build/test-results/test'
[Pipeline] sh (Moving test results to system location: build/test-results/test)+ mkdir -p build/test-results/test build/test-results/test
+ cp -rf 'build/test-results/test/*' build/test-results/test
+ true
cp: cannot stat 'build/test-results/test/*': No such file or directory
[Pipeline] sh (Counting test results in build/test-results/test)+ ls -la 'build/test-results/test/*.xml'
+ wc -l
ls: cannot access build/test-results/test/*.xml: No such file or directory
[Pipeline] ech)DEBUG: Found 0 test files in 'build/test-results/test'
[Pipeline] ech)DEBUG: No xUnit results for stashing
[Pipeline] ech)DEBUG: [spock] ODS Component Pipeline 'spock-40'
ODS Build Artifacts 'spock':
{
"builds": {
},
"deployments": {
},
"failedStage": "odsPipeline error",
"testResultsFolder": "build/test-results/test",
"testResults": "0"
} (took 98246 ms)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // load
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] ech)DEBUG: Collected ODS build artifacts for repo 'spock': [builds:[:], deployments:[:], testResultsFolder:build/test-results/test, testResults:0, CREATED_BY_BUILD:Beta/40]
[Pipeline] }
[Pipeline] // dir
[Pipeline] ech)WARN: Error: aborting due to previous errors in repo 'spock'.
[Pipeline] }
Failed in branch spock
[Pipeline] // parallel
[Pipeline] }
Failed in branch Test
[Pipeline] // parallel
[Pipeline] ech)
WARN: Error occured within the orchestration pipeline: Error: aborting due to previous errors in repo 'spock'.
[Pipeline] ech)
DEBUG: [jira-update-release-EDPST-132] (took 1858 ms)
[Pipeline] ech)
[Test] **** ENDED orchestration stage **** (took 105397 ms)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] ech)
[ods-mro-pipeline] **** ENDED orchestration pipeline **** (took 223616 ms)
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Error: aborting due to previous errors in repo 'spock'.
at org.ods.orchestration.util.MROPipelineUtil.executeODSComponent(MROPipelineUtil.groovy:122)
at ___cps.transform___(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.constructorCall(DefaultInvoker.java:25)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:97)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at jdk.internal.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:107)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at jdk.internal.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:89)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at jdk.internal.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Finished: FAILURE
Additional context n.a
@albertpuente The EDP only creates entries in the DIL for failing test cases in Q/P, not for random errors.
@metmajer - are you folks sure (and this was tested e2e? :D)
If you take a look into the spock quickstarter . if gradle fails: https://github.com/opendevstack/ods-quickstarters/blob/master/e2e-spock-geb/Jenkinsfile.template#L33 we use 'error' - and that will fail downstream because of the above . From a compliance pow: ... this should result in an installation discrepency :))
@clemensutschig discrepancies are created from Jira Bugs assigned to the currently deployed version. Jira Bugs are created by EDP automatically, but only for failing tests. In case of a random pipeline failure, docs will not be generated. Remember?
@metmajer I do remember - but a "generic" failure in a "test" component - should create a bug & discrepency ... Not super urgent - but for the backlog
We should create a bug in this case - and create a discrepency from it .. (to really fit into the flow)