aws-codebuild-jenkins-plugin icon indicating copy to clipboard operation
aws-codebuild-jenkins-plugin copied to clipboard

Published 20 hours ago • verified publisher icon awslabs

ConcurrentModificationException in CodeBuildAction

Open vedarthk opened this issue 7 years ago • 20 comments
trafficstars

I am using:

Jenkins: 2.89.3 AWS Codebuild plugin: com.amazonaws:aws-codebuild:0.17 (maven ID) Pipeline plugin: org.jenkins-ci.plugins.workflow:workflow-aggregator:2.5 (maven ID)

And I have a multibranch pipeline with Jenkinsfile which looks something like this:

pipeline {
    agent { label 'master' }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        stage('Test Run') {
            failFast true
            parallel {
                stage('Test Run 1') {
                    steps {
                        timestamps {
                            withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'aws']]) {
                                awsCodeBuild credentialsId: 'CodeBuild', credentialsType: 'jenkins', projectName: 'Proj1', region: 'us-east-1', sourceControlType: 'project', sourceVersion: "${env.GIT_COMMIT}"
                            }
                        }
                    }
                }
                stage('Test Run 2') {
                    steps {
                        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'aws']]) {
                            awsCodeBuild credentialsId: 'CodeBuild', credentialsType: 'jenkins', projectName: 'Proj2', region: 'us-east-1', sourceControlType: 'project', sourceVersion: "${env.GIT_COMMIT}"
                        }
                    }
                }
                stage('Test Run 3') {
                    steps {
                        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'aws']]) {
                            awsCodeBuild credentialsId: 'CodeBuild', credentialsType: 'jenkins', projectName: 'Proj3', region: 'us-east-1', sourceControlType: 'project', sourceVersion: "${env.GIT_COMMIT}", envVariables: "[{VAR, value1}]"
                        }
                    }
                }
                stage('Test Run 4') {
                    steps {
                        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'aws']]) {
                            awsCodeBuild credentialsId: 'CodeBuild', credentialsType: 'jenkins', projectName: 'Proj3', region: 'us-east-1', sourceControlType: 'project', sourceVersion: "${env.GIT_COMMIT}", envVariables: "[{VAR, value2}]"
                        }
                    }
                }
                stage('Test Run 5') {
                    steps {
                        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'aws']]) {
                            awsCodeBuild credentialsId: 'CodeBuild', credentialsType: 'jenkins', projectName: 'Proj3', region: 'us-east-1', sourceControlType: 'project', sourceVersion: "${env.GIT_COMMIT}", envVariables: "[{VAR, value3}]"
                        }
                    }
                }
                stage('Test Run 6') {
                    steps {
                        withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'aws']]) {
                            awsCodeBuild credentialsId: 'CodeBuild', credentialsType: 'jenkins', projectName: 'Proj3', region: 'us-east-1', sourceControlType: 'project', sourceVersion: "${env.GIT_COMMIT}", envVariables: "[{VAR, value4}]"
                        }
                    }
                }
            }
        }
    }
}

in stages Test Run 3 to Test Run 6 I am using same Codebuild project but with different environment variable values which execute different scripts in the build When I am running the job with above pipeline I am getting following exception:

java.util.ConcurrentModificationException
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:907)
	at java.util.ArrayList$Itr.next(ArrayList.java:857)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
Caused: java.lang.RuntimeException: Failed to serialize CodeBuildAction#logs for class CodeBuildAction
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:181)
	at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1070)
	at hudson.BulkChange.commit(BulkChange.java:98)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1377)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:412)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

I am not familiar with Jenkins plugins, can anyone suggest any pointers, where can I look for more information, etc.

vedarthk avatar Feb 07 '18 05:02 vedarthk

Thanks for bringing this to our attention. We'll keep you posted as we investigate the root cause.

leoherran-aws avatar Feb 07 '18 16:02 leoherran-aws

I wasn't able to reproduce this issue--each one of these stages should be separated from one another, but the error message seems to indicate otherwise. Where in the build is this happening (after the CodeBuild build starts or before)? Does it still happen with the legacy parallel syntax?

node  {
    parallel (    
      'Test Run 1': { awsCodeBuild ... },
      'Test Run 2': { awsCodeBuild ... },
      'Test Run 3': { awsCodeBuild ... }
    )
}

leoherran-aws avatar Feb 22 '18 21:02 leoherran-aws

@leobaran-aws the occurrence is random, but most of the time it occurs in between ( in my case, each step takes around 10 minutes to complete) Also I have upgraded Jenkins from 2.89.3 to 2.108 (today) and the exception has stopped.

Thanks for looking into the issue.

vedarthk avatar Feb 24 '18 17:02 vedarthk

I'm seeing the same error and stacktrace. I'm using: AWS Codebuild 0.36 Jenkins 2.150.2 Pipeline: 2.5

And I'm using scripted pipeline code, something like

number.times { index ->
  jobs["Worker $index"] = {
            def build = awsCodeBuild(credentialsId: '...',
                  credentialsType: 'jenkins',
                  projectName: '...',
                  region: 'us-east-1',
                  envVariables: "...",
                  sourceControlType: 'project',
                  buildTimeoutOverride: "...")
  }
}
parallel jobs

It seems like when there are enough branches of these codebuild builds (seems like 30 right now), and the builds run long enough (> 30min), there's a very high chance of seeing ConcurrentModificationException. I've tried running 300 5-min jobs in parallel and they're fine, so I'm guessing it's something related to the time it takes and/or the amount of logs?

would appreciate any pointers, thank you in advance!

anna-yn avatar Feb 21 '19 22:02 anna-yn

Thanks for notifying us of this issue. Can you post a stack trace?

leoherran-aws avatar Feb 21 '19 22:02 leoherran-aws

yup here it is:

java.util.ConcurrentModificationException
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:907)
	at java.util.ArrayList$Itr.next(ArrayList.java:857)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:73)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
Caused: java.lang.RuntimeException: Failed to serialize CodeBuildAction#logs for class CodeBuildAction
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
Caused: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class org.jenkinsci.plugins.workflow.job.WorkflowRun
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.util.XStream2.toXMLUTF8(XStream2.java:313)
	at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:34)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.save(WorkflowRun.java:1278)
	at hudson.BulkChange.commit(BulkChange.java:98)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1446)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:417)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

anna-yn avatar Feb 21 '19 23:02 anna-yn

Thanks. I have a feeling it is related to an abundance of build logs when running multiple long-running builds. I'll look into where they're stored on the Jenkins instance.

leoherran-aws avatar Feb 21 '19 23:02 leoherran-aws

Thanks for the quick response! I took a look at the logs for these jobs, so for the ones that crash sometimes each build has ~400 .log files and the log file is ~100M, while the smaller jobs that can be run in 300 parallel branches produced ~1k .log files and the log file is ~10M. So maybe there was just too many logs?

Also according to the stack trace the exception went through here, could it be something about the size of the batch of log, or its content?

anna-yn avatar Feb 22 '19 00:02 anna-yn

Is it possible to have cloudwatch logs enabled, but not have the codebuild plugin copy it back to Jenkins? That way user can click into the codebuild dashboard to see the real time logs, and Jenkins wouldn't be overwhelmed

anna-yn avatar Feb 22 '19 07:02 anna-yn

It definitely does seem like it could be the Jenkins instance running out of space. Ideally the plugin would gracefully handle this exception, but I'll implement an option to disable CloudWatch logs streaming to the Jenkins instance.

leoherran-aws avatar Feb 22 '19 21:02 leoherran-aws

Implemented in fce052cbb8c9f9ea6f700685c80a2a13727144a1. This will be released in version 0.37 of the plugin, which should be available in a couple of hours. Give it a try and let us know if it unblocks your use case. Thanks again for the feedback!

leoherran-aws avatar Feb 22 '19 22:02 leoherran-aws

Oh wow that was quick! Thank you so much!! I'll try it out and let you know what happens!

anna-yn avatar Feb 22 '19 23:02 anna-yn

Tried out the new version, works perfectly! Seems like the error has stopped. Thank you again!

anna-yn avatar Feb 24 '19 01:02 anna-yn

What about the original crash/error? Will it also be fixed? We're trying to move a big pipeline to codebuild and we're having this problem...

rubenfonseca avatar Feb 27 '19 07:02 rubenfonseca

Yes, we are still trying to reproduce and root cause the issue. For the moment, CloudWatch streaming is not feasible for builds that generate a lot of logs. Can you disable CloudWatch streaming in your pipeline as a workaround?

leoherran-aws avatar Feb 27 '19 16:02 leoherran-aws

Thanks for the effort. Disabling streaming is not really an option since developers want to see the output of the build logs for debugging when the build goes red. And we're not planning on creating AWS accounts for them to check cloudwatch.

rubenfonseca avatar Feb 27 '19 16:02 rubenfonseca

@Nevvea7 @rubenfonseca What's the size (number of lines) of the CloudWatch logs for one of your builds? You should be able to find this in the CodeBuild dashboard (jenkins/job/project-name/build-number/codebuild-build-uuid). Also can you tell me the total size of the log file on the Jenkins instance for one of your pipeline builds?

leoherran-aws avatar Feb 27 '19 17:02 leoherran-aws

For me each build is about 3k lines, and the size of the log file is about 40M if I turn cloudwatch on and the parent build passes by chance. I think 10M might be a good size - I've tried kicking off 300 concurrent builds with one parent job, which produced a 10M log on jenkins, and they were fine

anna-yn avatar Feb 27 '19 17:02 anna-yn

Thanks! And do you use the CodeBuild dashboard to monitor individual builds or just use the Jenkins console output?

leoherran-aws avatar Feb 27 '19 17:02 leoherran-aws

For these builds I disabled the CloudWatch streaming and told my coworkers to go look on CodeBuild if they really want to see what's going on. @rubenfonseca maybe you can use the S3 logs option, so the logs get saved in S3, and if tests fail you just copy back the log from S3 and archive it for developers to see?

anna-yn avatar Feb 27 '19 22:02 anna-yn