generic-webhook-trigger-plugin
generic-webhook-trigger-plugin copied to clipboard
jenkinsci
Large webhook, Argument list too long (MAX_ARG_STRLEN)
Hi there, I've got a problem with large webhook from Jira. Is it possible to save it to file?
Version report
Jenkins and plugins versions report: 1.76
Reproduction steps
- JIRA Server
- Send webhook(276kb) to Jenkins with Generic Webhook Trigger
Scripted pipeline
properties([
pipelineTriggers([
[$class: 'GenericTrigger',
genericVariables: [
[key: 'var_webhook', value: '$'],
],
tokenCredentialId: 'webhook-jira-production',
printContributedVariables: true,
printPostContent: true,
silentResponse: false
]
])
])
Results
Caused by: java.io.IOException: Cannot run program "git" (in directory "/storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira"): error=7, Argument list too long
Expected result: Build is OK and I can save webhook data to file for processing in steps
Actual result: Build failed.
22:09:55
22:09:55
22:09:55 Obtained Jenkins/pipelines/im_Webhook_Jira.groovy from git git@gitlab/myrepo.git
22:09:55 Resume disabled by user, switching to high-performance, low-durability mode.
22:09:55 [Pipeline] Start of Pipeline
22:09:55 [Pipeline] node
22:09:55 Running on Jenkins in /storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira
22:09:55 [Pipeline] {
22:09:55 [Pipeline] stage
22:09:55 [Pipeline] { (Init)
22:09:56 [Pipeline] echo
22:09:56 [info] checkout scm
22:09:56 [Pipeline] checkout
22:09:56 The recommended git tool is: git
22:09:56 using credential jenkins-ssh-keyfile
22:09:56 Cloning the remote Git repository
22:09:56 Cloning repository git@gitlab/myrepo.git
22:09:56 > git init /storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira # timeout=10
22:09:56 ERROR: Error cloning remote repo 'origin'
22:09:56 hudson.plugins.git.GitException: Could not init /storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1048)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:803)
22:09:56 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1227)
22:09:56 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
22:09:56 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
22:09:56 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
22:09:56 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
22:09:56 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
22:09:56 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
22:09:56 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
22:09:56 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
22:09:56 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
22:09:56 at java.base/java.lang.Thread.run(Thread.java:829)
22:09:56 Caused by: hudson.plugins.git.GitException: Error performing git command: git init /storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2689)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2611)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2607)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1974)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1046)
22:09:56 ... 12 more
22:09:56 Caused by: java.io.IOException: Cannot run program "git" (in directory "/storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/Webhook_Receiver_Jira"): error=7, Argument list too long
22:09:56 at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
22:09:56 at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
22:09:56 at hudson.Proc$LocalProc.<init>(Proc.java:253)
22:09:56 at hudson.Proc$LocalProc.<init>(Proc.java:222)
22:09:56 at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
22:09:56 at hudson.Launcher$ProcStarter.start(Launcher.java:508)
22:09:56 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2674)
22:09:56 ... 16 more
22:09:56 Caused by: java.io.IOException: error=7, Argument list too long
22:09:56 at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
22:09:56 at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
22:09:56 at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
22:09:56 at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
22:09:56 ... 22 more
Are you using some value from the webhook with the git plugin? What value? What does the json look like? Can you not have a more specific jsonpath?
You can save var_webhook to a file as part of your build. Just like you would save any variable to a file. You can google that.
Yes, I do. But build failed before that I can process it. I found same issues and thats is jenkins/node issue:
- https://issues.jenkins.io/browse/JENKINS-65958?page=com.atlassian.jira.plugin.system.issuetabpanels%3Achangehistory-tabpanel
- https://stackoverflow.com/questions/42168308/github-webhook-triggers-jenkins-job-with-too-large-json-payload-error
Looks unrelated to this plugin.
I would resolve the values I need with this plugin. And have a pipeline where I do git clone. I would not use the git plugin.
And I've remove any other plugins from job, make freejob with single build sh step. The issue reproduces
[JIRA_WEBHOOK_NO_GIT] $ /bin/sh -xe /tmp/jenkins17235265673168013619.sh
FATAL: command execution failed
java.io.IOException: error=7, Argument list too long
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
Caused: java.io.IOException: Cannot run program "/bin/sh" (in directory "/storage/webdata/JENKINS_HOME/workspace/WEBHOOKS/JIRA_WEBHOOK_NO_GIT"): error=7, Argument list too long
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
at hudson.Proc$LocalProc.<init>(Proc.java:253)
at hudson.Proc$LocalProc.<init>(Proc.java:222)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
at hudson.Launcher$ProcStarter.start(Launcher.java:508)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:144)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:21)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:808)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:164)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:516)
at hudson.model.Run.execute(Run.java:1889)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:100)
at hudson.model.Executor.run(Executor.java:433)
Build step 'Execute shell' marked build as failure
Finished: FAILURE
I do it with just single "echo" as stub(without access to genericVariable). Webhook's payload printed in the build output and than build failed on sh stage.
Any update on this? @facetheheat did you get a workaround?
@tomasbjerre I am getting the same error "java.io.IOException: error=7, Argument list too long", if the JSON payload exceeds the maximum size allowed for an environment variable.
Is there a way to access the HTTP request object and extract the payload? or convert the payload(post content) to file instead of setting it as ENV VAR?
I do it with just single "echo" as stub(without access to genericVariable). Webhook's payload printed in the build output and than build failed on sh stage.
same issue over here
@tomasbjerre It is very easy to reproduce just send a JSON payload over 2MB and try to access the variable, it will crash.
If I configure a pipeline like:
node {
properties([
pipelineTriggers([
[$class: 'GenericTrigger',
genericVariables: [
[
key: 'all',
value: '$'
]
],
causeString: 'Triggered on $ref',
tokenCredentialId: 'theid',
printContributedVariables: true,
printPostContent: true,
silentResponse: false
]
])
])
stage("build") {
sh '''
echo hello
'''
}
}
And trigger it with the given in a previous comment.
curl -v -H "Content-Type: application/json" http://localhost:8080/jenkins/generic-webhook-trigger/invoke?token=abc123 --data @/home/bjerre/Hämtningar/gvphwwdq.txt
I can see that the code reaches: https://github.com/jenkinsci/generic-webhook-trigger-plugin/blob/70b68f6de359b1add3f7b8264eb967a9d13f1043/src/main/java/org/jenkinsci/plugins/gwt/GenericTrigger.java#L181
And the job is also triggered, but fails with:
java.io.IOException: error=7, Argumentlistan för lång
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
at hudson.Proc$LocalProc.<init>(Proc.java:252)
at hudson.Proc$LocalProc.<init>(Proc.java:221)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
at hudson.Launcher$ProcStarter.start(Launcher.java:508)
at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:320)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
Caused: java.io.IOException: Cannot run program "nohup" (in directory "/home/bjerre/workspace/generic-webhook-trigger-plugin/work/workspace/pipe"): error=7, Argumentlistan för lång
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
at hudson.Proc$LocalProc.<init>(Proc.java:252)
at hudson.Proc$LocalProc.<init>(Proc.java:221)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
at hudson.Launcher$ProcStarter.start(Launcher.java:508)
at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234)
at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:320)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.run(WorkflowScript:25)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Finished: FAILURE
But if I change the pipeline so that it only resolves one variable, like this:
node {
properties([
pipelineTriggers([
[$class: 'GenericTrigger',
genericVariables: [
[
key: 'all',
value: '$.timestamp'
]
],
causeString: 'Triggered on $ref',
tokenCredentialId: 'theid',
printContributedVariables: true,
printPostContent: true,
silentResponse: false
]
])
])
stage("build") {
sh '''
echo hello
'''
}
}
The job succeeds.
I don't see this as a bug in the plugin, I see it as a bug in Jenkins. There seems to be an upper limit on number of parameters.
It is a common error, can be googled: https://www.google.com/search?q=%22error%3D7%2C+Argumentlist
Conclusion
Only solution, for now, is to only resolve the values you actually need. Use more specific JSONPath:s.