gitlab-plugin
gitlab-plugin copied to clipboard
jenkinsci
Upgrade from 1.5.13 to 1.5.19 when updating a merge request now returning 500 error code back to Gitlab.
We are using Jenkins 2.282 running the Gitlab 1.5.13 plugin successfully with the latest version of Gitlab [13.9.3] for at least a year now. A recent Gitlab 1.5.19 plugin was available [seems like it was a jump from 1.5.13 -> 1.5.19 that became available] in Jenkins so we upgraded to the latest version and now our regular Webhooks requests that are triggered during a merge request update are now receiving a 500 error back from Jenkins.
When tailing the Jenkins logs when Gitlab sends the merge request webook to Jenkins, I see the following error in the logs.
2021-03-08 19:47:36.692+0000 [id=300] WARNING o.e.j.s.h.ContextHandler$Context#log: Error while serving https://jenkins.myjenkinsite.com/project/Gitlab_Merge_Request_Check
java.lang.IllegalStateException: No credentials found for credentialsId:
at com.dabsquared.gitlabjenkins.connection.GitLabConnection.getApiToken(GitLabConnection.java:155)
at com.dabsquared.gitlabjenkins.connection.GitLabConnection.getClient(GitLabConnection.java:131)
at com.dabsquared.gitlabjenkins.connection.GitLabConnectionConfig.getClient(GitLabConnectionConfig.java:66)
at com.dabsquared.gitlabjenkins.connection.GitLabConnectionProperty.getClient(GitLabConnectionProperty.java:74)
at com.dabsquared.gitlabjenkins.trigger.handler.AbstractWebHookTriggerHandler.setCommitStatusPendingIfNecessary(AbstractWebHookTriggerHandler.java:67)
at com.dabsquared.gitlabjenkins.trigger.handler.AbstractWebHookTriggerHandler.handle(AbstractWebHookTriggerHandler.java:53)
at com.dabsquared.gitlabjenkins.trigger.handler.merge.MergeRequestHookTriggerHandlerImpl.handle(MergeRequestHookTriggerHandlerImpl.java:83)
at com.dabsquared.gitlabjenkins.trigger.handler.merge.MergeRequestHookTriggerHandlerImpl.handle(MergeRequestHookTriggerHandlerImpl.java:41)
at com.dabsquared.gitlabjenkins.GitLabPushTrigger.onPost(GitLabPushTrigger.java:492)
at com.dabsquared.gitlabjenkins.webhook.build.MergeRequestBuildAction$1.performOnPost(MergeRequestBuildAction.java:78)
at com.dabsquared.gitlabjenkins.webhook.build.BuildWebHookAction$TriggerNotifier.run(BuildWebHookAction.java:54)
at hudson.security.ACL.impersonate2(ACL.java:423)
at hudson.security.ACL.impersonate(ACL.java:435)
at com.dabsquared.gitlabjenkins.webhook.build.MergeRequestBuildAction.execute(MergeRequestBuildAction.java:75)
at com.dabsquared.gitlabjenkins.webhook.build.BuildWebHookAction.execute(BuildWebHookAction.java:31)
at com.dabsquared.gitlabjenkins.webhook.GitLabWebHook.getDynamic(GitLabWebHook.java:44)
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
Caused: java.lang.reflect.InvocationTargetException
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:400)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:455)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.dabsquared.gitlabjenkins.webhook.GitLabWebHook$GitlabWebHookCrumbExclusion.process(GitLabWebHook.java:53)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:127)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:133)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:834)
However, When I downgrade the Gitlab plugin back to 1.5.13, it all works as normal.
Hmm, It seems a few merge requests triggers returned a proper 200 code back but most of the merge request triggers returned back a 500 error code.
@breisig Could you please check if are credential ID specified on gitlab connection or not? According to error it look like you have connection with GitLab URL and without Credentials specified
same here too. Exactly the same message reported on first post.
Edit: 10 minutes before I wrote the comment above I revert to version 1.5.13 and all back to normal, so the problem is on vesion 1.5.19 for sure.
It seems that it may be caused by changes made in PR #916, which implemented the ability to use folder-level GitLab Token API credentials for GitLab connection. This PR was added in 1.5.16, so at the moment this and higher versions are affected (up to 1.5.20).
Tried the 1.5.21-SNAPSHOT version and same issue. https://repo.jenkins-ci.org/ui/repos/tree/General/snapshots%2Forg%2Fjenkins-ci%2Fplugins%2Fgitlab-plugin%2F1.5.21-SNAPSHOT%2Fgitlab-plugin-1.5.21-20210330.191424-1.hpi
@monstrfolk This artifact was created on March 30, so it doesn't contain the latest changes, including those that may fix the issue. Guess we have to wait for the release version, or you can try to build the newest SNAPSHOT artifact from sources.
The plugin was broken for a month now, can we have a release with the fix? If it's fixed on master why the issue is not closed? Don't want to complain but it's very confusing actually.
I think it is related to the credential plugin ? https://github.com/jenkinsci/gitlab-plugin/issues/411#issuecomment-879817802 When I trigger with the previous configuration (no credential error when trying to send build status) I get a 200. When I trigger with the credential set I immediatelly get a 500, which might strongly indicate a bug. I'll look in the logs
I can confirm it is related to the XML configuration of the credential plugin (moreover stacktrace says so). Here is how I fixed it :
- On your job configuration, check use alternative credential, select any credential (even the default one) and save
If I was using the default dropdown, here was the configuration saved (vs previous working configuration)
<com.dabsquared.gitlabjenkins.connection.GitLabConnectionProperty plugin="[email protected]">
- <gitLabConnection></gitLabConnection>
+ <gitLabConnection>Gitlab-CI</gitLabConnection>
<jobCredentialId></jobCredentialId>
<useAlternativeCredential>false</useAlternativeCredential>
</com.dabsquared.gitlabjenkins.connection.GitLabConnectionProperty>
With the force alternative credential, it gets the correct job credential id set.
--- Version 6
+++ Version 8
@@ -91,13 +91,13 @@
<trim>false</trim>
</hudson.model.StringParameterDefinition>
</parameterDefinitions>
</hudson.model.ParametersDefinitionProperty>
<com.dabsquared.gitlabjenkins.connection.GitLabConnectionProperty plugin="[email protected]">
- <gitLabConnection></gitLabConnection>
- <jobCredentialId></jobCredentialId>
- <useAlternativeCredential>false</useAlternativeCredential>
+ <gitLabConnection>Gitlab-CI</gitLabConnection>
+ <jobCredentialId>gitlab-builder-api-key</jobCredentialId>
+ <useAlternativeCredential>true</useAlternativeCredential>
</com.dabsquared.gitlabjenkins.connection.GitLabConnectionProperty>
<com.sonyericsson.rebuild.RebuildSettings plugin="[email protected]">
<autoRebuild>false</autoRebuild>
<rebuildDisabled>false</rebuildDisabled>
</com.sonyericsson.rebuild.RebuildSettings>
This is not a viable workaround when you have more than 200 jobs to change one by one. I still need a final solution.
This is not a viable workaround when you have more than 200 jobs to change one by one. I still need a final solution.
Yeah I here you. As another workaround you could use a script to modify your 200 jobs xml directly. I can't do it myself with the way inheritance plugin handles XML files but I have fewer jobs
@builder-main tested workaround and confirm that it resolves issue
Yeah ok, though it looks like a tiny bug that should be solved IMO, having to remember this workaround each time you create a job (or as previous person stated when you have a lot) is a no go.