jenkinsci
[JENKINS-27041] Thousands of threads created by thousands of GitHub pushes
A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:
"Waiting to acquire /…/workspace/… : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x… nid=0x… in Object.wait() [0x…]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
- locked (a hudson.slaves.WorkspaceList)
at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
- locked (a hudson.slaves.WorkspaceList)
at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Looking at the logs, there were 1150 GitHub push notifications for this job over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.
Such a high thread count had other knock on effects, such as high CPU usage when counting threads.
So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.
Originally reported by 
recampbell, imported from: Thousands of threads created by thousands of GitHub pushes
- assignee: recampbell
- status: Open
- priority: Major
- component(s): github-plugin
- label(s): performance
- resolution: Unresolved
- votes: 3
- watchers: 8
- imported: 2025-12-08
Raw content of original issue
A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:
"Waiting to acquire /…/workspace/… : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x… nid=0x… in Object.wait() [0x…] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245) - locked <0x…> (a hudson.slaves.WorkspaceList) at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224) - locked <0x…> (a hudson.slaves.WorkspaceList) at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446) at hudson.model.AbstractProject._poll(AbstractProject.java:1425) at hudson.model.AbstractProject.poll(AbstractProject.java:1336) at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81) at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)Looking at the logs, there were 1150 GitHub push notifications for this job over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.
Such a high thread count had other knock on effects, such as high CPU usage when counting threads.
So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.
- environment:
1.580.2
danielbeck:
- Original comment link
Raw content of original comment:
Isn't this covered by the global config option Max # of concurrent polling that appears once you have 10+ top level items?
Related: https://github.com/jenkinsci/jenkins/pull/1230#discussion_r12500721
Isn't this covered by the global config option Max # of concurrent polling that appears once you have 10+ top level items?
Related: https://github.com/jenkinsci/jenkins/pull/1230#discussion_r12500721
integer:
- Original comment link
Raw content of original comment:
Is it still actual? We have in TODO change polling algo that will be lighter.
Is it still actual? We have in TODO change polling algo that will be lighter.
jglick:
- Original comment link
Raw content of original comment:
Isn't this covered by the global config option Max # of concurrent polling
Probably not, since here polling is being called directly from the trigger, rather than simply asking for polling to be scheduled as SCMTrigger does. At least IIUC. And GitHubPushTrigger is still using MasterComputer.threadPoolForRemoting rather than Timer.get() which would bound the number of threads in use concurrently. It is also not using SequentialExecutionQueue correctly, because it fails to override Runnable.equals to reflect the identity of the Job.
Isn't this covered by the global config option Max # of concurrent polling
Probably not, since here polling is being called directly from the trigger, rather than simply asking for polling to be scheduled as SCMTrigger does. At least IIUC. And GitHubPushTrigger is still using MasterComputer.threadPoolForRemoting rather than Timer.get() which would bound the number of threads in use concurrently. It is also not using SequentialExecutionQueue correctly, because it fails to override Runnable.equals to reflect the identity of the Job.
integer:
- Original comment link
Raw content of original comment:
jglick queue shouldn't merge the same job with different actions. Either you will lose all builds for different commits in one job/repo.
jglick queue shouldn't merge the same job with different actions. Either you will lose all builds for different commits in one job/repo.
integer:
- Original comment link
Raw content of original comment:
It using SequentialExecutionQueue that is in Descriptor and wraps remoteThreadPool.
It using SequentialExecutionQueue that is in Descriptor and wraps remoteThreadPool.
integer:
- Original comment link
Raw content of original comment:
Btw, see stacktrace.
Btw, see stacktrace.
integer:
- Original comment link
Raw content of original comment:
But in the current state when GH trigger calls GIT polling, that has custom algorithm for detecting new commits (that may never end triggering builds), gh plugin triggering probably should be merged per job.
Imho let's better migrate to commitNotify, WDYT?
But in the current state when GH trigger calls GIT polling, that has custom algorithm for detecting new commits (that may never end triggering builds), gh plugin triggering probably should be merged per job.
Imho let's better migrate to commitNotify, WDYT?
jglick:
- Original comment link
Raw content of original comment:
You can coalesce pushBy users but only try to lock the workspace for polling once I guess. In other words, if you have received a thousand POSTs from three users altogether, but are still waiting in WorkspaceList.acquire, just let Git polling run once, and if changes are found, schedule one build with a CauseAction (scheduleBuild2) with multiple GitHubPushCause links. Or modify GitHubPushCause to support a set of users.
What I am unsure about is what happens if someone configures GitSCM with a wildcard branch spec (against my advice—use multibranch instead!); in that case there might have been commits to multiple branches, requires a corresponding number of builds to be scheduled. Now for /git/notifyCommit this is handled by JenkinsAbstractProjectListener passing RevisionParameterAction, which is a QueueAction so it forces multiple builds to arise. It seems that GitHubPushTrigger handles this case only insofar as every push results in an attempt to schedule a queue item, even when only one branch is configured, as found in this issue.
You can coalesce pushBy users but only try to lock the workspace for polling once I guess. In other words, if you have received a thousand POSTs from three users altogether, but are still waiting in WorkspaceList.acquire, just let Git polling run once, and if changes are found, schedule one build with a CauseAction (scheduleBuild2) with multiple GitHubPushCause links. Or modify GitHubPushCause to support a set of users.
What I am unsure about is what happens if someone configures GitSCM with a wildcard branch spec (against my advice—use multibranch instead!); in that case there might have been commits to multiple branches, requires a corresponding number of builds to be scheduled. Now for /git/notifyCommit this is handled by JenkinsAbstractProjectListener passing RevisionParameterAction, which is a QueueAction so it forces multiple builds to arise. It seems that GitHubPushTrigger handles this case only insofar as every push results in an attempt to schedule a queue item, even when only one branch is configured, as found in this issue.
integer:
- Original comment link
Raw content of original comment:
Atm github push plugin ONLY kicks scm polling. So if you configured wildcard it will go to git scm polling algo (that sucks JENKINS-30475 JENKINS-30345). The only thing that gh do -> kick existed poll. So in theory switch from GHpush to generic pollscm will do the same. I see 3 ways of triggering jobs: 1) kik poll() 2) put in queue with predefined actions and parameters (like github-pullrequest-plugin do atm) 3) send to branchNotify()
commitNotify() or whatever else listener exists in git plugin
Atm github push plugin ONLY kicks scm polling. So if you configured wildcard it will go to git scm polling algo (that sucks JENKINS-30475">JENKINS-30475 JENKINS-30345">JENKINS-30345). The only thing that gh do -> kick existed poll. So in theory switch from GHpush to generic pollscm will do the same.
I see 3 ways of triggering jobs:
1) kik poll()
2) put in queue with predefined actions and parameters (like github-pullrequest-plugin do atm)
3) send to branchNotify() commitNotify() or whatever else listener exists in git plugin
integer:
- Original comment link
Raw content of original comment:
If user configured wildcard, then GHpush trigger shouldn't bother with it's calculation and 2) is bad variant (but will work for other planned trigger types). 1) Kicking polling is not efficient, so 3) sounds like the right variant.
If user configured wildcard, then GHpush trigger shouldn't bother with it's calculation and 2) is bad variant (but will work for other planned trigger types).
1) Kicking polling is not efficient, so 3) sounds like the right variant.
lanwen:
- Original comment link
Raw content of original comment:
notifyCommit required enabled scm trigger. More than that - it will trigger all jobs with enabled scm trigger (even without GH trigger). And we can just remove trigger from plugin and handle triggering from root action
So it changes existing use case of this plugin and brakes all existed jobs with only GH push trigger. Also with notifyCommit you can disable unwanted triggering only with checkbox "Ignore post commit hooks".
notifyCommit required enabled scm trigger. More than that - it will trigger all jobs with enabled scm trigger (even without GH trigger). And we can just remove trigger from plugin and handle triggering from root action
So it changes existing use case of this plugin and brakes all existed jobs with only GH push trigger. Also with notifyCommit you can disable unwanted triggering only with checkbox "Ignore post commit hooks".
danielbeck:
- Original comment link
Raw content of original comment:
queue shouldn't merge the same job with different actions. Either you will lose all builds for different commits in one job/repo.
Have your action implement Queue.QueueAction to control this behavior.
(Update) Noticed only now this is obsolete as Jesse already addressed it. Sorry about that.
queue shouldn't merge the same job with different actions. Either you will lose all builds for different commits in one job/repo.
Have your action implement Queue.QueueAction to control this behavior.
(Update) Noticed only now this is obsolete as Jesse already addressed it. Sorry about that.
jglick:
- Original comment link
Raw content of original comment:
notifyCommit required enabled scm trigger.
Sure, though the schedule may be empty. Or @daily, which is wise anyway.
it will trigger all jobs with enabled scm trigger (even without GH trigger)
Only those whose SCM URL matches the notified URL.
we can just remove trigger from plugin and handle triggering from root action
No, the purpose of the GH-specific trigger IIUC is to interpret the web hook format that GH sends natively, since you have no opportunity to set up a custom push event script.
with notifyCommit you can disable unwanted triggering only with checkbox "Ignore post commit hooks".
I am not sure what your point is here.
notifyCommit required enabled scm trigger.
Sure, though the schedule may be empty. Or @daily, which is wise anyway.
it will trigger all jobs with enabled scm trigger (even without GH trigger)
Only those whose SCM URL matches the notified URL.
we can just remove trigger from plugin and handle triggering from root action
No, the purpose of the GH-specific trigger IIUC is to interpret the web hook format that GH sends natively, since you have no opportunity to set up a custom push event script.
with notifyCommit you can disable unwanted triggering only with checkbox "Ignore post commit hooks".
I am not sure what your point is here.
lanwen:
- Original comment link
Raw content of original comment:
points to not use notifyCommit: 1. It breaks all existing setups (you should explicitly enable scm trigger, which usually not enabled) 2. If you don't want to trigger job by notifyCommit you should explicitly turn it off with additional git behaviour option (most of people don't know about this option) - it maybe in case of nightly builds, when we check git only at exact time to start heavy build.
so it should be implemented without notifyCommit. Maybe we should use GH events api to poll state changing
points to not use notifyCommit:
1. It breaks all existing setups (you should explicitly enable scm trigger, which usually not enabled)
2. If you don't want to trigger job by notifyCommit you should explicitly turn it off with additional git behaviour option (most of people don't know about this option) - it maybe in case of nightly builds, when we check git only at exact time to start heavy build.
so it should be implemented without notifyCommit. Maybe we should use GH events api to poll state changing
integer:
- Original comment link
Raw content of original comment:
1. Probably can be done optional, default in GH trigger as new, existed like it was.
1. Probably can be done optional, default in GH trigger as new, existed like it was.
djryan:
- Original comment link
Raw content of original comment:
+1 for any improvements around this area. I've currently got rapidly increasing threads every time there's a github push and these only come down slowly when the machine stops being busy (which is rare). Eventually, java hits the thread limit and then either one of the slaves drops its job or the whole thing dies and requires a reboot:
Waiting to acquire C:\dev : Computer.threadPoolForRemoting [#1070] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:255) hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:234) hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1467) hudson.model.AbstractProject._poll(AbstractProject.java:1444) hudson.model.AbstractProject.poll(AbstractProject.java:1355) com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:73) com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:98) hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Waiting to acquire C:\dev : Computer.threadPoolForRemoting [#1070]
+1 for any improvements around this area. I've currently got rapidly increasing threads every time there's a github push and these only come down slowly when the machine stops being busy (which is rare). Eventually, java hits the thread limit and then either one of the slaves drops its job or the whole thing dies and requires a reboot:
Waiting to acquire C:\dev : Computer.threadPoolForRemoting [#1070] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:503) hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:255) hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:234) hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1467) hudson.model.AbstractProject._poll(AbstractProject.java:1444) hudson.model.AbstractProject.poll(AbstractProject.java:1355) com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:73) com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:98) hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Waiting to acquire C:\dev : Computer.threadPoolForRemoting [#1070]