plugin rapidly drains GitHub API quota in poll mode

[ixdy]

We recently upgraded to 1.32.8 (from 1.31.4, since many intervening versions have been broken in different ways), and we're quickly burning through all of our GitHub API quota:

$ while true; do curl -slL https://api.github.com/rate_limit?access_token=[elided] | jq .resources.core.remaining ; sleep 1; done
592
591
589
587
586
584
582
575
564
553
545
535
529
519
509
500
491
483
478
476
473

Note that I restarted the plugin about 15m ago; this ratelimit value starts at 5000.

Looking at the Jenkins log, it looks like its querying each PR multiple times per minute. Refreshing the log, I'll see things like

Jul 05, 2016 10:05:04 PM INFO org.jenkinsci.plugins.ghprb.GhprbPullRequest updatePR
Pull request #16,062 was updated on repo kubernetes/kubernetes but there aren't any new comments nor commits; that may mean that commit status was updated.
Jul 05, 2016 10:05:04 PM INFO org.jenkinsci.plugins.ghprb.GhprbPullRequest updatePR
Pull request #13,925 was updated on repo kubernetes/kubernetes but there aren't any new comments nor commits; that may mean that commit status was updated.
Jul 05, 2016 10:05:04 PM INFO org.jenkinsci.plugins.ghprb.GhprbPullRequest updatePR
Pull request #13,216 was updated on repo kubernetes/kubernetes but there aren't any new comments nor commits; that may mean that commit status was updated.

repeated again and again for the same PRs, even though nothing is changing.

[mmitche]

What is the poll rate set at?

[ixdy]

Every two minutes, though it appeared to be polling more frequently than that.

[ixdy]

We haven't had any issues with 1.31.4 using the same settings.

[benpatterson]

may or may not be related but linking #204 here

[JoeNunnelley]

it would seem we hit this issue with 1.39. reverting the 1.35 seems to have improved it.

[toidiu]

is there a status on this?

[AboorvaDevarajan]

Still seeing this in v1.39. Downgrading the plugin to v1.35 seems to have brought down the rate of requests. Is there a better way to track the requests sent from GHPRB to github, so that we can triage this issue?

[mmitche]

@AboorvaDevarajan I had, a long time ago, instrumented the github API library so that it logged when it made an actual call to the REST API (rather than some cached data).

What are the characteristics of your server? Do you have lots of jobs that use the plugin?

[toidiu]

we have two different jenkins running with the same jobs. The one with version 1.36.0 uses fewer requests that the version with 1.39.0

github rate limit resets every hour(5000 api calls per hour) and by the end of the hour there is a difference of almost 500 requests.

[AboorvaDevarajan]

@mmitche There are around 50 jobs in total, in which 12 are using ghprb polling (2 mins polling interval for each), and another 12 are using ghprb webhook.

But clearly there is a difference between v1.35 and v1.39 (with above configuration) with v1.37 requests made is a lot, around 100-200/min. with v1.35, requests made are around 20-30/min.

I will see if I can capture some data using sniffer for more info.

[mmitche]

@AboorvaDevarajan That's not a lot of jobs. I think it's likely that there were only a few changes. The plugin treats each job as its own unit and doesn't share data between them so it can have horrible multipliers of API requests.

[bstinsonmhk]

Is this with the latest github-api plugin installed? I'm curious if someone has instrumented against https://github.com/kohsuke/github-api/pull/283 yet.

[toidiu]

I am noticing that version 1.35 makes 1 req/repo.

Version 1.39 makes (1 req/ PR in repo) + (1 req/ PR in repo if there is a comment on the PR)

That is a significant increase in the # of requests.

[bjoernhaeuser]

Are you using hooks or only polling?

[toidiu]

@bjoernhaeuser that diagnosis was for polling only.

We are in the process of switching to hooks now(we had to jump thru some hoops because our Jenkins setup is in a private/internal network) since that is a more scalable and reliable method.

[bjoernhaeuser]

We switched to hooks some month ago - also having a little bit too complex networking. Hooks work very great.

[RoyTinker]

@toidiu, May I ask how you switched to hooks for a Jenkins server behind the firewall?

[julianh2o]

Any update on this? We've run into the same issue with hitting the rate limit despite the fact that we only have a few different repositories being polled. (We're hoping to ultimately use hooks.. but being behind a firewall makes this more difficult)

We're using GHPRB 1.40.0

It seems like the solution is to use GitHub's conditional requests to keep from unnecessarily draining the rate limits.

[RoyTinker]

Filed a separate issue for adding If-Modified-Since to request headers when polling PRs: #653

[sgleske-ias]

I recommend using webhooks and not polling. There's tech such as https://webhookrelay.com/ which I've personally used if you don't want to expose your Jenkins instance to the internet.

[rusenask]

thanks @sgleske-ias for mentioning it :) Last year I've wrote a blog post on configuring it: https://webhookrelay.com/blog/2017/11/23/github-jenkins-guide/

[eplodn]

Jenkins log is full of the messages about very old PRs (some are more than a year old):

Jul 10, 2018 11:14:42 AM INFO org.jenkinsci.plugins.ghprb.GhprbPullRequest updatePR
Pull request #1,117 was updated on repo <myorg>/<myrepo> but there aren't any new comments nor commits; that may mean that commit status was updated.

This repeats every few minutes, and nothing is changing in that PR.

[RoyTinker]

@eplodn PR polling hits all open PRs, even every old ones. That's part of why peoples' quotas are getting drained so quickly.

[eplodn]

@RoyTinker I guess my problem is with Pull request #1,117 was updated; as far as I can see it wasn't.

[unphased]

Sad to see this six year old issue is still pending without any workaround or resolution. Might need to go ahead and try the webhook relay.