forge-pull hits rate limits

[aspiers]

I hit F y to forge-pull, and after some seconds I got this:

error in process filter: if: HTTP Error: 403, "Forbidden", "/graphql", ((documentation_url . "https://developer.github.com/v3/#abuse-rate-limits") (message . "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."))

It would be great if rate limiting was built in to avoid this.

[vermiculus]

Rate limiting is a bit difficult with graphql as the cost of a query is calculated at run time. Forge's current methodology of always pulling the entire repository's information won't play well with this in the end.

[vermiculus]

I say that with the confidence of someone who knows what they're talking about – but I don't actually. I'm just 'pretty sure' about forge's behavior from a slightly-firmer-than-vague recollection from weeks ago.

[tarsius]

Is this when trying to fetch a huge repository? I'm actually a bit surprised this happened. I had some debug statements that printed the relevant information and it never got even close to being close to the limit.

But it should be possible to slow down when getting close to the limit. It might be a while until I implement that.

[kommen]

I also wanted to give forge a try, but the forge-pull failed with the same error as posted above for me as well.

This is a large, but not very large repository with ~2000 PRs (~20 open) and 1700 issues (~200 open).

[vermiculus]

This is a large, but not very large repository with ~2000 PRs (~20 open) and 1700 issues (~200 open).

Isn't magit itself about this size?

[aspiers]

@tarsius commented on null:

Is this when trying to fetch a huge repository?

I don't think so but unfortunately I can't remember which repo I ran it on - apologies for not being able to provide more details. It was pretty much the first time I tried Forge so I thought maybe it's not hard to reproduce - but tried again just now with (IIRC) a much bigger repo and it was fine, even though it took ~60s or so to fetch almost 2k PRs, as opposed to only a few seconds when I originally saw the error from the rate limiting. Maybe there was some other factor at play, e.g. the mess I made of initial token creation (as mentioned in https://github.com/magit/ghub/pull/78).

[tarsius]

I don't consider that a huge repository and it is indeed at the same magnitude as Magit.

It's weird that you hit the rate limit. While neither ghub nor forge deal with rate limiting properly I did use some temporary debug statements for a while and that indicated that Forge didn't get even close to going over the limit when fetching such a repository for the first time.

[tarsius]

We should consider the rate limiting headers and report when we get close. Most of that probably has to be implemented in ghub.

[vermiculus]

We should consider the rate limiting headers and report when we get close.

Do learn from my lessons in ghubp-ratelimit – namely that GHE instances often have rate-limiting disabled and /rate_limit will return 404 if so.

[agzam]

I'm hitting the same error while with one of our private repos:

error in process filter: ghub--signal-error: HTTP Error: 403, "Forbidden", "/graphql", ((documentation_url . "https://developer.github.com/v3/#abuse-rate-limits") (message . "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."))

[mpereira]

Same here. It worked after a few tries:

Pulling mesosphere/dcos-commons...
error in process filter: ghub--signal-error: HTTP Error: 403, "Forbidden", "/graphql", ((documentation_url . "https://developer.github.com/v3/#abuse-rate-limits") (message . "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."))
error in process filter: HTTP Error: 403, "Forbidden", "/graphql", ((documentation_url . "https://developer.github.com/v3/#abuse-rate-limits") (message . "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."))
Pulling mesosphere/dcos-commons...done
Storing mesosphere/dcos-commons...done

[timor]

I am currently trying to forge-pull NixOS/nixpkgs. On the first try, I got a 502 (see #20). On the second invocation (couple of minutes later), I got this 403 with a an "abuse detection triggered" message.

(Magit 20190228.2035, Git 2.18.1, Emacs 26.1, gnu/linux)

[mbolivar]

Update: a series of retries over a few hours eventually got me over the hill.

I also hit this issue trying to pull https://github.com/zephyrproject-rtos/zephyr (roughly 9800 closed PRs, 400 open PRs, 5500 closed issues, 800 open issues), even with multiple retries. I am successfully able to forge-pull a much smaller repository.

Here are the *Messages* errors:

error in process filter: HTTP Error: 403, "Forbidden", "/graphql", ((documentation_url . "https://developer.github.com/v3/#abuse-rate-limits") (message . "You have triggered an abuse detection mechanism. Please wait a few minutes before you try again."))

error in process filter: ghub--signal-error: HTTP Error: 502, "Bad gateway", "/graphql", ((data . "null") (errors ((message . "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `FCB5:1DC5:81A5B9:11A3BC7:5CF7E463` when reporting this issue."))))

error in process filter: HTTP Error: 502, "Bad gateway", "/graphql", ((data . "null") (errors ((message . "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `FCB5:1DC5:81A5B9:11A3BC7:5CF7E463` when reporting this issue."))))

error in process filter: ghub--signal-error: HTTP Error: 502, "Bad gateway", "/graphql", ((data . "null") (errors ((message . "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `FCB5:1DC5:81B920:11A42E1:5CF7E46D` when reporting this issue."))))

[mbolivar]

According to their info pages, I am using:

• Forge version 0.1.0 (v0.1.0-113-g9a4ce77+1)
• Magit version 2.90.1 (v2.90.1-670-g01efbb2de+1).
• Ghub version 3.2.0 (v3.2.0-10-g9017033+1).

[Silex]

This happens to me on https://github.com/bbatsov/projectile if you want to debug.

EDIT: actually, maybe it's simply because I did @ f f twice as I did other things and couldn't know if forge was still pulling, maybe some forge-currently-operating-p or forge-status would help reguarding this :wink:

[ddoc]

I too am getting errors on nixpkgs.

error in process filter: HTTP Error: 502, "Bad gateway", "/graphql", ((data) (errors ((message . "Something went wrong while executing your query. This may be the \
result of a timeout, or it could be a GitHub bug. Please include `B34A:3998:DBEFB7:E97D62:62DBC025` when reporting this issue."))))

(This was my first post to github using magit-forge. Wow is it cool! I wish I could use it with nixpkgs though. :frowning:)

[choppsv1]

I get this error with FRRouting/frr not sure how to use forge with this repo, it doesn't seem to make any progress after the error which would at least allow for retrying until it finished.

Pulling FRRouting/frr...
error: HTTP Error: 502, "Bad gateway", "/graphql", ((data) (errors ((message . "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `C4AC:8BD5:63547CE:CCF20B5:63FCBD24` when reporting this issue."))))
error in process filter: HTTP Error: 502, "Bad gateway", "/graphql", ((data) (errors ((message . "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `C4AC:8BD5:63547CE:CCF20B5:63FCBD24` when reporting this issue."))))

I wrote some elisp to fetch all 12000+ topics one by one and that worked; however, I still can't use @ f f to update things, same error.

(defun forge-pull-next-3000 (start)
  "Fetch the range of topics from [start - 2999, start]"
  (interactive "Ntopic number: ")
  (let ((repo (forge-get-repository t))
        (end (max (- start 2999) 0))
        failed
        topic)
    (while (<= end start)
      (setq topic (ignore-errors (forge--pull-topic repo (forge-issue :repository (oref repo id) :number start))))
      (unless topic
          (setq failed (append failed (list start))))
      (sleep-for 1)
      (setq start (- start 1)))
    failed
    ))

Version:

GNU Emacs 28.2 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0) of 2023-01-29
Magit 20230120.1616 [>= 3.3.0.50-git], Git 2.39.2, Emacs 28.2, gnu/linux
.emacs.d/elpa/28.2/develop/forge-20230222.1917

[mrcnski]

Same error here. Does this not work for large repos (7000+ issues/PRs)?

[tarsius]

We don't actually know how the sausage is made, which makes it hard to find workarounds.

It is also quite likely that there are different causes for "this" issue. It seems that we used to get "exceeded limit" errors, while lately we are getting "something wrong, shrug" errors. That might just be because the implementation changed on their end, but it could also means we are dealing with different problems.

I think it is actually quite hard to exceed the overall rate limit.

My educated guess is that we get the "something wrong, won't tell you more" error when calculating a single response takes longer than some undisclosed time limit. The reason we get a 502 "Bad gateway" error, likely is that the server that responds to our request isn't the one that calculates the result, but it is the one that implements the timeout. If it didn't get a response in time, then it gives up and gives us a "something is wrong, my guess is as good as yours" error.

I am guessing that repeated requests sometimes succeed, is that earlier requests causes the relevant data to be loaded into memory, so next time the same data is requested, that part of the work is already done and it takes less time overall, which might just keep us below the timeout.

---

So why do some individual requests take too long?

For edges that return a collection of items we already limit to 100 items per request. We request that explicitly, but if we didn't, we would get the same number, as that is the default.

The problem is that such edges can be nested. Fetching, in one request, 100 issues with 10 post on average might not be a problem. But fetching 100 issues with 40 posts on average, might just be too much.

I believe, the fact that this seems to happen more often with large repositories is somewhat coincidental. It is just more likely in very active repositories that there are issues that are so active that they push up the average number of posts too far.

---

Luckily we added support for limiting the number of items per edge further. This can be done globally using ghub-graphql-items-per-request, or for just one repository using the Git variable forge.graphqlItemLimit.

I've set this to 50 in a few repositories in which I used to sometimes get a 502, and I cannot remember seeing any such errors since.

Please give that a try for a while, and report back if that solves the issue for you too.

[tarsius]

I've set this to 50 in a few repositories in which I used to sometimes get a 502, and I cannot remember seeing any such errors since.

Okay... jinxed it. Got one right now.

[tarsius]

Or on the contrary, this might demonstrate that this approach works.

The request was in this repository, for which I have not reduced the limit until now. Simply repeating the request did not help. However, limiting to 50 items in this repository, did help. :partying_face:

[mrcnski]

I got the same error with 50, but 20 seems to have worked for me, though it took a while. (And it only displays 65 PRs/issues, not sure if that is expected or not.) The repo is https://github.com/paritytech/polkadot if it helps.

Random idea, but could it be because I generated a "classic" non-expiring token?

Edit: another idea, maybe it has to do with the amount of activity per issue/PR? Because there can be a large number of comments per item in that repo, and a large number of commits per PR. Actually I would guess that's the issue in my case.