Support pagination of GraphQL API responses

[carlocab]

Verification

• [X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature

We have a paginate_rest method in GitHub::API:

https://github.com/Homebrew/brew/blob/e9a1cd41d78c7c99ccf415be24727756cfa3ec5f/Library/Homebrew/utils/github/api.rb#L280-L287

We should have a similar paginate_graphql method for paginating responses from the GitHub GraphQL API.

What is the motivation for the feature?

Using the GraphQL API with pagination is a much better fit for querying repository pull requests.

Currently, we use either the Search API, which is unreliable, or the REST API, which is slow possibly because it returns much more information than we need.

See also #15398.

How will the feature be relevant to at least 90% of Homebrew users?

This would help make CI run more efficiently, which affects all Homebrew users.

What alternatives to the feature have been considered?

The status quo.

[MikeMcQuaid]

As in the other related issue: if it has no native dependencies: it may be worth exploring https://github.com/octokit/octokit.rb for our API usage. It has support for caching, pagination and various other things that would benefit us.

[issyl0]

GraphQL pagination learnings: https://til.simonwillison.net/github/graphql-pagination-python

[Bo98]

As in the other related issue: if it has no native dependencies: it may be worth exploring https://github.com/octokit/octokit.rb for our API usage. It has support for caching, pagination and various other things that would benefit us.

This is a good idea I think.

One key difference is that this uses Net::HTTP rather than curl. How much that matters I'm not entirely sure yet in terms of proxies etc.

But I reckon it wouldn't be too difficult to add a Faraday adapter (Octokit uses Faraday under the hood) that uses curl instead. Faraday has good modularity for that, to the extent that faraday-net_http actually lives in a completely separate gem.

[Bo98]

Oh just reading now what this issue is about: note that octokit.rb does not support GraphQL pagination (or really anything GraphQL at all). So it won't actually help us resolve this particular issue.

[MikeMcQuaid]

One key difference is that this uses Net::HTTP rather than curl. How much that matters I'm not entirely sure yet in terms of proxies etc.

Yeh, that's true (and probably the main reason we didn't use it until now).