[Bug][github_graphql] issue_collector gets stuck at Collect Issues task for Github enterprise

[ksarabi-sp]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

2024-09-09T16:58:31.112954005Z time="2024-09-09 16:58:31" level=info msg=" [pipeline service] [pipeline #1] [task #2] executing subtask Collect Issues" 2024-09-09T16:58:31.119890379Z time="2024-09-09 16:58:31" level=info msg=" [pipeline service] [pipeline #1] [task #2] [Collect Issues] start graphql collection" 2024-09-09T16:58:31.146203954Z time="2024-09-09 16:58:31" level=info msg=" [pip eline service] [pipeline #1] [task #2] rate limit remaining exhausted, waiting for next period."

2024-09-09T17:01:30.770231161Z time="2024-09-09 17:01:30" level=info msg=" [pipeline service] [pipeline #1] [task #2] github graphql init success with remaining 0/0 and will reset at 0001-01-01 00:00:00 +0000 UTC" 2024-09-09T17:04:30.832605306Z time="2024-09-09 17:04:30" level=info msg=" [pipeline service] [pipeline #1] [task #2] github graphql init success with remaining 0/0 and will reset at 0001-01-01 00:00:00 +0000 UTC" 2024-09-09T17:07:30.891184522Z time="2024-09-09 17:07:30" level=info msg=" [pipeline service] [pipeline #1] [task #2] github graphql init success with remaining 0/0 and will reset at 0001-01-01 00:00:00 +0000 UTC"

What do you expect to happen

not to get stuck with collecting issues task

How to reproduce

1- create a connector by connecting to github enterprise 2- select one repo 3- create project and add above github connector to it 4- start collect data

Anything else

No response

Version

v1.0.1-beta9

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[dosubot[bot]]

Hello @ksarabi-sp! I'm here to help you with any bugs, questions, or contributions you have for the project. Let's tackle this issue together!

The issue_collector gets stuck at the "Collect Issues" task for GitHub Enterprise in version v1.0.1-beta9 because the CollectIssues function does not handle pagination correctly. Specifically, the BuildQuery function does not properly set the skipCursor variable, which is necessary for paginating through the issues.

Here is the relevant code snippet:

err = apiCollector.InitGraphQLCollector(api.GraphqlCollectorArgs{
    GraphqlClient: data.GraphqlClient,
    PageSize:      10,
    BuildQuery: func(reqData *api.GraphqlRequestData) (interface{}, map[string]interface{}, error) {
        query := &GraphqlQueryIssueWrapper{}
        if reqData == nil {
            return query, map[string]interface{}{}, nil
        }
        ownerName := strings.Split(data.Options.Name, "/")
        variables := map[string]interface{}{
            "pageSize":   graphql.Int(reqData.Pager.Size),
            "skipCursor": (*graphql.String)(reqData.Pager.SkipCursor),
            "owner":      graphql.String(ownerName[0]),
            "name":       graphql.String(ownerName[1]),
        }
        return query, variables, nil
    },
    GetPageInfo: func(iQuery interface{}, args *api.GraphqlCollectorArgs) (*api.GraphqlQueryPageInfo, error) {
        query := iQuery.(*GraphqlQueryIssueWrapper)
        return query.Repository.IssueList.PageInfo, nil
    },
    ResponseParser: func(queryWrapper any) (messages []json.RawMessage, err errors.Error) {
        query := queryWrapper.(*GraphqlQueryIssueWrapper)
        issues := query.Repository.IssueList.Issues
        for _, rawL := range issues {
            if since != nil && since.After(rawL.UpdatedAt) {
                return messages, api.ErrFinishCollect
            }
            messages = append(messages, errors.Must1(json.Marshal(rawL)))
        }
        return
    },
})

The skipCursor is not being updated correctly, causing the pagination to fail and the task to get stuck ^[1].

To continue talking to Dosu, mention @dosu.

[klesh]

1. Did the GraphQL enabled on the GitHub connection page?
2. Were there concurrently pipelines running at them same time?

[ksarabi-sp]

• Did the GraphQL enabled on the GitHub connection page? Yes, It is enabled by default.
• Were there concurrently pipelines running at them same time? No, only one pipeline was running to collect data from a single repository.

[realhuseyn]

@klesh GraphQL option is not shown on UI

[klesh]

@ksarabi-sp @realhuseyn Are you two working together? 😂

Is it Github Cloud or Enterprise?

[realhuseyn]

@klesh noo :D I am using Github Cloud Enterprise

[klesh]

@realhuseyn Could you try using multiple PATs(from different Github accounts) and see if it helps?

[realhuseyn]

@klesh I will try. Currently, I am using two different PAT. I will add a new one

[realhuseyn]

@klesh I tried. I have added 3 PATs and my pipeline works normally. But I want to use Github App, because it has more rate limits than PAT.

[ksarabi-sp]

1. Did the GraphQL enabled on the GitHub connection page?
2. Were there concurrently pipelines running at them same time?

@klesh I am using Github enterprise 3.14 and do not have any rate limit, but still have this issue when using it for our GHE.

[klesh]

@klesh I tried. I have added 3 PATs and my pipeline works normally. But I want to use Github App, because it has more rate limits than PAT.

@realhuseyn I completely agree! It would be fantastic if someone could address and resolve this issue.

[klesh]

1. Did the GraphQL enabled on the GitHub connection page?
2. Were there concurrently pipelines running at them same time?

@klesh I am using Github enterprise 3.14 and do not have any rate limit, but still have this issue when using it for our GHE.

@ksarabi-sp That seems unusual. Your logs indicate that your GHE was rejecting API requests due to rate limiting. Perhaps you could write a simple script to make concurrent API requests and check if the same error occurs. You can determine the request rate by searching for “interval” in the log.

[ksarabi-sp]

1. Did the GraphQL enabled on the GitHub connection page?
2. Were there concurrently pipelines running at them same time?

@klesh I am using Github enterprise 3.14 and do not have any rate limit, but still have this issue when using it for our GHE.

@ksarabi-sp That seems unusual. Your logs indicate that your GHE was rejecting API requests due to rate limiting. Perhaps you could write a simple script to make concurrent API requests and check if the same error occurs. You can determine the request rate by searching for “interval” in the log.

@klesh are you sure it is getting API limit from GHE? since we do not have API limit in our GHE server, where in GHE we can see if there is any API limit? is it possible that calling API in Github.com?

[klesh]

@ksarabi-sp I took another look at the code , and it seems like it is assuming there is rate limit anyway. Can you try setting some sort of rate-limit, like 5000/hour, and see if the problem is gone? There may be a bug here.

[ksarabi-sp]

@klesh Please let me know how and where I can set 5000/hour rate-limit?

[jeffschaper]

I'm experiencing the same issue

[jeffschaper]

Here are more details for my specific issue:

DevLake version: v1.0.1@e061ef2 GitHub Enterprise Server version: 3.13.3 GHES rate limiting enforced: No GHES API endpoint: https://github.<company>.com/api/v3 There doesn't appear to be an option for graphql in the data connection configuration - at least, not in the UI. I can confirm "enableGraphql": false using the {{baseUrl}}/plugins/github/connections API endpoint.

Relevant logs:

time="2024-11-04 15:12:32" level=info msg=" [pipeline service] [pipeline #1] [task #2] rate limit remaining exhausted, waiting for next period."

time="2024-11-04 16:21:35" level=info msg=" [pipeline service] [pipeline #1] [task #2] github graphql init success with remaining 0/0 and will reset at 0001-01-01 00:00:00 +0000 UTC"

[klesh]

@ksarabi-sp it is in the connection editing page

[acurags]

I get the same error on self hosted Github Enterprise without rate limiting set. The pipeline seems stuck. Even with rate limit set to 4,500 it doesn't complete.

creating scheduler for api "https://api.github..net/", number of workers: 6, 4500 reqs / 1h0m0s (interval: 800ms)" github graphql init success with remaining 0/0 and will reset at 0001-01-01 00:00:00 +0000 UTC" [Collect Issues] start graphql collection" rate limit remaining exhausted, waiting for next period."

[klesh]

@acurags It might be caused by the GHE not returning any rate limit headers at all. Can you check if there are rate limit settings on the GHE to make it output those headers to verify the theory?

[cpoenisch]

Hi @klesh, We are encountering the same issue on our GHE instance with disabled rate limits. It seems that the null response on any GraphQL query requesting the available rate limits is the problem here.

Sample query on https://<ghe>/api/graphql:

query {
  rateLimit {
    cost
    limit
    remaining
    resetAt
  }
}

returns

{
    "data": {
        "rateLimit": null
    }
}

The github_graphql plugin probably misinterprets this here and on the client side it throws the limit exhausted log blocking the task completely.

I hope this will help you to find a suitable solution to this problem.

Additionally, here are the response headers with fallback to default value of 5000, even if the rate limits are disabled. In this case, the value for “X-RateLimit-Used” remains at 0 and the value for “X-RateLimit-Remaining” is never reduced.

{
    "key": "X-RateLimit-Limit",
    "value": "5000"
},
{
    "key": "X-RateLimit-Remaining",
    "value": "5000"
},
{
    "key": "X-RateLimit-Reset",
    "value": "1731516776"
},
{
    "key": "X-RateLimit-Used",
    "value": "0"
},
{
    "key": "x-ratelimit-resource",
    "value": "graphql"
}

[klesh]

Thanks for the information.

[klesh]

@cpoenisch Hi, Please upgrade to https://github.com/apache/incubator-devlake/releases/tag/v1.0.2-beta5 and see if the problem is fixed. Thanks.

[J0shSkqlq]

Hey @klesh, Colleague of @cpoenisch here, Unfortunately I can't confirm whether your changes fixed our Problem since we now enabled the API rate limits in our GHE as a workaround. Thank you for your rapid fix nonetheless!

[klesh]

@J0shSkqlq Glad it worked out for you.

[jeffschaper]

Hey @klesh, Colleague of @cpoenisch here, Unfortunately I can't confirm whether your changes fixed our Problem since we now enabled the API rate limits in our GHE as a workaround. Thank you for your rapid fix nonetheless!

@J0shSkqlq – Could you provide more details about how you handled this workaround? Did you set an arbitrary API rate limit in your GHE, such as 1,000,000 requests?

[J0shSkqlq]

@jeffschaper - Exactly, we just set an arbitrary limit. 50,000 request in our case.

[jeffschaper]

I found a workaround that might be relevant to this issue.

I disabled GraphQL when establishing a GitHub data connection via {{baseUrl}}/plugins/github/connections POST DevLake API call with the following payload.

{
  "name": "github",
  "endpoint": "https://github.<company>.com/api/v3",
  "authMethod": "AccessToken",
  "token": "<token>",
  "proxy": "",
  "enableGraphql": false
}

As a result, the data collection process seems slower, but the pipelines don't fail as they did before.

My team is looking forward to the official release of v1.0.2 so we can re-enable GraphQL and speed up the data collection process for our projects.

I hope this bit of information is able to help others get unstuck until the next stable release is published.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.