incubator-devlake icon indicating copy to clipboard operation
incubator-devlake copied to clipboard
verified publisher icon apache

[Bug][Devlake][GitHub] GitHub app connection data source usability issues

Open petercort opened this issue 1 year ago • 9 comments

Search before asking

  • [x] I had searched in the issues and found no similar issues.

What happened

Creating a GitHub App connection isn't working as expected.

  1. No organizations or repositories are found 'naturally' after the connection is created
  2. Can't multiselect to add repositories when they are specified into the datasources search

Image Image Image Image Image

What do you expect to happen

I would expect that the repositories in the organization where the app has been installed would auto populate and I could select multiples to add as a data source.

How to reproduce

  1. Configure a GitHub App as a connection
  2. Add data sources - no

Anything else

I was able to reproduce this behavior with a GitHub app installed into an organization and my personal account.

I'm willing to take a crack at a PR if it's not a massive thing. I'll spend some time setting up a debugger and seeing if I can get more info. Might need someone more skilled than me (cough cough copilot) to fix this.

I also tested this with the v0.21.0@ac4f7fe and it's reproducible there.

Here's the error message formatted nicely:

level=error msg="HTTP 400 error
    caused by: attached stack trace
     -- stack trace:
        | github.com/apache/incubator-devlake/helpers/pluginhelper/api.(*DsScopeApiHelper[...]).PutMultiple
        | /app/helpers/pluginhelper/api/ds_scope_api_helper.go:114  
        | github.com/apache/incubator-devlake/plugins/github/api.PutScopes  
        | /app/plugins/github/api/scope_api.go:42
        | github.com/apache/incubator-devlake/server/api.handlePluginCall.func1
        | /app/server/api/router.go:135
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
        | github.com/apache/incubator-devlake/server/api.OAuth2ProxyAuthentication.func1
        | /app/server/api/middlewares.go:95
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
        | github.com/apache/incubator-devlake/server/api.RestAuthentication.func1
        | /app/server/api/middlewares.go:117
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
        | github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
        | github.com/gin-gonic/gin.LoggerWithConfig.func1
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:240
        | github.com/gin-gonic/gin.(*Context).Next
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174
        | github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620
        | github.com/gin-gonic/gin.(*Engine).ServeHTTP
        | /go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576
        | net/http.serverHandler.ServeHTTP
        | /usr/local/go/src/net/http/server.go:2936
        | net/http.(*conn).serve
        | /usr/local/go/src/net/http/server.go:1995
        | runtime.goexit
        | /usr/local/go/src/runtime/asm_arm64.s:1172
    Wraps: (2) invalid data row (400)
    Error types: (1) *withstack.withStack (2) *errutil.leafError"

Version

v1.0.0-beta11@15c4c09

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

petercort avatar Jun 21 '24 16:06 petercort

Hello, @petercort! I'm here to assist you with any bugs, questions, or contribute to the repository. Let me know how I can help you while we wait for a human maintainer.

To resolve the issue where creating a GitHub App connection does not auto-populate organizations or repositories and does not allow multiselect to add repositories, ensure the following:

  1. Correct Permissions: Verify that the GitHub App has the necessary permissions. The required permissions for collecting data from repositories are:

    • Repository
      • Actions
      • Administration
      • Checks
      • Commit statuses
      • Contents
      • Deployments
      • Issues
      • Metadata
      • Pull requests
    • Organization
      • Members

  2. Endpoint URL: Ensure the Endpoint URL is correctly set to https://api.github.com/ and ends with a /.

  3. Personal Access Token: If using a Personal Access Token (PAT), ensure it has the required permissions:

    • For public repositories:
      • repo:status
      • repo_deployment
      • read:user
      • read:org
    • For private repositories:
      • repo
      • read:user
      • read:org

  4. GraphQL API: If using fine-grained PATs, ensure the Use GraphQL APIs option is disabled as fine-grained PATs do not support GraphQL APIs.

  5. Proxy Settings: If behind a corporate firewall or VPN, ensure the Proxy URL is correctly configured.

  6. Rate Limits: Ensure the rate limit settings do not exceed the maximum allowed by GitHub.

  7. Add Data Scopes: After creating the connection, ensure you add the data scopes correctly. Choose the GitHub repositories you wish to collect either by finding them in the miller column or searching. Note that you can only add public repositories through the search box.

  8. Test and Save Connection: Click Test Connection, if the connection is successful, click Save Connection to add the connection.

If all the above settings are correct and the issue persists, refer to the GitHub Configuration Guide for detailed instructions [1][2][3][4][5].

Additionally, specific error messages or logs can indicate why the organizations or repositories are not being auto-populated. Functions like listGithubOrgRepos, listGithubAppInstalledRepos, and listGithubUserOrgs return errors if they encounter issues during the API calls. These errors are propagated back to the caller, which can then log or display them to indicate the issue [6].

To continue talking to Dosu, mention @dosu.

dosubot[bot] avatar Jun 21 '24 17:06 dosubot[bot]

Can you share the request payload from your browser under the Network tab? Please remove any sensitive information before you post but keep the overall structure.

klesh avatar Jun 24 '24 04:06 klesh

For the connection being created and scopes loading:

Once the connection is created, I've got 2 calls http://localhost:4000/api/plugins/github/connections/11/scopes?page=1&pageSize=10&blueprints=true Response:

{"count":0,"scopes":[]}

http://localhost:4000/api/plugins/github/connections/11/remote-scopes Response:

{
  "children": [
      {
          "type": "scope",
          "parentId": "petes-fre-shavocado-stand",
          "id": "UID",
          "name": "roadwork",
          "fullName": "petes-fre-shavocado-stand/roadwork",
          "data": null
      }
  ],
  "nextPageToken": ""
}

Multi selecting scope on a manual search

http://localhost:4000/api/plugins/github/connections/11/scopes

payload for adding multiple repos

petercort avatar Jun 24 '24 15:06 petercort

@petercort We are facing same issue while integrating Github app with devlake , we have validated all the required permissions and end point url and still facing same issue @klesh

Mantesh-Globant avatar Jul 02 '24 07:07 Mantesh-Globant

Seems like there are multiple problems to me:

  1. http://localhost:4000/api/plugins/github/connections/11/remote-scopes is not responding correctly: it doesn't return repositories under the org.
  2. the config-ui is sending null to the PUT http://localhost:4000/api/plugins/github/connections/11/scopes API

Am I right?

klesh avatar Jul 04 '24 03:07 klesh

@klesh from what I can see that's correct.

I've been running into a lot of issues setting up a debug environment to get more info about the calls, so I can't offer much more at the moment :(

petercort avatar Jul 10 '24 11:07 petercort

There are a couple of bugs in the Github App implementation. I suggest that you use tokens as a workaround for the time being, It might take quite a while to fix all those problems.

klesh avatar Jul 11 '24 08:07 klesh

I had same issue , I did a workaround to add scopes to the connection programmatically using first github API to fetch repository id's and then add them to connection using devlake API. However pipelines are generally flaky with the setup and have got a weird error on the 1st and last step of the pipeline without any actual error in the logs , with pipeline status set to Partially Successful.

sanjchhoker avatar Jul 16 '24 12:07 sanjchhoker

@sanjchhoker Can you help submit a new issue and describe what happened in detail ? It seems a differcent problem.

d4x1 avatar Aug 09 '24 10:08 d4x1

Seems like there are multiple problems to me:

  1. http://localhost:4000/api/plugins/github/connections/11/remote-scopes is not responding correctly: it doesn't return repositories under the org.
  2. the config-ui is sending null to the PUT http://localhost:4000/api/plugins/github/connections/11/scopes API

Am I right?

This behavior has been fixed in #8088 🎊

petercort avatar Sep 23 '24 14:09 petercort