[Feature][Github] Refresh token for GitHub Apps authentication

[marcelofabricanti]

Search before asking

• [x] I had searched in the issues and found no similar feature requirement.

Use case

The goal is to connect DevLake to GitHub via a GitHub App Token, as this is the recommended way by GitHub.

About authentication with a GitHub App

Description

Currently, the GitHub App token expires every hour and stops collecting data.

The Feature Request is to renew the GitHub App token using the refresh_token obtained in token generation.

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[marcellodesales]

Is there a PR already? This should be just using the https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/refreshing-user-access-tokens ...

[ysinghc]

@marcellodesales I cant see any open PRs for this issue so can i work on this?

[klesh]

@ysinghc I think it is OK because @marcellodesales didn't select the "Yes I am willing to submit a PR!" checkbox. Just assigned the issue to you, happy coding.

[ysinghc]

@klesh I will start understanding the codebase and come up the proposed solution for this update

[ysinghc]

@klesh I will implement a token refresh mechanism by :

Updating the connection model: I will modify the GithubConnection data model to store two new fields obtained from the initial OAuth exchange: the refreshToken and its expiry timestamp.

Centralizing the token management: I will introduce a token management layer or service within the GitHub plugin. This service will be responsible for providing a valid access token to any data collection task.

Implement a "GetAccessToken" Function: Instead of tasks accessing the token directly from the connection model, they will call a new function. The logic for this function will be as follows:

a. Check if the current access_token is still valid b. If it is valid, return it c. If it is expired check if the refresh_token is still valid d. If the refresh token is valid use it to make a POST request to github's token endpoint to get a new access_token and a new refresh_token e. Update the GithubConnection record in the database with these new tokens and their expiry dates f. Return the new access_token g. If the refresh token is also expired then the process fails and the user must re-authenticate

Integrate into API client creation: I will refactor the code that creates the authenticated GitHub API client. Before a data collection task runs this creation logic will now use the getaccesstoken function to ensure it is initialized with a valid token refreshing it automatically if necessary.

[klesh]

Thank you for sharing your plan in advance; that's a wise step.

Your plan looks solid overall. However, I'm curious about the potential scenario where a token might expire during the collection process. We've received reports indicating that if a collection run exceeds an hour, the token can expire, leading to the failure of the entire pipeline.

[ysinghc]

I can use a dynamic token provider that proactively refreshes the token shortly before it expires.

Also, I am very hesitant on publishing any commits until i am able to test my code so can you please guide me a little as to how to setup a testing bench for devlake, i have tried reading the documentaiton and using docker, but it just takes forever and never finishes building.

[klesh]

I can use a dynamic token provider that proactively refreshes the token shortly before it expires.

Also, I am very hesitant on publishing any commits until i am able to test my code so can you please guide me a little as to how to setup a testing bench for devlake, i have tried reading the documentaiton and using docker, but it just takes forever and never finishes building.

Good question. I use VSCode, here are my configurations to launch the backend for debugging:

tasks.json for building plugins

{
	"version": "2.0.0",
	"tasks": [
		{
			"label": "debugPlugins",
			"type": "shell",
			"command": "${workspaceFolder}/backend/scripts/build-plugins.sh",
			"options": {
				"env": {
					"PKG_CONFIG_PATH": "/usr/local/lib/pkgconfig",
					// "DEVLAKE_PLUGINS": "github,github_graphql,org,refdiff,dora,gitextractor",
					// "DEVLAKE_PLUGINS": "github,gitlab,org,refdiff,dora,gitextractor",
					// "DEVLAKE_PLUGINS": "jira",
					// "DEVLAKE_PLUGINS": "tapd",
					"DEVLAKE_PLUGINS": "zentao",
					// "DEVLAKE_PLUGINS": "pagerduty",
					"DEVLAKE_DEBUG": "1",
					"PLUGIN_DIR": "${workspaceFolder}/backend/bin/plugins"
				}
			}
		}
      ]
}

launch.json

{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Backend",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "backend/server/main.go",
      "cwd": "backend",
      "env": {
        "FORCE_MIGRATION": "true",
        "VIRTUAL_ENV": "${workspaceFolder}/backend/python/venv",
        "DISABLED_REMOTE_PLUGINS": "true",
        "LD_LIBRARY_PATH": "/usr/local/lib",
        "PLUGIN_DIR": "${workspaceFolder}/backend/bin/plugins"
      },
      "preLaunchTask": "debugPlugins"
    }
  ]
}

[cedriclecoz]

is there a PR or something which I could try to pull in my environment to test that? current PR above looks empty and closed. I am trying to add check runs and status checks but first collection on some of my repos is quite big and can take multiple hours so it keeps failing with Github App. I've switched my Connection to use a PAT for now to try to collect everything, but it takes way longer due to restricted set of api credits ^^

[ysinghc]

One of the checks failed during my last PR, so rn i am in the process of fixing that. I will try to have a pr you can use by the end of this week.

[klesh]

One of the checks failed during my last PR, so rn i am in the process of fixing that. I will try to have a pr you can use by the end of this week.

Thanks for following up