Document How To Install Python Package from a Private Repo

[dthomas-sensonix]

I have a number of Python Packages in private (company) repos and I am using GitHub Actions to run pytest on commits. One of the repos depends on packages from other repos. When pip runs from the Action, I see the following error:

Collecting pyconfig@ git+ssh://[email protected]/sxi/pyconfig@master
Running command git clone -q 'ssh://****@github.com/sxi/pyconfig' /tmp/pip-install-wglwufhp/pyconfig
Cloning ssh://****@github.com/sxi/pyconfig (to revision master) to /tmp/pip-install-wglwufhp/pyconfig
Warning: Permanently added the RSA host key for IP address '140.82.114.4' to the list of known hosts.
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@github.com/sxi/pyconfig' /tmp/pip-install-wglwufhp/pyconfig Check the logs for full command output.
##[error]Process completed with exit code 1.

Please document how the user can grant access to private repos to the Action. For example, I solved the problem using the following:

1. For every Python package, create a step to check out the private repo

    - name: Checkout pyconfig from a private repo
      uses: actions/checkout@v2
      with:
        repository: <company>/pyconfig
        token: ${{ secrets.ACCESS_TOKEN }}
        path: pyconfig

2. Modify the “Install dependencies” stop to pip install each Python package sourced from a private repo

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install /home/runner/work/<path>/pyconfig

While this works, it is a little tedious. Tech Support suggested I use HTTPS with a username and password to check out the packages from the private repositories. I would prefer to not use this method. It would likely require me to create and maintain a "fake" user account just for checking repositories in GitHub Actions. I would much prefer to use a personal access token (like I did above), but in a more simplified manner.

[ritamganguli]

I want to work on #104

[ctoth]

Is there a canonical way to do this now in 2022?

[SJCaldwell]

I would also love to know if there was a canonical way to do this in 2022.

[ctoth]

So far the best way I've found to do this is by creating a Machine User and inviting it to the organization.

Then, in requirements.txt you can add lines like:

git+ssh://[email protected]/<org>/<package>.git

Generate and add an SSH key for the machine user, then use something like

shimataro/ssh-key-action

to set the key in your workflow from a secret.

There is more information at Using organization Python package in Github actions without Python repository although this only seems to work for one package and so that's why I ended up having to use the machine user approach as usually if you depend on one dependency in an organization you will depend on others.

Hope somebody finds this useful.

[victorsevero]

Not really canonical but this workaround might be a little better:

Just add git config --global url."https://${{ secrets.GH_TOKEN }}@github".insteadOf https://github at the beggining of your run step (before installing dependencies).

[fuji44]

I tackled the same issue a long time ago. My conclusion at the time was much the same as the one described by croth.

Then I learned that GitHub recommends authentication via HTTPS, and I felt the need to reconsider this issue, but I neglected it.

Recently, I had to do some maintenance on an old project, so I thought about my own best practices at this point. It is a combination of the methods described by croth and victorsevero.

In other words, do the following.

• Define requirements.txt and pyproject.toml with the https schema.
• Add the following steps at the beginning of the CI
  • Register private key with ssh-agent.
  • Add URL rewrite setting to Git config
    • ex. git config --global url."ssh://[email protected]/".insteadOf https://github.com/

I checked and found that for GitHub Actions, this can be easily accomplished by using webfactory/ssh-agent. The README is very carefully written, so I recommend reading it.

[tyriis]

I have the following up and running successful:

    steps:
      # https://github.com/actions/checkout
      - name: checkout
        uses: actions/[email protected]

      # https://github.com/marketplace/actions/setup-python
      - name: Setup Python
        uses: actions/[email protected]
        with:
          python-version: "3.10"

      - name: replace requirements
        run: sed -i "s/ssh:\/\/git@github\.com/https:\/\/${{ secrets.GH_USERNAME }}:${{ secrets.GH_PAT_WITH_ACCESS_TO_OTHER_REPOS }}@github\.com/g" requirements.txt

      - name: install
        run: pip install -e . && pip install -r requirements.txt

      - name: test
        run: python test.py

will try out another variant with a gh app token.

[blieusong]

Same issue, and had to resort to @victorsevero (and others') solution

[aripollak]

There's another option now that I believe is a bit more secure than managing a separate machine user and its associated credentials: Create a new Github App with "Contents" permissions, add it to the private repositories that the Github Action needs to access, and use something like https://github.com/marketplace/actions/action-github-app-token to get an access token that can be used during git checkout. Unfortunately this still requires the app's private key to be saved as a secret for the Action.

[kenichi]

@aripollak this solution works Pretty Good as of today, and has a nice restrictive access element to it. Thank you 🙇🏽‍♂️

[Aylr]

@tyriis Thank you for this elegant dependency-free solution!

[pmabres]

@aripollak How did you manage to successfully use a Github App token with pip? I can use the private_key as a way of configuring the checkout action, but unsure how to re-use the private key to let pip authenticate and clone dependencies. I figured I could use the private key and setup an ssh agent https://github.com/marketplace/actions/webfactory-ssh-agent using this action, but this does not work. I might have to check how the checkout action does it and go from there

[pmabres]

So I got it all wrong on my previous comment. If you are authenticating with a Github App, you can follow these steps to create a new one:

https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app

Then, once your app has been created and added to the required repos (or if you add it to the top level of your organization, you can select which repos the app has access to), you can use create-github-app-token action to generate an installation token that can access multiple repositories.

The only issue is this token can be used to sign-in using https. We can override global urls with git config --global url as stated in the comments above by @victorsevero and @fuji44. A similar approach is followed here: https://github.com/actions/checkout/blob/main/src/git-auth-helper.ts

In my case I was using ssh in my requirements.txt file, so these also get overriden by the global git configuration.

    - name: Get our app token from Github App
      uses: actions/create-github-app-token@v1
      id: app_token
      with:
        app-id: ${{ secrets.APP_ID }}
        private-key: ${{ secrets.APP_PRIVATE_KEY }}
        # owner is required, otherwise the creds will fail the checkout step
        owner: ${{ github.repository_owner }}
    - name: Checkout from GitHub
      uses: actions/[email protected]
      with:
        submodules: true # In my case I'm also using this token to clone sub-modules
        token: ${{ steps.app_token.outputs.token }}
    # This step is necessary to allow pip install private packages hosted in github.
    - name: Setup token for Python installation
      run: git config --global url."https://oauth2:${GH_TOKEN}@github.com".insteadOf ssh://[email protected] # replace with whatever authentication method you're currently using
      env:
        GH_TOKEN: ${{ steps.app_token.outputs.token }}

Thanks to this person for suggesting this approach: https://github.com/actions/checkout/issues/287#issuecomment-1830291196

more info on github app authentication: https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/making-authenticated-api-requests-with-a-github-app-in-a-github-actions-workflow and generating authentication tokens for an installation (which is what create-github-app-token does for us): https://docs.github.com/en/rest/apps/apps?apiVersion=2022-11-28#create-an-installation-access-token-for-an-app

Remember also to create your APP_ID and PRIVATE_KEY secret variables from your App configuration: Store the App's ID in your repository environment variables (example: APP_ID) Store the App's private key in your repository secrets (example: PRIVATE_KEY)

[mohit2512sharma]

So I got it all wrong on my previous comment. If you are authenticating with a Github App, you can follow these steps to create a new one:

https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app

Then, once your app has been created and added to the required repos (or if you add it to the top level of your organization, you can select which repos the app has access to), you can use create-github-app-token action to generate an installation token that can access multiple repositories.

The only issue is this token can be used to sign-in using https. We can override global urls with git config --global url as stated in the comments above by @victorsevero and @fuji44. A similar approach is followed here: https://github.com/actions/checkout/blob/main/src/git-auth-helper.ts

In my case I was using ssh in my requirements.txt file, so these also get overriden by the global git configuration.

    - name: Get our app token from Github App
      uses: actions/create-github-app-token@v1
      id: app_token
      with:
        app-id: ${{ secrets.APP_ID }}
        private-key: ${{ secrets.APP_PRIVATE_KEY }}
        # owner is required, otherwise the creds will fail the checkout step
        owner: ${{ github.repository_owner }}
    - name: Checkout from GitHub
      uses: actions/[email protected]
      with:
        submodules: true # In my case I'm also using this token to clone sub-modules
        token: ${{ steps.app_token.outputs.token }}
    # This step is necessary to allow pip install private packages hosted in github.
    - name: Setup token for Python installation
      run: git config --global url."https://oauth2:${GH_TOKEN}@github.com".insteadOf ssh://[email protected] # replace with whatever authentication method you're currently using
      env:
        GH_TOKEN: ${{ steps.app_token.outputs.token }}

Thanks to this person for suggesting this approach: actions/checkout#287 (comment)

more info on github app authentication: https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/making-authenticated-api-requests-with-a-github-app-in-a-github-actions-workflow and generating authentication tokens for an installation (which is what create-github-app-token does for us): https://docs.github.com/en/rest/apps/apps?apiVersion=2022-11-28#create-an-installation-access-token-for-an-app

Remember also to create your APP_ID and PRIVATE_KEY secret variables from your App configuration: Store the App's ID in your repository environment variables (example: APP_ID) Store the App's private key in your repository secrets (example: PRIVATE_KEY)

Does this method work in an organization setting? I have two private repositories, one is a python package and other one is where I want to install. I did the things mentioned, but it fails. Could it be because the repositories are in organization? I get the following error:

[email protected]: Permission denied (publickey).
  fatal: Could not read from remote repository.

[pmabres]

@mohit2512sharma . This works in an organization and it is what we're currently using. Make sure your GitHub app has permissions to access your organization repositories. Also make sure that you're referencing your GitHub repos in your requirements as ssh so that they get replaced correctly by the git global configuration

[mohit2512sharma]

@mohit2512sharma . This works in an organization and it is what we're currently using. Make sure your GitHub app has permissions to access your organization repositories. Also make sure that you're referencing your GitHub repos in your requirements as ssh so that they get replaced correctly by the git global configuration

Thanks for your comment. @pmabres . Your comment does work, unfortunately I had a typo in my requirements.txt and for hours I couldn't find it.