[Docs] Install from private git repository with token

[tom-mcclintock]

Feature Request

Relates to #835. Other recent, potentially useful conversations include e.g. #6629.

TLDR - I would like to be able to poetry install an environment that includes a package found in a private github repo from within a github action in another private repository (e.g. CI), and use a token instead of a user/pw for credentials. I was not able to find an example of this in the docs.

Example - I have a local project and in my pyproject.toml I have a private dependency that looks like this:

my-private-package = {git = "[email protected]:my-org/my-private-package.git"}

Since I have SSH credentials locally, I can poetry install and obtain this package no problem.

However, I found the poetry docs lacking in examples for how to do this from a location that does not have SSH credentials. In my case, I am trying to install my-private-package from within a github action of some other project.

This comment from #835 suggests using a token to obtain permissions at the poetry add step, but does not solve my issue, because my-private-repo is already in the pyproject.toml file in question (so poetry add isn't relevant).

I ended up not using poetry to install the private code (I stuck it in an optional group in pyproject.toml), and instead used pip within the venv:

      - name: Activate poetry venv and install private package
        run:  |
          source .venv/bin/activate
          pip install git+https://my-org:${{ secrets.SOME_TOKEN}}@github.com/my-org/my-private-package.git

This is not ideal, because it means my local install pattern is different than the pattern used in the action.

I'd like to request this example be formalized and added to the poetry docs, since it seems from the discussions in #835 that it should be possible, but I don't have enough github acumen to figure out how.

Big thanks!

[madeinoz67]

have this same issue and came across this. Thanks have had to use this non-ideal workaround.

I have also attempted ad ding a secondary repository however am seeing the packat appended to the URL and hence can't find it.

A possible solution is using an existing stored keyring credential similar to that of how secondary repository credentials are stored, and maybe a definition of private-package rather than a private-repository to differentiate between the two?

[elliott-king]

Another workaround is to have a git SSH key dedicated to the task. Then you can do something awkward in your CI container like:

mkdir /root/.ssh && \
echo "$GITHUB_SSH_PRIVATE_KEY" > /root/.ssh/id_rsa && \
chmod 600 /root/.ssh/id_rsa && \
echo "Host *" > ~/.ssh/config && echo " StrictHostKeyChecking no" >> ~/.ssh/config && \
poetry install && \
rm -rf /root/.ssh/

EDIT: since it's all one line, the ssh key shouldn't end up in your image layers. Feel free to correct me.

[alex-linx]

We are also facing this issue at our organizations, any solution to allow using poetry install with the github token from github actions?

[wt-asw]

We have also bumped into this, is there a timeline for this feature to be added?

[alex-linx]

A workaround I found for this is to set the url like this:

REPO_URL="https://oauth2:${{ inputs.github_token }}@github.com/${GITHUB_REPOSITORY}.git"

I know this works for github, not sure about others.

I had a github action that would patch the pyproject.toml file replacing the url, before building the docker:

REPO_URL="https://oauth2:${{ inputs.github_token }}@github.com/${GITHUB_REPOSITORY}.git"
sed -i "s#my-package = .*#my-package = { git = \"$REPO_URL\" }#g" pyproject.toml

However this has a drawback that the token is now baked into the image.

In my case I added a command inside the dockerfile that deleted the files that contained it:

        rm ./pyproject.toml && \
        rm -f /venv/.venv/lib/python3.9/site-packages/my-package-*.dist-info/direct_url.json; \

But this means you can't run poetry install again in the image after the initial install.

[Scrashdown]

We have also bumped into this issue in my organization.

In our case, we also need to specify a specific git revision of the package to download. It looks like it is already possible with poetry add git+<repo url>#<commit sha>. So to summarize it would be nice if one could pass an auth token and a commit SHA to specify which version to download.

[nitsujri]

Another workaround is to have a git SSH key dedicated to the task. Then you can do something awkward in your CI container like:

mkdir /root/.ssh && \
echo "$GITHUB_SSH_PRIVATE_KEY" > /root/.ssh/id_rsa && \
chmod 600 /root/.ssh/id_rsa && \
echo "Host *" > ~/.ssh/config && echo " StrictHostKeyChecking no" >> ~/.ssh/config && \
poetry install && \
rm -rf /root/.ssh/

EDIT: since it's all one line, the ssh key shouldn't end up in your image layers. Feel free to correct me.

A better work-around option today to loading SSH key with Docker is secrets.

RUN --mount=type=secret,id=gh_priv_key,target=/root/.ssh/id_rsa \
    /root/.local/bin/poetry install --without dev --no-root

docker build --secret id=gh_priv_key,src=path/to/private_key .

Purpose built to not show up in image layers nor log outputs.

[codethief]

To access private git repositories through HTTPS (as opposed to SSH) I arrived at this solution:

export GIT_USER="user"  # Don't do this in production
export GIT_PASS="pass"  # Don't do this in production
git config --global credential.helper '!echo "username=${GIT_USER}"; echo "password=${GIT_PASS}";'

poetry install --no-root

pyproject.toml contains something like

[tool.poetry.dependencies]
# …
my-private-package = {git = "https://gitlab.com/path/to/my/private/repo.git" }

Background/references:

• https://github.com/orgs/python-poetry/discussions/3794
• https://git-scm.com/docs/gitcredentials
• https://stackoverflow.com/questions/8536732/can-i-hold-git-credentials-in-environment-variables

[elliott-king]

To access private git repositories through HTTPS (as opposed to SSH) I arrived at this solution:

export GIT_USER="user"
export GIT_PASS="pass"
git config --global credential.helper '!echo "username=${GIT_USER}"; echo "password=${GIT_PASS}";'

poetry install --no-root

FYI if using Docker, this will save your git username & pwd in your docker image layers, which are not private. I do not recommend this.

This issue is related to CI/CD, not just Docker, so the above user's script will depend on your use case.

[codethief]

@elliott-king Maybe I should have been clear about the role of my code snippet above: It was merely supposed to demonstrate how to get poetry to use credentials that are present in the environment. It wasn't meant as something you copy verbatim into your Dockerfile or anything.

More specifically, in Docker you would inject the git credentials using docker build --secret=… (docs), in Kaniko you would store the credentials in some file in special /kaniko folder. Either way, you can then source or cat that file in your script-that-runs-poetry-install.sh which, in turn, you would RUN in your Dockerfile.

In fact, in the case of Docker, BuildKit provides a shortcut to avoid the boilerplate script-that-runs-poetry-install.sh wrapper.

[abn]

Using http auth for git repositories is described in https://python-poetry.org/docs/dependency-specification/#git-dependencies with examples.

If you want to use git-credentials you will have to use the system git client as described in the doc. But do this only if you really need it.

If there is a specific question around this please feel free to raise a discussion or ask on our discord server.

[codethief]

@abn The suggestion from the page you linked is to use poetry config http-basic.git-org-project username token. However, this is a rather bad solution from a security POV because it will persist the credentials in the Poetry config and hence, if we dockerize our application, in the image layer. In Docker you typically want to use build secrets (see my previous comment), i.e. environment variables that don't get persisted.

In other works: If authentication only works by persisting credentials in files, it tends to be a security risk. There are hacks to work around the persistence but they are easy to mess up.

[abn]

@codethief thank you for sharing you POV.

Please see https://python-poetry.org/docs/repositories#configuring-credentials for information on how you can use environment variables or in your case build args.

For a desktop environments (and I guess in container environments were secret storage is enabled) use case Poetry uses keyring to store credentials into your secret store rather than persisting it on file.

Furthermore, you could potentially also rely on multi stage container buidls to only have our virtualenv and application persisted in your runtime container build.

[codethief]

Please see https://python-poetry.org/docs/repositories#configuring-credentials for information on how you can use environment variables or in your case build args.

Thanks, I remember trying export POETRY_PYPI_TOKEN_FOO=… a while ago and it turned out to not be an option for us but I don't remember why.

Furthermore, you could potentially also rely on multi stage container buidls to only have our virtualenv and application persisted in your runtime container build.

True. Unfortunately, in our case, we make heavy use of remote caching, so all image layers (even of intermediate stages) get pushed to the container registry and we don't want to have credentials lying around there.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.