python-poetry
Poetry doesn't respect private repository's branch/rev in pyproject.toml
Description
We are currently having a problem how poetry uses the private github repository's branch naming for cloning.
All the following issues are present when trying to install from a docker image, locally it works.
We use a multi-repository style structuring for our codebase and several repositories depend on each other:
utils, database and image-processing
database depends on utils
image-processing depends on database and utils
The issue occured when we tried poetry install from within the image-processing repository.
We have two branches in database: main and develop
The only difference between the two is that while main uses the utils repository as a submodule and a local path:
[tool.poetry.dependencies]
utils = { path = "./utils"} # local path for submodule
develop uses the github URL with branch name:
[tool.poetry.dependencies]
utils = { git = "[email protected]:greehill/utils.git", branch = "main" } # could be develop, but it doesn't matter
When the poetry install command is called from image-processing's docker build command, we get some logs indicating where the problem happened (see Poetry Runtime Logs)
It shows that it tries to run:
git clone --recurse-submodules -- REPOSITORY VENV_LOCATION
but it doesn't specify the branch name which was provided in the pyproject.toml.
Because of this the git clone command will have to provide a username/password which could not be fulfilled due to being in an automated docker build.
We also tried providing a github personal access token by injecting it into the docker build as a secret and use the git insteadof in the git config to replace the url to use this token but it still failed because it tried to clone the submodules which required the username/password authentication for some reason.
Proposed solution would be to use the git clone command's -b BRANCH_NAME and preferably the --single-branch options.
Workarounds
I don't know of any workarounds
Poetry Installation Method
install.python-poetry.org
Operating System
Ubuntu 20.04.6 LTS
Poetry Version
Poetry (version 1.8.3)
Poetry Configuration
cache-dir = "/home/gabor/.cache/pypoetry"
certificates.project.cert = false
experimental.system-git-client = true
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs" # /home/gabor/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true
Python Sysconfig
No response
Example pyproject.toml
No response
Poetry Runtime Logs
#11 15.30 CalledProcessError
#11 15.30
#11 15.30 Command '['git', 'clone', '--recurse-submodules', '--', '[email protected]:greehill/database.git', '/root/.cache/pypoetry/virtualenvs/image-processing-9TtSrW0h-py3.12/src/database']' returned non-zero exit status 1.
are you claiming that cloning the whole repository requires different credentials than cloning the branch? this surprises me.
#6348 looks like it was an attempt at what you are asking for but apparently too simplistic. Perhaps that pull request and the comments on it will point you in the right direction if you would like to contribute.
Anyway this issue is basically duplicate #2412, albeit with a new reason. Please leave a comment there describing your new use case, and close this one out
@dimbleby Thank you, you are right, it's a strange thing that happened on our side, it took us a good few days until we located where the issue originates from. We probably could just update both branches to use the github url with the token injection and the issue will go away, now that I think about it.
Does poetry clone the default branch and then checkout the branch/rev/tag defined in the pyproject.toml?