Poetry can't install from private repo inside of tox environment on Gitlab-CI

[dromer]

Description

In our CI we are currently using exported requirements files and pip to install our dependencies inside of tox testing. This works fine, but is annoying to keep synchronized since poetry is managing the project.

Moving to running poetry directly in tox revealed that it's unable to install our internal python package from a private repository. Two strategies were tried (creating a config and using env vars) that seem to work correctly locally, but not in the CI.

I have an example project here: https://github.com/dromer/gitlab_tox_poetry It uses a private python package called libloco. (I've anonymized the repo urls. so this won't be directly functional)

This branch tries to use the CI_JOB_TOKEN: https://github.com/dromer/gitlab_tox_poetry/tree/poetry_job_token

This branch tries to use the POETRY_HTTP_BASIC_ env vars: https://github.com/dromer/gitlab_tox_poetry/tree/poetry_http_basic_env

Any suggestions on what else to try are welcome.

Duplicate ticket on Tox repo (wasn't sure where to post): https://github.com/tox-dev/tox/issues/3599

Workarounds

Export requirements.txt

Poetry Installation Method

pip

Operating System

Debian

Poetry Version

2.1.4

Poetry Configuration

cache-dir = "/home/dreamer/.cache/pypoetry"
data-dir = "/home/dreamer/.local/share/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
python.installation-dir = "{data-dir}/python"  # /home/dreamer/.local/share/pypoetry/python
repositories.mygitlab.url = "https://gitlab-01.mydomain.com/api/v4/projects/411/packages/pypi/simple"
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/dreamer/.cache/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false
warnings.export = false

Python Sysconfig

sysconfig.log

Paste the output of 'python -m sysconfig', over this line.

Example pyproject.toml

Poetry Runtime Logs

poetry-runtime.log

Paste the output of 'poetry -vvv <command>', over this line.

[abn]

@dromer can you maybe provide some more clarity on how the attempt to use poetry within tox is failing? Does it fail to authenticate? Does it succeed but the subsequent runs fail for lack of dependencies?

It could be keyring config related? See documentaiton. You can also see other examples there.

Some sanity-check questions:

1. Does vanilla poetry install work as expected within your CI environment and in your container?
2. Does the tox command work locally inside the container?

[dromer]

tox runs fine, however I now realize it might indeed be a keyring issue, missed the end of this in the logs:

Checking keyring availability: Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Backend 'fail Keyring' is not suitable
No valid keyring backend was found
Unavailable
Installing dependencies from lock file

Unfortunately I wasn't able to figure this out before the end of my contract and I no longer can work on the issue. Have linked the issue to the company ticket and hopefully someone else can pick it up.

[dromer]

Btw I did also try on a different run to set set_env = PYTHON_KEYRING_BACKEND=keyring.backends.fail.Keyring and with poetry config keyring.enabled false it still can't install the private package.

I would need to setup a private gitlab in order to reproduce this, but I'm unlikely to have the time for this at the moment.