Ubuntu 22.04, Python 3.10, and broken virtual environment layouts

[colindean]

• [X] I am on the latest Poetry version.

• [X] I have searched the issues of this repo and believe that this is not a duplicate.

• [X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu Jammy

• Poetry version: 1.2.0

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/colindean/27d36dcc7cf78f722f374ebd5bfd970f (also contains log output and a brief description of env setup steps)

Issue

root@80d0e404badc:/what# poetry self add poetry-plugin-bundle
Using version ^1.0.0 for poetry-plugin-bundle

Updating dependencies
Resolving dependencies... (3.8s)

[Errno 2] No such file or directory: 'python'

I believe the problem is in here:

https://github.com/python-poetry/poetry/blob/f1af3f8eec26112b7271c4d3c602957d08cff3a9/src/poetry/utils/env.py#L1231-L1248

Ubuntu doesn't provide a python command anymore, just python3:

root@80d0e404badc:/what# python --version
bash: python: command not found
root@80d0e404badc:/what# python3 --version
Python 3.10.4

Seeing as though all supported versions of Python on which Poetry runs should also provide a python3 executable, perhaps the fallback here should be python3?

https://github.com/python-poetry/poetry/blob/f1af3f8eec26112b7271c4d3c602957d08cff3a9/src/poetry/utils/env.py#L1178

[colindean]

N.b. I hadn't even run poetry install yet and just now went I did…

root@80d0e404badc:/what# poetry install
Creating virtualenv what--AkdOZe_-py3.10 in /root/.cache/pypoetry/virtualenvs

[Errno 2] No such file or directory: 'python'

[colindean]

I appear to be unable to tell poetry which Python to use, either:

root@80d0e404badc:/what# poetry env use $(which python3)

[Errno 2] No such file or directory: 'python'

[neersighted]

I think this is actually the old DEB_PYTHON_INSTALL_LAYOUT=deb issue -- can you set that in the environment and reinstall Poetry/poetry install from scratch? python will always exist as far as Poetry is concerned as it is called for by the virtual environment spec.

[colindean]

I used that envvar at Poetry installation time… see repro here. Does it need to be set all of the time, as in, added to the user's shellrc?

I can confirm that changing https://github.com/python-poetry/poetry/blob/f1af3f8eec26112b7271c4d3c602957d08cff3a9/src/poetry/utils/env.py#L1178 to python3 restores expected functionality for poetry install and poetry env use … but not for poetry self add ….

[colindean]

root@80d0e404badc:/what# export DEB_PYTHON_INSTALL_LAYOUT=deb
root@80d0e404badc:/what# poetry self add poetry-plugin-bundle
Using version ^1.0.0 for poetry-plugin-bundle

Updating dependencies
Resolving dependencies... (16.5s)

[Errno 2] No such file or directory: 'python'

[neersighted]

We're still trying to fully understand how we can work around the regression in Debian/Ubuntu as they don't appear willing to walk back the changes. I am not sure why you're having that issue with self add, but I can confirm for you that on the ubuntu:22.04 container with the python3 package I can install Poetry, run the Poetry test suite (after poetry install), and poetry self add without issue if I set DEB_PYTHON_INSTALL_LAYOUT=deb before running any Python.

It is important to note that the issue is not in Poetry, but is in the virutualenv package which we consume. It is still considered unsolved there, as the changes made by Debian/Ubuntu result in an invalid virtual environment according to the spec: https://github.com/pypa/virtualenv/issues/2350

[colindean]

OK. To confirm, does DEB_PYTHON_INSTALL_LAYOUT=deb need to stay forever, then? As in, rather that just needing for it to be set at installation time, it should be set the user's shell rc or profile?

[neersighted]

From a practical standpoint, it's the known workaround right now. Fundamentally this is unsolved and upstream seems intransigent or indifferent (see: https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1967920).

The deadsnakes PPA solves this with a patch on top of the Debian 3.10 patches. Ideally this would get solved in virtualenv, but things are stalled there too. We haven't put too much effort into figuring out how to work around it in Poetry (but the solution definitely is NOT just adapting to local/bin as that is a violation of the virtual env spec and other tools will break), if that is even possible without somehow trying to monkey-patch the virtualenv module.

The best solution is to use a Python.org Python (aka a build from source installed into /opt/ or using python-build, or pyenv or a python docker hub image, or the like), or to use a deadsnakes build, until some upstream solves this, or we figure out a not-awful way to work around it here in the downstream.

[colindean]

Thanks for the expansion. I understand now. I encountered this problem while trying Poetry 1.2.0 in a container to test how installation of plugins would work (globally at Poetry level or per-project), as I eventually went to Discord to discuss!

Thanks for your time on this, I really appreciate it.

[colindean]

I should add that my team's normal way of managing our Pythons is through pyenv for our workstations and conda for our production containers. So, this issue isn't a major blocker for us!

[dimbleby]

ubuntu provides the python-is-python3 package, you could install that.

(I'm not asserting that what poetry is doing is either right or wrong, just noting that this is a likely workaround)

[neersighted]

I don't think that's what is going on here @dimbleby -- some of the issues I linked here are possibly related, but that's more end-user error than anything. This issue is tracking a broken virtualenv layout (and the 'missing python' is caused by python being at local/bin/python in the venv, in violation of the standard layout).

[dimbleby]

maybe there's more than one thing mixed up in this thread, I got this far:

Ubuntu doesn't provide a python command anymore, just python3:

to which the above is certainly an answer

[neersighted]

Fair -- I'll likely hide all comments related to that as off-topic as they're irrelevant troubleshooting before the real problem was defined.

[garar]

Hi!

Is there a workaround to this issue? I've tried using DEB_PYTHON_INSTALL_LAYOUT=deb variable when using poetry add but it seem that it doesn't work. Is there some other way of using DEB_PYTHON_INSTALL_LAYOUT=deb?

Thanks!

[neersighted]

You should ideally set it before ever installing Poetry -- however, I am reasonably sure that setting it during env creation (e.g. poetry env use, poetry add without an existing env, etc) and during subsequent poetry install/poetry add/poetry update) should be sufficient. I do not work regularly with a Ubuntu/Debian distro, so I am guessing to an extent (I just export it in .bashrc when it comes up).

[AdamWRichardson]

I also ran into this problem and didn't know about the DEB_PYTHON_INSTALL_LAYOUT=deb workaround so will try that now. On my digging I may have found some more useful information as I too thought it was a problem with the way poetry handles the system python but the trace suggests otherwise.

The first "poetry error" (by which I don't mean that poetry is doing anything wrong) in the trace comes from the env_manager.create_venv() command which, as the name suggests, creates a new venv using python3-venv. When I have a look at this new venv it contains <venv_path>/share and <venv_path>/lib but no <venv_path>/bin which should contain the python executables as well as the python symlink.

This seems very weird because when I create the venv myself the bin directory does exist so I'm not sure what's happening internally for this bin directory to not get made.

Like I say I hope this is helpful to others and helps any debugging/patching effort either here or in Ubuntu upstream. Will try the environment variable setup now

[yrro]

I found that installing Debian's python3-poetry results in a perfectly working Poetry. Looking at https://sources.debian.org/src/poetry/1.3.2%2Bdfsg-3/debian/patches/ I don't believe this is because they've patched Poetry themselves.

[gaby]

I have been trying for weeks to get Poetry working with Ubuntu 22.04:

Tried:

• pip3 install poetry
• apt install python3-poetry
• curl -sSL https://install.python-poetry.org | python3 -

Running anything related to poetry simply fails, this is a brand new Ubuntu 22.04 VM

[n8jhj]

Had this issue as well inside VSCode's integrated terminal. It seems VSCode does some kind of venv activation automatically when starting up the integrated terminal, which you can get out of with the deactivate command. However, it seems this is not Poetry's "preferred" method of venv usage, which is through either poetry run ... or poetry shell. In the latter case, you can get out with exit. Not sure what VSCode is doing there, but probably good to be aware of if you're using it.

The end result in my case was that removing Poetry (1.6.1) and then reinstalling Poetry (1.7.1) with the install script solved my issue. After I had done so, however, I noticed I was still in the venv in VSCode's integrated terminal. I wonder what would have happened if I had used deactivate and then tried poetry env use again, which is where I was seeing this bug manifest.