poetry
poetry copied to clipboard
python-poetry
Poetry with `virtualenvs.create false` installs packages into the wrong place on Ubuntu
- [x] I am on the latest Poetry version.
- [x] I have searched the issues of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).
- OS version and name: Ubuntu 20.04
- Poetry version: 1.2.0
- Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/amcinnes/8c61ad88b37a7740dd93efa976f85ed7
Issue
To reproduce the issue:
docker run --rm -it ubuntu:20.04@sha256:af5efa9c28de78b754777af9b4d850112cad01899a5d37d2617bb94dc63a49aa
apt-get update && apt-get -y install python3-pip
python3 -m pip install poetry==1.2.0
poetry config virtualenvs.create false
poetry init -n
poetry add awscli
aws --version
Expected behaviour (seen with poetry 1.1.14): the aws command runs successfully.
# aws --version
aws-cli/1.25.70 Python/3.8.10 Linux/5.19.7-arch1-1 botocore/1.27.69
Observed behaviour (with poetry 1.2.0): the aws command fails.
# aws --version
Traceback (most recent call last):
File "/usr/bin/aws", line 19, in <module>
import awscli.clidriver
ModuleNotFoundError: No module named 'awscli'
This seems to be because with poetry 1.2.0, awscli has been installed into /usr/lib/python3.8/site-packages/awscli which is not in the Python path.
root@7f452a8de1d3:/# python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
>>> import sys
>>> print(sys.path)
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/root/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
>>> import sysconfig
>>> print(sysconfig.get_paths())
{'stdlib': '/usr/lib/python3.8', 'platstdlib': '/usr/lib/python3.8', 'purelib': '/usr/lib/python3.8/site-packages', 'platlib': '/usr/lib/python3.8/site-packages', 'include': '/usr/include/python3.8', 'platinclude': '/usr/include/python3.8', 'scripts': '/usr/bin', 'data': '/usr'}
Looks like this is the old Debian patching of distutils without patching sysconfig acting up again, combined with our apparent failure to make use of our modified detection code:
https://github.com/python-poetry/poetry/blob/c4b2253793cd6b41a99e25e479e40b776cca0a0e/src/poetry/utils/env.py#L193-L222
If I had to guess, it's because we always treat the target environment as a VirtualEnv instead of a GenericEnv and thus don't invoke that script.
Honestly, this is such an edge case (and mostly the result of bad Debian packaging) that I'm not sure there's much value in fixing it -- it's easily resolved if you make use of a self-compiled Python or virtual environments (there is no reason to avoid them in a container, and doing so exposes you to all sorts of sharp edges like this).
This feels like another instance of #6398 -- I would highly suggest looking at the pattern here https://github.com/python-poetry/poetry/issues/6397#issuecomment-1236327500 for something that is as ergonomic, but unlikely to be broken by distro packaging decisions.
Some more reading on this issue:
https://github.com/mesonbuild/meson/issues/8739 https://github.com/gramineproject/graphene/blob/6ba10c60b94a066d516bb9121c31280c0ba69468/Scripts/get-python-platlib.py https://sources.debian.org/src/python3.7/3.7.3-2+deb10u3/debian/patches/distutils-install-layout.diff/
I also suppose I'll ping @abn as historically he's the one interested in trying to paper over these bad decisions.
This feels like another instance of https://github.com/python-poetry/poetry/issues/6398
Thanks @neersighted - this is indeed an instance of that. The context here from our perspective is that the OS package for awscli is quite old and lagging, which admittedly is not your concern at all.
But, that led to us wanting to manage certain OS level dependencies via python packaging rather than OS level packaging. At the time that seemed like a reasonable thing to do.
We were originally just doing something simple like python3 -m pip install awscli==1.2.3 or whatever. This came with it's own set of problems. So then we wanted to use better python dependency management tools so that all dependencies were locked/manageable rather than just the top level one, which had an added benefit of people able to use other dependency management tools such as renovate.
I hope that adds some context around situations where managing OS level dependencies via a python dependency management tool might be desirable. Perhaps the path forward for us is to manipulate the default $PATH rather than try and shoehorn global executables into /usr/loca/bin/?
Perhaps the path forward for us is to manipulate the default $PATH
a sensible pattern is, pipx-style, to install awscli or whatever into a virtual environment but then put a symlink in your path pointing at the executable in that environment.
I'd definitely say that this is one of the rare cases where 'I know what I'm doing and the global environment is the best solution' is the case... Certainly, MRs to fix it are welcome since this is functionality we should have -- it's just functionality that only comes up during installs to the global environment on 4+ year old Debian releases :laughing:
If anyone would like advice on working on this, please feel free to reach out -- otherwise I'll try to look at it, but the timeline on that may be a while.
That being said, this is a bit of a square peg round hole situation, and ultimately pipx or something else probably is a more natural fit.
I'm running into this with a simpler setup again with virtualenvs = false. Dependencies aren't being installed into the right location, so the system python can't find them.
root@3d366f729e32:/app# python -m site
sys.path = [
'/app',
'/usr/lib/python38.zip',
'/usr/lib/python3.8',
'/usr/lib/python3.8/lib-dynload',
'/root/.local/lib/python3.8/site-packages',
'/usr/local/lib/python3.8/dist-packages',
'/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (exists)
USER_SITE: '/root/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True
# (after a poetry install including flask)
root@851269ec027a:/app# find / | grep flask
/usr/lib/python3.8/site-packages/flask
...
I guess for the time being we will start using venvs
@Mattwmaster58 Please report what OS version and Python package version you are encountering this with.
Facing this with python 3.8 on Ubuntu 20.04. on docker image mcr.microsoft.com/playwright/python:v1.24.0-focal
I had to replace to pip install poetry==1.1.14 to get this working expected. but with current poetry version the issue persist.
@neersighted @manojatlas is my exact case where I'm seeing the problem. I can share a minimum docker file if you'd like
@neersighted @manojatlas is my exact case where I'm seeing the problem. I can share a minimum docker file if you'd like
Does that mean you're on the same image (or at least OS version)?
Okay, thanks for the info -- in the mean time, is there a reason that you cannot use virtual environments (like @amcinnes)? There are myriad reasons to prefer them, from being insulated from the packaging choices of the distro, to avoiding mixing unexpected Python code into your environment, to being able to copy them across multi-stage container builds.
No, not any substantial reason. We ended up that it to get around this bug.
The reason I wanted to disable virtual environment was that inside the docker image Playwright is install in system environment and I use poetry then the dependencies will be installed in the virtual environment. so it wont work as cant find packages.
So for this I had to disable the virtual environment and install all dependencies in the system.
@Mattwmaster58 : what alternative way you've used? also if you can share your minimum docker file would be great.
@manojatlas nothing special really, I can share if you want.
Playwright images don't have client libraries preinstalled, only necessary libs and binaries to run the browsers it supports. Installing in a venv does not prevent you from using those preinstalled browsers.
Maybe out of topic question: ok in that case I have to activate virtual environment when I want to execute tests?
Yes, or use poetry run
On Mon, Sep 12, 2022, 10:30 PM manojatlas @.***> wrote:
ok in that case I have to activate virtual environment when I want to execute tests?
— Reply to this email directly, view it on GitHub https://github.com/python-poetry/poetry/issues/6459#issuecomment-1244883223, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGI56LIXAADXU5BVLHBOPB3V577OBANCNFSM6AAAAAAQILJ47U . You are receiving this because you were mentioned.Message ID: @.***>
I was trying to find a bug report opened against Ubuntu describing this behavior. I think it's this one, but I'd appreciate it if someone else can confirm: https://bugs.launchpad.net/ubuntu/+source/python3-defaults/+bug/1408092
The biggest reason I'm doubting myself is that this one has been open since 2015, but no apparent progress.
We do have code to work around this, but it is just not being triggered properly -- so there is a Poetry bug, but Poetry would not have to do anything special if the distro patches were applied properly/consistently (trying to get this information out of distutils is fraught).
Honestly, this is such an edge case (and mostly the result of bad Debian packaging) that I'm not sure there's much value in fixing it -- it's easily resolved if you make use of a self-compiled Python or virtual environments (there is no reason to avoid them in a container, and doing so exposes you to all sorts of sharp edges like this).
I see that the conversation has moved on quite a bit since this comment about the bug being an edge case, but I just wanted to push back on it a little bit: not only are Ubuntu and Debian a considerable portion of the total Linux space on their own, but also the official Python docker containers are built from Debian bullseye and buster (unless we want to use alpine). I would think poetry would hope to work out of the box with the official Python images.
Edit: not to say that it isn't annoying that this issue has persisted for so long in those distros, but it really does seem like something poetry needs to be resilient to.
The official Python images are just fine as they use a Python.org build of Python and not a patched/hacked distro-supplied Python. The edge case is mostly installing to system site-packages using a Debian-derived Python, which is definitely a thing people do, but is often ill-advised and will always be prone to sharp edges due to deviation from the standard Python ecosystem.
It's not so much that Poetry isn't resilient to it -- it's that Debian makes undocumented and invasive adjustments to code that isn't meant to be exposed/portable to any external tooling like Poetry, and we have to constantly hit a moving target that is not documented anywhere. It's hard to be truly robust and resilient when Debian doesn't make their patches in a way that is stable or consumable.
If the linked launchpad issue is ever fixed this will just work -- but as long as Debian monkeypatches distutils and doesn't properly set sysconfig paths (like specified in the Python Packager's documentation) this will remain a moving target as we have to introspect unexported internals of a deprecated module.
Adding to the argument by saying that I'd very much like to use official tensorflow docker images and they use system python. When adding my custom packages to the default install through poetry and virtualenvs.create=false it results in the bug described here.
Using a virtual environment would defeat the purpose of using an image with everything pre-installed (including jupyter notebook).
It's also causing issues with the Tensorflow images. And of course I don't want a virtual environment there, since my Docker image will blow up with two TF installations. Plus, it is quite hacky to access the python executable from the created virtual environment in a script then.
+1 for this issue. We shouldn't need to use a virtual environment inside a Docker image. I'm facing the issue when using an ubuntu:22.04 base image with poetry==1.3.1. Making the setting poetry config virtualenvs.create false and then installing packages via poetry install results in packages being install in the wrong location. This was not an issue in poetry==1.1.14.
Plus one for this - it's not just aws cli it's a problem for us deploying our docker containers - we want our installed [tool.poetry.scripts] to be runnable at system level without having to prepend with poetry run.
In a container, you can use ENV VIRTUAL_ENV=/path/to/env PATH=/path/to/env/bin:$PATH to easily activate a virtual environment. Combine that with python -m venv /path/to/env and the fact that Poetry will install into an already-activated environment instead of managing its own, and you have a much better solution.