System site packages is not seen as a Python package

[niniack]

• Poetry version: Poetry master branch
• Python version: 3.8.10
• OS version and name: Pop!_OS 22.04 LTS
• pyproject.toml: toml file

• [ ] I am on the latest stable Poetry version, installed using a recommended method. I am not because I want to use the patch introduced in commit 237dff088d348e110839367416ca659e85a691af
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have consulted the FAQ and blog for any relevant entries or release notes.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

My poetry install consistently fails with the error

Directory /usr/local/lib/python3.8/dist-packages for torch does not seem to be a Python package

Here is the gist that shows the verbose output of the failure. None of the packages get installed

I am using an NVIDIA Pytorch container that has torch configured as I need it. I would like to use Poetry to manage my project. Ideally, I would like Poetry to handle everything except my torch and torchvision dependencies.

In order to achieve this, I am always sure to:

1. run poetry config virtualenvs.options.system-site-packages true
2. pin torch = { version = "1.14.0a0+410ce96", allow-prereleases = true} in my pyproject.toml

When I run poetry lock, this is what the seection for torch looks like

...
[[package]]
name = "torch"
version = "1.14.0a0+410ce96"
description = "Tensors and Dynamic neural networks in Python with strong GPU acceleration"
optional = false
python-versions = "*"
files = []
develop = false

[package.source]
type = "directory"
url = "../../usr/local/lib/python3.8/dist-packages"
...

This comment in in PR #8359 caught my eye.

IMO, it would be nice to have a test in test_installed_repository.py that checks that a package from system site packages does not have source_type == "directory".

If a system site package was to be seen as a directory, would that mean this patch would fail?

I've tried to remove [package.source in the lock file and poetry install still fails. Here is the verbose output.

Removing just the source so that the lock file looks like:

[package.source]
url = "../../usr/local/lib/python3.8/dist-packages"

gives the same result.

Ultimately, I would like to pin my torch package in my toml file, use system-site-packages in my venv, not mess with the lock file, and have poetry accept my installation as a Python package that already exists and doesn't need to be updated/reinstalled

[dimbleby]

smells like there's an unwanted direct_url.json lying around on the docker image; else why would poetry think that this was a directory dependency?

[dimbleby]

might be fixed, or anyway progressed, by #8549

however

• doubtful that one can be merged without a unit test
• proper fix is likely for nvidia not to leave unwanted direct_url.json files lying around in their containers

[niniack]

Thanks for leaving an update! I switched my setup to avoid this problem, but I'm happy to give this setup a shot again when I have some more time.

[felix-last]

Same issue in docker image tensorflow/tensorflow:2.17.0-gpu-jupyter

• Set POETRY_VIRTUALENVS_CREATE=false in order to install to system packages directory
• Removing all direct_url.json did not help
• Error message: Directory /usr/lib/python3/dist-packages for blinker does not seem to be a Python package

The error only happens when installing with --sync flag.

Another workaround would be to use poetry only to generate a requirements.txt (poetry export --format=requirements.txt > requirements.txt), then installing with pip install -r requirements.txt).

MWE Dockerfile

Failing

FROM tensorflow/tensorflow:2.17.0-gpu-jupyter

ENV POETRY_VIRTUALENVS_CREATE=false

RUN apt update && \
    apt -fy install \
    curl

RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py && rm get-pip.py

RUN pip install poetry
RUN poetry init
RUN poetry add redis
RUN poetry install --sync

Passing

FROM tensorflow/tensorflow:2.17.0-gpu-jupyter

ENV POETRY_VIRTUALENVS_CREATE=false

RUN apt update && \
    apt -fy install \
    curl

RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py && rm get-pip.py

RUN pip install poetry
RUN poetry init
RUN poetry add redis
RUN poetry install