pypa
Suspected module collision when introducing namespace package as dependency of package
Description
We extracted a module of our package into a namespace package. When upgrading to the new version without first manually uninstalling the existing version, there is an error.
Specifically...
kfp==2.0.1
kfp has a sub-module dsl. from kfp import dsl works fine.
kfp==2.1.2
kfp takes a dependency on a namespace package kfp-dsl, which is released separately. from kfp import dsl still works fine if the user is not installing kfp==2.1.2 on top of an existing kfp==2.0.1 install (e.g., runs pip uninstall kfp first).
Problem
When kfp==2.0.1 is already installed, running pip install kfp==2.1.2 doesn't put the package into a valid state.
Expected behavior
When kfp==2.0.1 is installed, pip install kfp==2.1.2 should put the package into a valid state.
pip version
23.2.1
Python version
3.7
OS
MacOS 13.5
How to Reproduce
Success:
-
pip install --upgrade pip -
pip install kfp==2.0.1 -
pip uninstall kfp==2.0.1 -
pip install kfp==2.1.2 -
python -c 'import kfp'
Failure:
-
pip install --upgrade pip -
pip install kfp==2.0.1 -
pip install kfp==2.1.2 -
python -c 'import kfp'
Output
Success: Nothing
Failure:
AttributeError: module 'kfp.dsl.structures' has no attribute 'InputSpec'
Code of Conduct
- [X] I agree to follow the PSF Code of Conduct.
The same happens for top-level files without namespace packages.
Let's say that a package a ships foo.py in version 0.1.0. In version 0.2.0 it splits into a and b, foo.py moves to b (again to the root). Now installing a==0.2.0 produces different results depending on whether a==0.1.0 was previously installed in the environment. If it was, foo.py is gone forever until b is force-reinstalled.
Full repro script:
cd "$(mktemp -d)" && python -m venv .venv && . .venv/bin/activate
mkdir a
# mkdir: created directory 'a'
echo 'FOO = 1' >a/foo.py
echo 'from setuptools import setup; setup(name="a", version="0.1.0")' >| a/setup.py
python -m pip install ./a
# Processing ./a
# Installing build dependencies ... done
# Getting requirements to build wheel ... done
# Preparing metadata (pyproject.toml) ... done
# Building wheels for collected packages: a
# Building wheel for a (pyproject.toml) ... done
# Created wheel for a: filename=a-0.1.0-py3-none-any.whl size=1006 sha256=282c74d75d8ccf9053d009b7977b580da6984bdaa84340143caecbddf4d749e7
# Stored in directory: /tmp/pip-ephem-wheel-cache-1asno282/wheels/a7/45/16/1ca4d44f0d6b35a2c9e7c81a47f28a06ec9172a177bb4e557f
# Successfully built a
# Installing collected packages: a
# Successfully installed a-0.1.0
ls -la .venv/lib/python*/site-packages/
# total 4
# drwxrwxr-x 6 stas stas 140 Oct 6 02:43 .
# drwxrwxr-x 3 stas stas 60 Oct 6 02:42 ..
# drwxrwxr-x 2 stas stas 180 Oct 6 02:43 a-0.1.0.dist-info
# -rw-rw-r-- 1 stas stas 8 Oct 6 02:43 foo.py
# drwxrwxr-x 5 stas stas 180 Oct 6 02:42 pip
# drwxrwxr-x 2 stas stas 220 Oct 6 02:42 pip-24.0.dist-info
# drwxrwxr-x 2 stas stas 60 Oct 6 02:43 __pycache__
mkdir b
# mkdir: created directory 'b'
rm a/foo.py
echo 'FOO = 2' >b/foo.py
echo "from setuptools import setup; setup(name='a', version='0.2.0', install_requires=['b @ file://$(pwd)/b'])" >| a/setup.py
echo 'from setuptools import setup; setup(name="b", version="0.1.0")' >| b/setup.py
python -m pip install ./a
# Processing ./a
# Installing build dependencies ... done
# Getting requirements to build wheel ... done
# Preparing metadata (pyproject.toml) ... done
# Processing ./b (from a==0.2.0)
# Installing build dependencies ... done
# Getting requirements to build wheel ... done
# Preparing metadata (pyproject.toml) ... done
# Building wheels for collected packages: a, b
# Building wheel for a (pyproject.toml) ... done
# Created wheel for a: filename=a-0.2.0-py3-none-any.whl size=929 sha256=1f91a5775368d3346f05a97e41ebf1e975c0bc046b8a9f45c76500c489f7b87a
# Stored in directory: /tmp/pip-ephem-wheel-cache-c5zox3hn/wheels/a7/45/16/1ca4d44f0d6b35a2c9e7c81a47f28a06ec9172a177bb4e557f
# Building wheel for b (pyproject.toml) ... done
# Created wheel for b: filename=b-0.1.0-py3-none-any.whl size=1007 sha256=5d7c58fdb8263a95f988a505d3143bbaad99ded208d40ffd742820adb01d7d6f
# Stored in directory: /tmp/pip-ephem-wheel-cache-c5zox3hn/wheels/38/63/e1/deb3a056150192cf9560e6d17743362c5256bfc6b244b6bdf5
# Successfully built a b
# Installing collected packages: b, a
# Attempting uninstall: a
# Found existing installation: a 0.1.0
# Uninstalling a-0.1.0:
# Successfully uninstalled a-0.1.0
# Successfully installed a-0.2.0 b-0.1.0
ls -la .venv/lib/python*/site-packages/
# total 0
# drwxrwxr-x 6 stas stas 120 Oct 6 02:44 .
# drwxrwxr-x 3 stas stas 60 Oct 6 02:42 ..
# drwxrwxr-x 2 stas stas 180 Oct 6 02:44 a-0.2.0.dist-info
# drwxrwxr-x 2 stas stas 160 Oct 6 02:44 b-0.1.0.dist-info
# drwxrwxr-x 5 stas stas 180 Oct 6 02:42 pip
# drwxrwxr-x 2 stas stas 220 Oct 6 02:42 pip-24.0.dist-info
This has just hit mypy (python/mypy#20006) - we shipped a shared library with a previous release, and then it was moved to a separate package librt. Installing compiled dev mypy version now produces a broken install if mypy==1.18.2 was present in the environment.
