poetry
poetry copied to clipboard
python-poetry
Multiple constraints dependencies collapsing with groups
Description
A pyproject.toml using Poetry style dependencies, with a single requirement having different extras across multiple groups, will unhelpfully collapse all the version-specific requirements.
Before version 2, the different versions would all be expressed in the poetry.lock, but after version 2 only the lowest version is kept in the lockfile.
This causes a lot of issues downstream when installing to multiple Python versions, because the test pipeline assumes that wheels will be available for the locked version.
$ poetry --version
Poetry (version 2.0.1)
$ poetry debug info
Poetry
Version: 2.0.1
Python: 3.12.7
Virtualenv
Python: 3.12.1
Implementation: CPython
Path: /Users/user/poetry_test/.venv
Executable: /Users/user/poetry_test/.venv/bin/python
Valid: True
Base
Platform: darwin
OS: posix
Python: 3.12.1
Path: /Library/Frameworks/Python.framework/Versions/3.12
Executable: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12
Failing example pyproject.toml:
[project]
name = "poetry-group-extra-collapse"
version = "0.1.0"
requires-python = ">=3.6.2,<4"
description = ""
dynamic = ["dependencies"]
[tool.poetry.dependencies]
pandas = [
{ version = "*", python = ">=3.9" },
{ version = "<2.1", python = "~3.8" },
{ version = "<1.4", python = "~3.7.1" },
{ version = "<1.2", python = "<3.7.1,>=3.6.1" },
]
[tool.poetry.group.hdf.dependencies]
pandas = [
{ extras = ["hdf5"], version = "*", python = ">=3.9" },
{ extras = ["hdf5"], version = "<2.1", python = "~3.8" },
{ extras = ["hdf5"], version = "<1.4", python = "~3.7.1" },
{ extras = ["hdf5"], version = "<1.2", python = "<3.7.1,>=3.6.1" },
]
[build-system]
requires = ["poetry-core>=2.0.0,<3.0.0"]
build-backend = "poetry.core.masonry.api"
Failing poetry.lock:
# This file is automatically @generated by Poetry 2.0.1 and should not be changed by hand.
[[package]]
name = "numpy"
version = "1.19.5"
description = "NumPy is the fundamental package for array computing with Python."
optional = false
python-versions = ">=3.6"
groups = ["main"]
files = [
{file = "numpy-1.19.5-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:cc6bd4fd593cb261332568485e20a0712883cf631f6f5e8e86a52caa8b2b50ff"},
{file = "numpy-1.19.5-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:aeb9ed923be74e659984e321f609b9ba54a48354bfd168d21a2b072ed1e833ea"},
{file = "numpy-1.19.5-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:8b5e972b43c8fc27d56550b4120fe6257fdc15f9301914380b27f74856299fea"},
{file = "numpy-1.19.5-cp36-cp36m-manylinux2010_i686.whl", hash = "sha256:43d4c81d5ffdff6bae58d66a3cd7f54a7acd9a0e7b18d97abb255defc09e3140"},
{file = "numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl", hash = "sha256:a4646724fba402aa7504cd48b4b50e783296b5e10a524c7a6da62e4a8ac9698d"},
{file = "numpy-1.19.5-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:2e55195bc1c6b705bfd8ad6f288b38b11b1af32f3c8289d6c50d47f950c12e76"},
{file = "numpy-1.19.5-cp36-cp36m-win32.whl", hash = "sha256:39b70c19ec771805081578cc936bbe95336798b7edf4732ed102e7a43ec5c07a"},
{file = "numpy-1.19.5-cp36-cp36m-win_amd64.whl", hash = "sha256:dbd18bcf4889b720ba13a27ec2f2aac1981bd41203b3a3b27ba7a33f88ae4827"},
{file = "numpy-1.19.5-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:603aa0706be710eea8884af807b1b3bc9fb2e49b9f4da439e76000f3b3c6ff0f"},
{file = "numpy-1.19.5-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:cae865b1cae1ec2663d8ea56ef6ff185bad091a5e33ebbadd98de2cfa3fa668f"},
{file = "numpy-1.19.5-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:36674959eed6957e61f11c912f71e78857a8d0604171dfd9ce9ad5cbf41c511c"},
{file = "numpy-1.19.5-cp37-cp37m-manylinux2010_i686.whl", hash = "sha256:06fab248a088e439402141ea04f0fffb203723148f6ee791e9c75b3e9e82f080"},
{file = "numpy-1.19.5-cp37-cp37m-manylinux2010_x86_64.whl", hash = "sha256:6149a185cece5ee78d1d196938b2a8f9d09f5a5ebfbba66969302a778d5ddd1d"},
{file = "numpy-1.19.5-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:50a4a0ad0111cc1b71fa32dedd05fa239f7fb5a43a40663269bb5dc7877cfd28"},
{file = "numpy-1.19.5-cp37-cp37m-win32.whl", hash = "sha256:d051ec1c64b85ecc69531e1137bb9751c6830772ee5c1c426dbcfe98ef5788d7"},
{file = "numpy-1.19.5-cp37-cp37m-win_amd64.whl", hash = "sha256:a12ff4c8ddfee61f90a1633a4c4afd3f7bcb32b11c52026c92a12e1325922d0d"},
{file = "numpy-1.19.5-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:cf2402002d3d9f91c8b01e66fbb436a4ed01c6498fffed0e4c7566da1d40ee1e"},
{file = "numpy-1.19.5-cp38-cp38-manylinux1_i686.whl", hash = "sha256:1ded4fce9cfaaf24e7a0ab51b7a87be9038ea1ace7f34b841fe3b6894c721d1c"},
{file = "numpy-1.19.5-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:012426a41bc9ab63bb158635aecccc7610e3eff5d31d1eb43bc099debc979d94"},
{file = "numpy-1.19.5-cp38-cp38-manylinux2010_i686.whl", hash = "sha256:759e4095edc3c1b3ac031f34d9459fa781777a93ccc633a472a5468587a190ff"},
{file = "numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:a9d17f2be3b427fbb2bce61e596cf555d6f8a56c222bd2ca148baeeb5e5c783c"},
{file = "numpy-1.19.5-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:99abf4f353c3d1a0c7a5f27699482c987cf663b1eac20db59b8c7b061eabd7fc"},
{file = "numpy-1.19.5-cp38-cp38-win32.whl", hash = "sha256:384ec0463d1c2671170901994aeb6dce126de0a95ccc3976c43b0038a37329c2"},
{file = "numpy-1.19.5-cp38-cp38-win_amd64.whl", hash = "sha256:811daee36a58dc79cf3d8bdd4a490e4277d0e4b7d103a001a4e73ddb48e7e6aa"},
{file = "numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:c843b3f50d1ab7361ca4f0b3639bf691569493a56808a0b0c54a051d260b7dbd"},
{file = "numpy-1.19.5-cp39-cp39-manylinux1_i686.whl", hash = "sha256:d6631f2e867676b13026e2846180e2c13c1e11289d67da08d71cacb2cd93d4aa"},
{file = "numpy-1.19.5-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:7fb43004bce0ca31d8f13a6eb5e943fa73371381e53f7074ed21a4cb786c32f8"},
{file = "numpy-1.19.5-cp39-cp39-manylinux2010_i686.whl", hash = "sha256:2ea52bd92ab9f768cc64a4c3ef8f4b2580a17af0a5436f6126b08efbd1838371"},
{file = "numpy-1.19.5-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:400580cbd3cff6ffa6293df2278c75aef2d58d8d93d3c5614cd67981dae68ceb"},
{file = "numpy-1.19.5-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:df609c82f18c5b9f6cb97271f03315ff0dbe481a2a02e56aeb1b1a985ce38e60"},
{file = "numpy-1.19.5-cp39-cp39-win32.whl", hash = "sha256:ab83f24d5c52d60dbc8cd0528759532736b56db58adaa7b5f1f76ad551416a1e"},
{file = "numpy-1.19.5-cp39-cp39-win_amd64.whl", hash = "sha256:0eef32ca3132a48e43f6a0f5a82cb508f22ce5a3d6f67a8329c81c8e226d3f6e"},
{file = "numpy-1.19.5-pp36-pypy36_pp73-manylinux2010_x86_64.whl", hash = "sha256:a0d53e51a6cb6f0d9082decb7a4cb6dfb33055308c4c44f53103c073f649af73"},
{file = "numpy-1.19.5.zip", hash = "sha256:a76f502430dd98d7546e1ea2250a7360c065a5fdea52b2dffe8ae7180909b6f4"},
]
[[package]]
name = "pandas"
version = "1.1.5"
description = "Powerful data structures for data analysis, time series, and statistics"
optional = false
python-versions = ">=3.6.1"
groups = ["main"]
files = [
{file = "pandas-1.1.5-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:bf23a3b54d128b50f4f9d4675b3c1857a688cc6731a32f931837d72effb2698d"},
{file = "pandas-1.1.5-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:5a780260afc88268a9d3ac3511d8f494fdcf637eece62fb9eb656a63d53eb7ca"},
{file = "pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:b61080750d19a0122469ab59b087380721d6b72a4e7d962e4d7e63e0c4504814"},
{file = "pandas-1.1.5-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:0de3ddb414d30798cbf56e642d82cac30a80223ad6fe484d66c0ce01a84d6f2f"},
{file = "pandas-1.1.5-cp36-cp36m-win32.whl", hash = "sha256:70865f96bb38fec46f7ebd66d4b5cfd0aa6b842073f298d621385ae3898d28b5"},
{file = "pandas-1.1.5-cp36-cp36m-win_amd64.whl", hash = "sha256:19a2148a1d02791352e9fa637899a78e371a3516ac6da5c4edc718f60cbae648"},
{file = "pandas-1.1.5-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:26fa92d3ac743a149a31b21d6f4337b0594b6302ea5575b37af9ca9611e8981a"},
{file = "pandas-1.1.5-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:c16d59c15d946111d2716856dd5479221c9e4f2f5c7bc2d617f39d870031e086"},
{file = "pandas-1.1.5-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:3be7a7a0ca71a2640e81d9276f526bca63505850add10206d0da2e8a0a325dae"},
{file = "pandas-1.1.5-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:573fba5b05bf2c69271a32e52399c8de599e4a15ab7cec47d3b9c904125ab788"},
{file = "pandas-1.1.5-cp37-cp37m-win32.whl", hash = "sha256:21b5a2b033380adbdd36b3116faaf9a4663e375325831dac1b519a44f9e439bb"},
{file = "pandas-1.1.5-cp37-cp37m-win_amd64.whl", hash = "sha256:24c7f8d4aee71bfa6401faeba367dd654f696a77151a8a28bc2013f7ced4af98"},
{file = "pandas-1.1.5-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:2860a97cbb25444ffc0088b457da0a79dc79f9c601238a3e0644312fcc14bf11"},
{file = "pandas-1.1.5-cp38-cp38-manylinux1_i686.whl", hash = "sha256:5008374ebb990dad9ed48b0f5d0038124c73748f5384cc8c46904dace27082d9"},
{file = "pandas-1.1.5-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:2c2f7c670ea4e60318e4b7e474d56447cf0c7d83b3c2a5405a0dbb2600b9c48e"},
{file = "pandas-1.1.5-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:0a643bae4283a37732ddfcecab3f62dd082996021b980f580903f4e8e01b3c5b"},
{file = "pandas-1.1.5-cp38-cp38-win32.whl", hash = "sha256:5447ea7af4005b0daf695a316a423b96374c9c73ffbd4533209c5ddc369e644b"},
{file = "pandas-1.1.5-cp38-cp38-win_amd64.whl", hash = "sha256:4c62e94d5d49db116bef1bd5c2486723a292d79409fc9abd51adf9e05329101d"},
{file = "pandas-1.1.5-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:731568be71fba1e13cae212c362f3d2ca8932e83cb1b85e3f1b4dd77d019254a"},
{file = "pandas-1.1.5-cp39-cp39-manylinux1_i686.whl", hash = "sha256:c61c043aafb69329d0f961b19faa30b1dab709dd34c9388143fc55680059e55a"},
{file = "pandas-1.1.5-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:2b1c6cd28a0dfda75c7b5957363333f01d370936e4c6276b7b8e696dd500582a"},
{file = "pandas-1.1.5-cp39-cp39-win32.whl", hash = "sha256:c94ff2780a1fd89f190390130d6d36173ca59fcfb3fe0ff596f9a56518191ccb"},
{file = "pandas-1.1.5-cp39-cp39-win_amd64.whl", hash = "sha256:edda9bacc3843dfbeebaf7a701763e68e741b08fccb889c003b0a52f0ee95782"},
{file = "pandas-1.1.5.tar.gz", hash = "sha256:f10fc41ee3c75a474d3bdf68d396f10782d013d7f67db99c0efbfd0acb99701b"},
]
[package.dependencies]
numpy = ">=1.15.4"
python-dateutil = ">=2.7.3"
pytz = ">=2017.2"
[package.extras]
test = ["hypothesis (>=3.58)", "pytest (>=4.0.2)", "pytest-xdist"]
[[package]]
name = "python-dateutil"
version = "2.9.0.post0"
description = "Extensions to the standard Python datetime module"
optional = false
python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7"
groups = ["main"]
files = [
{file = "python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3"},
{file = "python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427"},
]
[package.dependencies]
six = ">=1.5"
[[package]]
name = "pytz"
version = "2024.2"
description = "World timezone definitions, modern and historical"
optional = false
python-versions = "*"
groups = ["main"]
files = [
{file = "pytz-2024.2-py2.py3-none-any.whl", hash = "sha256:31c7c1817eb7fae7ca4b8c7ee50c72f93aa2dd863de768e1ef4245d426aa0725"},
{file = "pytz-2024.2.tar.gz", hash = "sha256:2aa355083c50a0f93fa581709deac0c9ad65cca8a9e9beac660adcbd493c798a"},
]
[[package]]
name = "six"
version = "1.17.0"
description = "Python 2 and 3 compatibility utilities"
optional = false
python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7"
groups = ["main"]
files = [
{file = "six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274"},
{file = "six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81"},
]
[metadata]
lock-version = "2.1"
python-versions = ">=3.6.2,<4"
content-hash = "0239ab5103207be72febfbcd86599dcdad7565260bda677edc226c36d5513b45"
Changing the requirements to remove the hdf:
...
[tool.poetry.group.hdf.dependencies]
pandas = [
{ version = "*", python = ">=3.9" },
{ version = "<2.1", python = "~3.8" },
{ version = "<1.4", python = "~3.7.1" },
{ version = "<1.2", python = "<3.7.1,>=3.6.1" },
]
...
Expected poetry.lock:
...
[[package]]
name = "pandas"
version = "1.1.5"
...
markers = "python_full_version < \"3.7.1\""
...
[[package]]
name = "pandas"
version = "1.3.5"
...
markers = "python_full_version >= \"3.7.1\" and python_version < \"3.8\""
...
[[package]]
name = "pandas"
version = "2.0.3"
...
markers = "python_version >= \"3.8\" and python_version < \"3.9\""
...
[[package]]
name = "pandas"
version = "2.2.3"
...
markers = "python_version >= \"3.9\" and python_version <= \"3.11\" or python_version >= \"3.12\""
Workarounds
Remove extras, not much of a workaround though...
Poetry Installation Method
pipx
Operating System
MacOS 15.2
Poetry Version
Poetry (version 2.0.1)
Poetry Configuration
cache-dir = "/Users/user/Library/Caches/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs" # /Users/user/Library/Caches/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false
Python Sysconfig
sysconfig.log
Paste the output of 'python -m sysconfig', over this line.
Example pyproject.toml
[project]
name = "poetry-group-extra-collapse"
version = "0.1.0"
requires-python = ">=3.6.2,<4"
description = ""
dynamic = ["dependencies"]
[tool.poetry.dependencies]
pandas = [
{ version = "*", python = ">=3.9" },
{ version = "<2.1", python = "~3.8" },
{ version = "<1.4", python = "~3.7.1" },
{ version = "<1.2", python = "<3.7.1,>=3.6.1" },
]
[tool.poetry.group.hdf.dependencies]
pandas = [
{ extras = ["hdf5"], version = "*", python = ">=3.9" },
{ extras = ["hdf5"], version = "<2.1", python = "~3.8" },
{ extras = ["hdf5"], version = "<1.4", python = "~3.7.1" },
{ extras = ["hdf5"], version = "<1.2", python = "<3.7.1,>=3.6.1" },
]
[build-system]
requires = ["poetry-core>=2.0.0,<3.0.0"]
build-backend = "poetry.core.masonry.api"
Poetry Runtime Logs
poetry-runtime.log
Paste the output of 'poetry -vvv <command>', over this line.
seems like you have been relying on an implementation detail that happened to land one way at some point
the solution that poetry finds seems completely correct per the metadata for the associated packages, no?
if you have stronger constraints about which versions of pandas etc should be installed at which versions of python - eg lower bounds - then you should express those constraints in your dependencies.
I agree that pandas 1.1.5 technically satisfies all requirements, but don't you think it's a little strange that adding extras suddenly causes the resolver to behave in an unexpected way?
poetry gracefully resolving per-version is one of its best features, so it's a shame to see it fall over like this
"a little strange" is about the strongest thing one could say about it.
It has always been the case that the solver takes choices during its search and different solutions are possible according to quite arbitrary things. And the answer has always been: if you wanted a different solution, set constraints to tell the solver what you want.
My apologies, I didn't mean to incite anything, just checking whether I've finally gone crazy :)
Checking the output of poetry lock -vvv without a poetry.lock file already present (the file being there changes the behaviour as I'm sure you expect);
With extras:
Updating dependencies
Resolving dependencies...
1: fact: poetry-group-extra-collapse is 0.1.0
1: derived: poetry-group-extra-collapse
0: Duplicate dependencies for pandas
1: fact: poetry-group-extra-collapse depends on pandas (*)
1: fact: poetry-group-extra-collapse depends on pandas (<2.1)
1: fact: poetry-group-extra-collapse depends on pandas (<1.4)
1: fact: poetry-group-extra-collapse depends on pandas (<1.2)
1: fact: poetry-group-extra-collapse depends on pandas (*)
1: fact: poetry-group-extra-collapse depends on pandas (<2.1)
1: fact: poetry-group-extra-collapse depends on pandas (<1.4)
1: fact: poetry-group-extra-collapse depends on pandas (<1.2)
1: selecting poetry-group-extra-collapse (0.1.0)
1: derived: pandas (<1.2)
Without extras:
Updating dependencies
Resolving dependencies...
1: fact: poetry-group-extra-collapse is 0.1.0
1: derived: poetry-group-extra-collapse
0: Duplicate dependencies for pandas
0: Different requirements found for pandas (*) with markers python_version >= "3.9", pandas (<2.1) with markers python_version >= "3.8" and python_version < "3.9", pandas (<1.4) with markers python_full_version >= "3.7.1" and python_version < "3.8" and pandas (<1.2) with markers python_full_version >= "3.6.1" and python_full_version < "3.7.1".
1: Version solving took 0.002 seconds.
1: Tried 1 solutions.
0: Retrying dependency resolution with the following overrides ({Package('poetry-group-extra-collapse', '0.1.0'): {'pandas': <Dependency pandas>}}).
1: fact: poetry-group-extra-collapse is 0.1.0
1: derived: poetry-group-extra-collapse
1: fact: poetry-group-extra-collapse depends on pandas (*)
1: selecting poetry-group-extra-collapse (0.1.0)
1: derived: pandas
I'm going to see if I can track down that implementation detail
https://github.com/python-poetry/poetry/blame/88b2bab0f711a9eb2d3d61b0fa3ee370bf97e498/src/poetry/puzzle/provider.py#L608
# Too complicated to handle with overrides,
# fallback to basic handling without overrides.
Sounds about right, a general solution for this would probably be very complicated.
At least the feature the implementation change enabled is very cool: https://github.com/python-poetry/poetry/pull/9553
I wonder if it's possible to harmlessly adjust the duplicates_by_extras logic, it looks like d1[0].extras and d2[0].extras are both empty by the time the marker intersections occur.
Replacing this:
if all(len(d) == 1 for d in duplicates_by_extras.values()) and all(
d1[0].marker.intersect(d2[0].marker).is_empty()
for d1, d2 in itertools.combinations(
duplicates_by_extras.values(), 2
)
):
# Since all markers are mutually exclusive,
# we can trigger overrides.
deps = list(itertools.chain(*duplicates_by_extras.values()))
with this:
if True:
# Proceed at your own risk.
deps = list(itertools.chain(*duplicates_by_extras.values()))
Brings back the results I'm used to, at the cost of a few orders of magnitude in puzzle solving time, (20.351 seconds with 25 overrides vs 0.303 seconds with 3 overrides) and the risk of unpredictable behaviour
Looks like this now almost works correctly in Poetry 2.1.1 but now subtly mangles the python_full_version markers needed for 1.1.5 and 1.3.5:
...
[[package]]
name = "pandas"
version = "1.1.5"
...
markers = "python_version == \"3.7\""
...
[[package]]
name = "pandas"
version = "2.0.3"
...
markers = "python_version == \"3.8\""
...
[[package]]
name = "pandas"
version = "2.2.3"
...
markers = "python_version >= \"3.9\""
As before, replacing the combinations if clause with True fixes it again.