Dependencies get overriden when there are multiple versions of different markers

[frostming]

trafficstars

• [x] I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

$ pdm add pytest==4.6.*

The package requires:

[
    ...
    'more-itertools>=4.0.0,<6.0.0;python_version<="2.7"',
    'more-itertools>=4.0.0;python_version>"2.7"',
    ...
]

Actual behavior

Lockfile entry:

more-itertools = {marker = "python_version > \"2.7\"", version = ">=4.0.0"}

Expected behavior

Both entries are kept in lockfile.

Environment Information

pdm version:

# Paste the output of `pdm info && pdm info --env` below:

[frostming]

Workaround: Pin more-itertools>=4.0.0,<6.0.0.

[kmsquire]

We ran into a similar problem. We have a bunch of (private) wheels for different versions of python and different operating systems, which looks like:

    "awesome_package @ https://some.url.com/python/awesome_package-cp39-cp39-linux_x86_64.whl ; sys_platform == 'linux' and python_version == '3.9'",
    "awesome_package @ https://some.url.com/python/awesome_package-cp310-cp310-linux_x86_64.whl ; sys_platform == 'linux' and python_version == '3.10'",
    "awesome_package @ https://some.url.com/python/awesome_package-cp39-cp39-macosx_12_0_x86_64.whl ; sys_platform == 'darwin' and python_version == '3.9'",
    "awesome_package @ https://some.url.com/python/awesome_package-cp310-cp310-macosx_12_0_x86_64.whl ; sys_platform == 'darwin' and python_version == '3.10'",

When defined in the local project, ~~this seems to work fine (at least it resolves)~~ it resolves to the last item specified, as described in the OP. However, when this is defined in a dependency, I get an error,

🔒 Lock failed
Unable to find a resolution for awesome_package
because of the following conflicts:
  awesome_package @ https://some.url.com/python/awesome_package-2.21.0-cp310-cp310-linux_x86_64.whl ; sys_platform == "linux" and python_version == "3.10" (from <Candidate dependent-package 2.0.1 from None>)
  awesome_package @ https://some.url.com/python/awesome_package-2.21.0-cp310-cp310-macosx_12_0_x86_64.whl ; sys_platform == "darwin" and python_version == "3.10" (from <Candidate dependent-package 2.0.1 from None>)

This obviously isn't a conflict (they're for different platforms).

[pawamoy]

And I've encountered this as well yesterday. I rarely have complex dependencies, but got importlib-metadata conflicts (again) and wanted to specify:

• a certain range for importlib-metadata for Python less than 3.8,
• and another range for Python greater than or equal to 3.8,

but it seems only the last constraint is kept and used.

Fortunately, in this particular case, I was able to get away with the conflicts with a single importlib-metadata<5; python < '3.8', though I did have to specify another line for Flake8 or the resolution would loop indefinitely: flake8>=4; python_version >= '3.8'.

[theAeon]

Currently trying to do this for two different sys_platforms with local wheel files and I'm getting [KeyError]: ('numba', '0.57.0.dev0+872.g971345351.dirty', None, False) on install.

[mforbes]

Perhaps related is an extremely simple marker:

dependencies = [
    'six >= 1.16.0; python_version < "3.7"'
]
requires-python = "~=3.7.0"

This gives [InvalidPyVersion]: Invalid specifier: 'impossible' even though according to PEP 508, the dependency specification should be ignored.

Let me know if I should open this as a new issue: it obviously has nothing to do with "multiple" markers, but might be related to the same underlyng problem.

Details

$ pdm --version
PDM, version 2.4.9
$ pdm info
PDM version:
  2.4.9
Python Interpreter:
  /Users/mforbes/tmp/python/pdm/issue4/.venv/bin/python (3.7)
Project Root:
  /Users/mforbes/tmp/python/pdm/issue4
Project Packages:
  None
$ CONDA_PREFIX= pdm lock -v
STATUS: Resolving dependencies
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   six>=1.16.0; python_version < "3.7"
pdm.termui:   python>=3.7,<3.8
pdm.termui:   Adding requirement six>=1.16.0; python_version < "3.7"
pdm.termui:   Adding requirement python>=3.7,<3.8
pdm.termui: ======== Starting round 0 ========
[InvalidPyVersion]: Invalid specifier: 'impossible'

[commonism]

Had the same, the impossible error is not that helpful and could be improved with the name of the dependency which fails

[project]
name = "test"
version = "0.1"
description = "fails"
authors = [
    {name = "Example", email = "[email protected]"},
]
dependencies = [
    'typing_extensions; python_version<"3.9"',
]
requires-python = ">=3.8"
readme = "README.md"
license = {text = "MIT"}

[joaoleveiga]

any chance this might be picked up? The workaround we're going for is to have the same package as a dev-dependency with the python_version marker:

dependencies = [
    "package>=1.0.0",
]

[tool.dm.dev-dependencies]
test = [
    "package<2 ; python_version=='3.7'",
]

The goal being that:

1. For testing purposes, it will force the installation of "package" with a given version depending on the environment
2. For production purposes, it will resolve to whichever version is available (and compatible)

[Drachenfels]

@mforbes

The issue you have reported here seems to be about this -> https://github.com/pdm-project/pdm/issues/2179, as I understand you define package requirements for python as less than 3.7 but the project runs on 3.7.x. PDM appears to be very strict about it and says it's an impossible scenario.

[Drachenfels]

Similar scenario to this bug I observed with my pyproject:

dependencies = [
    "pandas~=1.1.0; python_version >= \"3.9\"",
    # our legacy code
    "pandas==0.24.2; python_version < \"3.9\"",
 ]

Despite PDM being configured to work with python3.9 via pdm-python pointing to $HOME/Envs/$PROJECT/bin/python3.9

Log output is as follows:

pdm.termui: ======== Start resolving requirements ========
pdm.termui:   pandas==0.24.2; python_version < "3.9"
pdm.termui:   python>=3.7
pdm.termui:   Adding requirement pandas==0.24.2; python_version < "3.9"
...
pdm.termui:   Adding requirement python>=3.7
pdm.termui: ======== Starting round 0 ========
unearth.auth: Found credentials in index url for $PACKAGE
unearth.preparer: Downloading <Link $INDEX/pypi/package/simple/pandas/0.24.2/    pandas-0.24.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (from $INDEX/pypi/package/simple/pandas/)> (16.7 MB)
pdm.termui: Error occurs

It looks like there are two distinct issues, one is that only the last package definition actually makes it from pyproject, and the second is that the python version is taken from pyproject.toml specifier and not actually the version of python that runs the install command.

My pdm version is: PDM, version 2.8.2

[frostming]

and the second is that the python version is taken from pyproject.toml specifier and not actually the version of python that runs the install command.

Not quite correct. The requires-python will be honored when locking while the current python version will be honored when installing. They are separate stages and serve for different purposes.

[Drachenfels]

They are separate stages and serve for different purposes.

This seems not to be the case with pandas, unless pandas 0.24.2 after downloading runs some additional pre-install steps that lock resolver triggers.

[thearchitector]

I am encountering the same bug when trying to make my library work across python versions. With the following, I would expect the dep to be 0.19 on 3.7 and 0.20 on >=3.8, but the last one overrides the former on every version of python.

requires-python = ">=3.7,<3.12"
dependencies = [
    "tortoise-orm[accel]<0.20,>=0.18.0 ; python_version~=\"3.7.0\"",
    "tortoise-orm[accel]>=0.18.0 ; python_version>=\"3.8\"",
]

---

i can verify this is the case if i run pdm export --pyproject -o requirements.txt. only the last entry is in the output file, and it changes if i swap the order.

[andreas-vester]

Same here:

[project]
...
dependencies = [
    "kaleido==0.0.3; platform_system!='Windows'",
    "kaleido==0.1.0post1; platform_system=='Windows'",
]

kaleido v0.0.3 will never get installed on NON-Windows machines.

I did some testing using pip directly on a Windows and Linux machine. pip seems to be working fine with my pyproject.toml. Having said that, maybe someone can confirm it?

@frostming Do you have any idea/road map to solve this issue in the foreseeable future? - Thanks.

[mjspeck]

Is any work being done on this issue? It's been open for over three years now.

[Jamim]

Hi everyone,

If no one is working on this issue at the moment, I can try to investigate and fix it. Just let me know if there is any progress on this already.

Thanks!

[frostming]

@Jamim Thanks a bunch! you can join the discord channel and we'll discuss how it will be going.

[thearchitector]

@Jamim apologies if you're on the Discord, your username didn't pop up when searched -- have you made progress on this, by chance?

[Jamim]

Hello @thearchitector,

apologies if you're on the Discord, your username didn't pop up when searched

You can find me on Discord as Aliaksei Urbanski or aliaksei.urbanski.

have you made progress on this, by chance?

Sorry, but not yet 😢 However, I'm still planning to work on this in the next few days.

[wgordon17]

@frostming @Jamim Apologies if this isn't the right issue, but I'm trying to understand how to apply this use case...

• Use psycopg[binary] in development (so it's a dev-dependency)
• Use psycopg[c] in production <-- Where do I put this one??

Does psycopg[c] go in dependencies, but it'll get overwritten in development? Or does it go into some "production-only" group?

[frostming]

Does psycopg[c] go in dependencies, but it'll get overwritten in development? Or does it go into some "production-only" group?

Just keep in mind there would be only one pinned version in the lock file, no matter what. You should avoid this by creating two separate lock files with -L/--lockfile option and select the groups you desired, not both. Specifically, this would be:

pdm lock --prod
pdm lock --without default --with dev -L pdm.dev.lock