Unable to upgrade a tightly-coupled dependency

[spenczar]

My project depends on two packages, dagster and dagster-k8s (and actually many more, including more dagster-* packages - but let's stick to two for this explanation).

Dagster uses a tightly-coupled versioning scheme. The dagster libraries (like dagster-k8s) are always pinned to exact versions of the dagster core library (dagster). So, for example:

• dagster-k8s 0.17.19 requires dagster==1.1.19
• dagster-k8s 0.17.20 requires dagster==1.1.20
• dagster-k8s 0.17.21 requires dagster==1.1.21 And so on. Each dagster release is accompanied by a release of all these library packages with a matching version.

I'd like to constrain the version of dagster that I use, and have pip-compile compute the appropriate version of all dagster libraries. This seems to work exactly once. After applying the constraint the first time, I can't upgrade.

I have a setup.cfg that includes

[options]
install_requires =
    dagster>=1.1.20
    dagster-k8s

I pip-compiled this a while ago, and it all worked. Dependencies are all resolved.

Now I want to upgrade to the latest dagster-core, dagster 1.1.21. I update my setup.cfg:

[options]
install_requires =
    dagster>=1.1.21
    dagster-k8s

I try to pip-compile -P dagster, but no dice, because the previous pip-compile step pinned dagster through the dagster-k8s dependency:

There are incompatible versions in the resolved dependencies:
  dagster==1.1.21 (from pip-tools-bug-demo-spenczar (pyproject.toml))
  dagster==1.1.20 (from dagster-k8s==0.17.20->pip-tools-bug-demo-spenczar (pyproject.toml))

I try it with the backtracking resolver, and get an analagous error:

pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('dagster==1.1.21'), parent=None), RequirementInformation(requirement=SpecifierRequirement('dagster==1.1.20'), parent=LinkCandidate('https://files.pythonhosted.org/packages/bb/02/6ff11cae47979838b5b7f7e8731c4bac4ad984be4b86fef99ddcfe257608/dagster_k8s-0.17.20-py3-none-any.whl (from https://pypi.org/simple/dagster-k8s/)'))]

I still get the same errors if I try pip-compile --upgrade-package dagster.

Environment Versions

1. OS Type: MacOS
2. Python version: Python 3.9.6 and Python 3.10.10, at least.
3. pip version: pip 23.0.1
4. pip-tools version: pip-compile, version 6.12.2

Steps to replicate

1. Make a barebones project

mkdir pip-tools-bug-demo
cd pip-tools-bug-demo
echo "[options]
install_requires =
    dagster==1.1.20
    dagster-k8s
" > setup.cfg

3. Run pip-compile

pip-compile --resolver=backtracking

4. Change the version constraint to upgrade dagster

echo "[options]
install_requires =
    dagster==1.1.21
    dagster-k8s
" > setup.cfg

5. Try to pip-compile again to upgrade

pip-compile --resolver=backtracking

Expected result

I expect a requirements.txt to be written with dagster==1.1.21 and dagster-k8s==0.17.21 specified.

Actual result

This is presented as unresolveable (which is not really true):

  ERROR: Cannot install UNKNOWN (pyproject.toml) and dagster==1.1.21 because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/Users/swnelson/.pyenv/versions/3.10.10/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/Users/swnelson/.pyenv/versions/3.10.10/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/swnelson/.pyenv/versions/3.10.10/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 385, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('dagster==1.1.21'), parent=None), RequirementInformation(requirement=SpecifierRequirement('dagster==1.1.20'), parent=LinkCandidate('https://files.pythonhosted.org/packages/bb/02/6ff11cae47979838b5b7f7e8731c4bac4ad984be4b86fef99ddcfe257608/dagster_k8s-0.17.20-py3-none-any.whl (from https://pypi.org/simple/dagster-k8s/)'))]

[AndydeCleyre]

Thanks! What happens when you pass the all-packages upgrade flag, using the backtracking resolver?

[spenczar]

@AndydeCleyre if I use pip-compile --resolver=backtracking --upgrade then things work fine, but it (of course) does upgrade all dependencies, not just the one I want to target.

[spenczar]

I can also do pip-compile --resolver=backtracking --upgrade-package dagster-k8s --upgrade-package dagster and things work. But this is essentially doing manual dependency resolution. I would expect that pip-compile, or its resolver, would infer this for me.

[AndydeCleyre]

The reproduction steps aren't quite complete I think, but I did reproduce using plain requirements.in instead. In my case I notice a KeyError:

  ERROR: Cannot install -r requirements.in (line 2) and dagster>=1.1.21 because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 316, in _backjump
    name, candidate = broken_state.mapping.popitem()
KeyError: 'dictionary is empty'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 434, in resolve
    success = self._backjump(causes)
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in _backjump
    raise ResolutionImpossible(causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('dagster>=1.1.21'), parent=None), RequirementInformation(requirement=SpecifierRequirement('dagster==1.1.20'), parent=LinkCandidate('https://files.pythonhosted.org/packages/bb/02/6ff11cae47979838b5b7f7e8731c4bac4ad984be4b86fef99ddcfe257608/dagster_k8s-0.17.20-py3-none-any.whl (from https://pypi.org/simple/dagster-k8s/)'))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/piptools/scripts/compile.py", line 592, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/piptools/resolver.py", line 593, in resolve
    is_resolved = self._do_resolve(
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/piptools/resolver.py", line 625, in _do_resolve
    resolver.resolve(
  File "/home/andy/.local/share/venvs/23b6cb24016aaef286cfe5dfc4e43654/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 101, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[AndydeCleyre]

@atugushev in case this helps jumpstart debugging:

  ERROR: Cannot install -r requirements.in (line 2) and dagster>=1.1.21 because these package versions have conflicting dependencies.
primary_ireq=<InstallRequirement object: dagster>=1.1.21 (from -r requirements.in (line 1)) editable=False>, version='1.1.20'
discarding
primary_ireq=<InstallRequirement object: dagster-k8s (from -r requirements.in (line 2)) editable=False>, version='0.17.20'
cause_exc=ResolutionImpossible([RequirementInformation(requirement=SpecifierRequirement('dagster>=1.1.21'), parent=None), RequirementInformation(requirement=SpecifierRequirement('dagster==1.1.20'), parent=LinkCandidate('https://files.pythonhosted.org/packages/bb/02/6ff11cae47979838b5b7f7e8731c4bac4ad984be4b86fef99ddcfe257608/dagster_k8s-0.17.20-py3-none-any.whl (from https://pypi.org/simple/dagster-k8s/)'))])
cause_ireq_names={'dagster'}
cause_existing_ireq=None

[PeterJCLaw]

This issue, or something very similar, seems to impact pydantic too. We don't get an error in that case, instead pip-compile -P pydantic just isn't able to upgrade the package.

Example inputs:

# requirements.in
pydantic

# requirements.txt ("via" comments omitted for brevity)
annotated-types==0.5.0
pydantic==2.1.1
pydantic-core==2.4.0
typing-extensions==4.8.0

Here running pip-compile -P pydantic doesn't result in an upgrade (for comparison pydantic==2.4.2 is the latest at the time of writing, depends on pydantic-core==2.10.1)

Running either pip-compile -P pydantic -P pydantic-core or pip-compile -P pydantic --resolver=legacy will suitably upgrade things.

Like dagster, pydantic versions appear to pin pydantic-core to an exact version.

[tboddyspargo]

I wanted to add another reproduction case, in case it helps hone in on a solution:

echo "s3fs[boto3]==2023.1.0" > test.txt

<<<"s3fs[boto3]>=2023.0.0
boto3>=1.26.100" pip-compile --no-header - -o test.txt

An existing requirements file seems to be a requirement, and the repro intentionally omits --upgrade or --upgrade-package flags because they are not the desired outcome or are too heavy-handed of a work-around.

What I expect to happen is that pip-compile (or pip?) will realize that aiobotocore[boto3]==2.4.2 (requirement coming from s3fs[boto3]==2023.1.0 in test.txt) is not a valid result, because it very narrowly constrains botocore<1.27.60,>=1.27.59, which is in conflict with botocore<1.30.0,>=1.29.100 (from boto3>=1.26.100 in the input constraints). So, the resolver should discard the current versions of s3fs[boto3]==2023.1.0 from test.txt and start backtracking (from the newest available?) version of s3fs[boto3] until it finds versions of all of its dependencies that are compatible with both each other and the specified constraints.

EDIT: I had an example here of something that would allow it to "work", but it was actually just not the desired result, so although it didn't error, it didn't accomplish anything. So far as I can tell, only adding -P aiobotocore (the indirect requirement with a very narrow constraint) allowed the resolver to successfully discard the previous version and find a correct resolution path.

~Perhaps because this situation requires discarding TWO previous package results is where the resolver gets stuck?~ Perhaps the key problem area has to do with discarding existing lockfile versions whose indirect requirements are explicitly constrained (the resolver should be more willing to discard existing versions from the lockfile, if they can't be resolved)? Anyway, I hope this helps!

EDIT: 12/5/23@14:00 - I corrected and simplified the repro examples.