`InvalidVersion` exception when invalid version used on Setuptools 66

[jstucke]

setuptools version

66.0.0

Python version

3.8

OS

Ubuntu 20.04

Additional environment information

only happening when not running inside a venv

Description

Trying to install certain pip packages like e.g. ssdeep results in an pkg_resources.extern.packaging.version.InvalidVersion exception. This seems to be related to this bug where certain ubuntu/debian packages install python packages (in this case python3-distro-info and python-debian) with versions that don't conform to PEP 440

Expected behavior

With setuptools <66.0.0 this does not cause an error

How to Reproduce

1. docker run -it --rm --entrypoint=bash ubuntu:focal
2. apt update && apt install python3 python3-pip python3-distro-info python-debian libfuzzy-dev -y
3. python3 -m pip install -U setuptools pip wheel
4. python3 -m pip install ssdeep

Output

python3 -m pip install --user ssdeep --no-cache-dir
Collecting ssdeep
  Downloading ssdeep-3.4.tar.gz (110 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.8/110.8 kB 3.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [28 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-3dwjl5j0/ssdeep_f2bd2f0d9f384d9a92bf4f0f6f3090d3/setup.py", line 108, in <module>
          setup(
        File "/home/vagrant/.local/lib/python3.8/site-packages/setuptools/__init__.py", line 86, in setup
          _install_setup_requires(attrs)
        File "/home/vagrant/.local/lib/python3.8/site-packages/setuptools/__init__.py", line 80, in _install_setup_requires
          dist.fetch_build_eggs(dist.setup_requires)
        File "/home/vagrant/.local/lib/python3.8/site-packages/setuptools/dist.py", line 874, in fetch_build_eggs
          resolved_dists = pkg_resources.working_set.resolve(
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 815, in resolve
          dist = self._resolve_dist(
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 844, in _resolve_dist
          env = Environment(self.entries)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1044, in __init__
          self.scan(search_path)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1077, in scan
          self.add(dist)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1096, in add
          dists.sort(key=operator.attrgetter('hashcmp'), reverse=True)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2631, in hashcmp
          self.parsed_version,
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2678, in parsed_version
          self._parsed_version = parse_version(self.version)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__
          raise InvalidVersion(f"Invalid version: '{version}'")
      pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '0.23ubuntu1'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[nachovizzo]

I'm also facing this error

[jaraco]

This issue is by design (#2497). Non-conformant versions have been discouraged since ~2014, deprecated in packaging since late 2020 and deprecated in Setuptools since Oct 2021.

Projects should adapt to provide conformant versions (i.e. 0.23+ubuntu1) or environments that rely on such packages should pin to Setuptools < 66. These changes are necessary to keep Setuptools healthy and aligned with the goals of the packaging ecosystem.

Unfortunately, deprecation and removal is the main way I know to inform downstream consumers of breaking changes like these.

If the disruption is too great or the workarounds aren't suitable, this project may consider backing out the changes for another limited period, but only as a mitigation measure. Please first explore mitigation measures, such as pinning setuptools in the build-system.requires in the projects or work with the pip team to allow specifying constraints on build system requirements.

I'm also facing this error

Please use the thumbs-up on the original post to register that this issue also affects your project.

[yishaibeeri]

Quick note that in some cases internal dependencies several steps down the chain from the package I'm using are bringing in the new version of setuptools, making it hard to impossible to pin setuptools as a user who's running pip install. It also means install workflows that worked yesterday are broken today, even when pinning all my requirements.txt rows. This means wide impact without an available workaround for users. Am I missing an available workaround? Please consider coordinating the rollout of this change with the pip team (et al) to allow for effective pinning

[karlicoss]

I'm getting this issue when I'm using setup.py with https://github.com/pypa/setuptools_scm I have the following configuration:

        use_scm_version={
            'version_scheme': 'python-simplified-semver',
            'local_scheme': 'dirty-tag',
        },
        setup_requires=['setuptools_scm'],

With setuptools 66.0.0, I get

...
           self._parsed_version = parse_version(self.version)
        File "/usr/local/lib/python3.10/dist-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__
          raise InvalidVersion(f"Invalid version: '{version}'")
      pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '1.1build1'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

When I roll back the setuptools version to 65.7.0, it works as expected (e.g. my package version ends up as 1.0.20220516.dev56)

[jstucke]

Am I missing an available workaround?

If I am not mistaken, using a virtual environment should work (since all the packages with offending versions seem to come via apt or equivalent). If the package you are trying to install or its dependencies do not conform to PEP 440, this will of course not help. Also uninstalling the system packages (python3-distro-info, python-debian, etc.) seems to work but they are probably needed by some other packages. python3-distro-info for example is used by update-manager-core, which is in turn used by update-manager, which is used by ubuntu-desktop, so it might not be feasible.

[kchomski]

I have the same issue with setuptools==65.5.1. When tagging and trying to install a local project I get:

pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '1.23.0-test.1.2.3'

$ pip list | grep setuptools
setuptools                65.5.1

1.23.0-test.1.2.3 is a valid semver 2.0.0 tag.

EDIT: this happens inside venv (unlike OP)

$ which python
/c/USERS/foobar/.virtualenvs/venv-3.8/Scripts/python

EDIT 2: Seems like an issue on my end - after checking PEP440 I learned that it's not fully compatible with Semantic Versioning:

Semantic versions containing a hyphen (pre-releases - clause 10) or a plus sign (builds - clause 11) are not compatible with this PEP and are not permitted in the public version field.

[jstucke]

Since the error message is somewhat cryptic (it would help if it also printed out the offending package in addition to the version), I dug around a bit to help match the version from the error message to the package that caused them:

• python-debian (is probably irrelevant because Python2.7 only)
  • 0.18ubuntu0.18.04.1 bionic
  • 0.1.36ubuntu1 focal
• python3-distro-info
  • 0.23ubuntu1 focal
  • 1.1build1 jammy
  • 1.1ubuntu1 kinetic
• python3-reportbug
  • 7.5.3~deb10u1 buster
  • 7.1.8ubuntu1 bionic
  • 7.6.0ubuntu1 focal
  • 11.4.1ubuntu1 jammy
  • 11.5.0ubuntu1 kinetic/lunar

[chavinlo]

Using a fresh venv worked for me, at least when compiling xformers.

[sethrh]

I'm also facing this error

Please use the thumbs-up on the original post to register that this issue also affects your project.

Thumbs up 👍, yes this broke my build. And thumbs down 👎, I'm staying late tonight to fix a bug that's a critical...spelling error.

Unfortunately, deprecation and removal is the main way I know to inform downstream consumers of breaking changes like these.

Yeah, it's a tough problem. But deprecated or not, I generally assume that core tools work. Version numbers in particular have been historically very lenient, so this is a really fundamental change that caught me off guard.

Build logs are noise, unless there's a problem. So while it's possible that I've seen the xxx is an invalid version and will not be supported in a future release warning flying by in the logs, to me, it isn't a call to action. It's just one line among hundreds.

Suffice to say, if something is going to break, I'd prefer a more detailed message, for example:

******************************************************************************** 
The version number <xxx> on package <yyy> package is no longer valid. 

By January 2023, you need to update version numbers to conform with PEP440 
or your builds may fail. You can retain the old behavior by passing 
the --allow-deprecated-version-numbers option to setup. 

See <some url> for details.
********************************************************************************

I'd also suggest putting the message at the end of the build log, not the beginning. Then it probably would be the first thing that I see after I run a build.

[Katz-Tal]

I'm having the same error (pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '0.23ubuntu1') using the following command:

pip3 install --upgrade pip wheel setuptools && \
pip3 install scikit-build && \
pip3 install opencv-python

I have tried to use older versions of setuptools and it's failing because of a different error Could not build wheels for opencv-python which use PEP 517 and cannot be installed directly

[astrojuanlu]

I see many people pinning setuptools but not many pull requests to transitive dependencies with invalid version numbers? I think it would be helpful to mention which packages are triggering this error.

[abravalheri]

Yeah, it's a tough problem. But deprecated or not, I generally assume that core tools work. Version numbers in particular have been historically very lenient, so this is a really fundamental change that caught me off guard.

Hi @sethrh, please note that this change is necessary so setuptools can update its dependency on the latest version of packaging. This is important because:

1. Newer versions packaging will accumulate bug fixes and improvements. If setuptools is stuck with an older version, users will not benefit from the changes.
2. Operating System packaging ecosystems want to be able to distribute setuptools and this may require harmonizing the versions of the dependencies on packaging with other tools such as pip (this depends on the OS).

Build logs are noise, unless there's a problem. So while it's possible that I've seen the xxx is an invalid version and will not be supported in a future release warning flying by in the logs, to me, it isn't a call to action. It's just one line among hundreds.

Please note that you can control the verbosity of build logs, for example:

python -m build -C--quiet

The output is "medium" verbose by default. The discussion if this is a good default or not is a completely separated thing (some people may agree that when there is an error in a CI build, it is good to have some logs, otherwise you have to change the scripts to be more-verbose and re-run the entire process again).

[jborman-exos]

To identify culprit packages a simple pip list will show all installed libs and versions.

On Ubuntu 20.04 python3-distro-utils installs distro-utils with the bad 0.23ubuntu1 version.

After running pip uninstall distro-utils I was able to avoid this error.

[abravalheri]

How to Reproduce

1. docker run -it --rm --entrypoint=bash ubuntu:focal
2. apt update && apt install python3 python3-pip python3-distro-info python-debian libfuzzy-dev -y
3. pip install -U setuptools pip
4. python3 -m pip install ssdeep

Hi @jstucke, it seems that what is happening is the following:

• ssdeep is a project that relies on deprecated pip behaviour i.e. it does not have a pyproject.toml file and pip handles that scenario by running python setup.py install
• In turn, python setup.py install uses pkg_resources to list and install the setup_requires arguments, but pkg_resources is a complicated piece of software and will try to parse the metadata of every single package already installed in your system[^1].
• When the metadata for python3-distro-info is parsed, the invalid version becomes problematic.

A workaround for this error is to use pip install --use-pep517 which will prevent the deprecated behaviour for being triggered.

This is also the reason why using a virtualenv will solve the problem.

[^1]: This entire workflow is deprecated...

[abravalheri]

TL;DR for those looking for workarounds:

Option 1: Try to use pip install --use-pep517 and see if this solves your problem Option 2: Try to always install packages using pip in a virtual environment

Please try to identify the packages/projects that are using invalid versions and notify the maintainers.

Meanwhile setuptools will keep trying to obliviate pkg_resources. Unfortunately I believe this change cannot be removed, since we do need to pursue compatibility with the latest versions of packaging.

[jvallar]

sudo apt-get -y -qq update
sudo apt-get install python3-pip python-dev build-essential
sudo pip3 install --upgrade setuptools
sudo pip3 install awscli --upgrade

I removed the setuptools and it works fine.

sudo apt-get -y -qq update
sudo apt-get install python3-pip python-dev build-essential
sudo pip3 install awscli --upgrade

[abravalheri]

@jvallar this happens because your system will use an older version of setuptools.

Have you tried instead my suggestion above (to add the --use-pep517 flag)?

[jstucke]

A workaround for this error is to use pip install --use-pep517 which will prevent the deprecated behaviour for being triggered.

Hi @abravalheri, I tried it out and can confirm that it installs without errors. Thank you for your response!

[pradyunsg]

FWIW, to provide a fuller explanation of what's happening here:

• When pip tries to build a project with the "legacy" build system, the executed setup.py for the package is exposed to the entire set of packages available on that Python installation.
• setuptools needs to determine what's installed to support setup_requires with the legacy install mechanism, for which it's calling pkg_resources.working_set.resolve which looks at the entire environment.
• As that scans the environment, it trips on the invalid version if a package in the environment has an improper version.
• Debian/Ubuntu's system Python have packages installed that have invalid versions (or your $organizations' Python packages)

---

Please consider coordinating the rollout of this change with the pip team (et al) to allow for effective pinning

puts on pip and packaging maintainer hat

Hi. I am probably the person who set the ball rolling on this form of strictness within the Python packaging ecosystem. I can also tell you that I've been involved in multiple discussions with setuptools' maintainers around this change (~all of those are in public spaces). :)

Pinning things would not have helped in this case -- this isn't related to how setuptools builds your package, or how pip handles the built project, or how the two of them interact. The issue is with the environment you're building in, which contains a package with a bad version -- specifically, in OP, it's because Debian's Python packages have done things that degrade the UX (honestly, it's not the first time). The only people who could seen and reported/handled this warning didn't report this to the right people/handle this in... 2-9 years.

I'm certain that, if the maintainers of setuptools knew about this failure mode, they would've done something to provide some mechanisms to deal with this. However, if we don't know about a failure mode, because Debian (or your internal packages) do things in the way that would trigger this failure mode and no one told us, we can't do anything about a failure mode we don't know about. And, once it's been broken, the churn costs have been paid already -- it doesn't really make sense to undo things at that point.

At the end of the day, this is not a straightforward failure -- it's not an issue with your package (which pip would've warned you about), following best-practices of isolated from your system-provided Python environment will not trigger this issue and the folks who made the "mistake" of using non-PEP 440 versions didn't action on the warnings when they were being presented.

---

Since this has been reported, there have been multiple improvements to make this less painful of a failure mode and https://github.com/pypa/setuptools/issues/3780 tracks making deprecations coming out of setuptools more visible.

I can confidently say that the maintainers of setuptools, pip etc are well aware of the network effects of these changes, and that we understand that it can be difficult to deal with increased strictness causing issues because a transitive dependency three layers down does things incorrectly. To that end, we've been using the mechanisms we have to communicate this. Both setuptools and pip have been printing warnings related to this, and pip is going to get stricter around this area as well in this year (the change needed to happen in setuptools first for a complicated reasons that I'm not gonna go into).

While I'm sure that there's more that the volunteers who maintain the foundational pieces of the Python ecosystem could have done to reduce pain for you, I would like to implore you to think about what you could've done to have flagged this to them.

---

Am I missing an available workaround?

Multiple ways to resolve/work around this have already been mentioned since this was posted. I'll try to reiterated on what they are (and hopefully, my explanations above help clarify how/why that fixes/avoids issues).

• If you "own" the package affected, you can do one of the following:

  • fix the version, if it's not compatible.
  • use the pyproject.toml-based build system which isolates your builds by adding a pyproject.toml to your project.

• If you do not "own" the problematic package, you should inform the maintainers of the package and can do one of the following:

  • pass --use-pep517 to pip[^1] to opt all builds into the pyproject.toml-based build system, or
  • isolating your installation process from the system, by working in a virtual environment that can't see system-provided packages.

---

Finally, I genuinely appreciate that setuptools as a project has made this change, even though it's a slightly painful one and a potentially disruptive one. I should note that this is a change that genuinely makes things better and, honestly, increased strictness in what these tools accept will make things better for the overall ecosystem.

[^1]: That option has a bad name, we'll change it to a better one some day -- but it does what we need, enabling the pyproject.toml-based build system for all projects.

PS: The setuptools team has wanted to get rid of pkg_resources for a vast array of suboptimal behaviours, like the one that triggered this issue, but it's a complicated problem and there hasn't been sufficient (primarily-volunteer) developer availability to drive that effort over the line.

[bdrung]

Let's get the affected packages fixed in Ubuntu. For the stable releases https://wiki.ubuntu.com/StableReleaseUpdates needs to be followed.

[ryancurrah]

Any pip package installed through apt I would think would be affected by this. For example I'm running into this problem with devscripts package. https://packages.debian.org/sid/devscripts

[jstucke]

Some more apt packages with "illegal" versions I found for focal:

apt package	python package	python package version
devscripts	devscripts	2.20.2ubuntu2
drslib	drslib	0.3.1p3
python3-duecredit	duecredit	0.7.0.debian3
python3-ubuntutools	ubuntu-dev-tools	0.176ubuntu20.04.1
twms	twms	0.06y

[pradyunsg]

Thanks for that list @jstucke and for the report @ryancurrah! Please consider reporting these bad versions to the maintainers of those packages in Debian/Ubuntu.

[ghost]

For people getting this error on mod_wsgi and using virtualenvs for your dependencies: you might have an obsolete mod_wsgi config. Pass python-home to WSGIDaemonProcess instead of python-path. With the python-path way, as soon as your system has one "broken" python package installed, even if not used by the web app, it's going to break.

This tip might seem self-evident when one reads the mod_wsgi documentation today, but if your config has been done a while ago, the python-path way used to be the suggested way, as the doc says:

Note that prior practice was that these ways of setting the Python module search path were used to specify the location of the Python virtual environment. Specifically, they were used to add the site-packages directory of the Python virtual environment. You should not do that.

This makes me think that quite a few projects will be affected by this intentional break in setuptools.

[kozmaz87]

Python was known for being permissive. This is a very JAVAesque push. Did having a git commit hash in the version number ever break dependency resolution for setuptools? NO! But setuptools going all nazi on versioning breaks everyone's solutions. Be permissive, not restrictive.

PEP 440 should allow for stuff such as 1.2.3.sha1

The use case is a common library you have to install from a repository in-house and you have 2 competing versions built for 2 checked-out feature branches but you would like to run your system testing with each of them and having consecutive unreleased version numbers for the 2 feature branches is not a solution.

PEP440 seems to concern itself with public packages released into the wild but forgets about important dev use-cases.

[kozmaz87]

Hi there @pradyunsg

So I read your explanation and it seems to me that your problem is with debian-based linux distributions. Would it not be more prudent that you enforce this versioning scheme in debian/ubuntu? I really do not see a reason for python to limit its dev versioning ability(the above explained sha1 inclusion ability) and whatever other usecases might be.... I think a package should be allowed to have whatever version if I want to call one version "rubberduck" and the other one "whatever" I should not be prevented from building and installing that package as long as I maintain both the repository and the packages. This way one has the freedom to build sophisticated python-based development workflows that may not have been foreseen by people who wrote PEP440.

Where the strict enforcement should be is public repositories not accepting certain versions or ignoring incorrect ones completely. And therefore ubuntu should not give poorly versioned packages to a "legacy" build system... I might add at this point that we are pushing everyone into a corner to support some legacy system instead of just losing it for the sake of ubuntu/debian users ONLY. Nobody else would be affected.

This is a bit similar to the case when systemd developers expect the entire linux community to bend over backwards for them when their code causes a new problem and now everyone else should adapt.

I do appreciate the fact that it has been communicated and the decision was made but I do not think setuptools should fail building my package WHATEVER version number I provide. I should be allowed to name things my way. This is the ethos of python. When it comes to shared resources like public repos... by all means they should be filtering what comes in. This is my take on it. If ubuntu breaks then ubuntu should make sure they do not provide you with garbage it should not be that we force everyone to bend over backwards just because ubuntu's versioning scheme does not align with pip. So what? In most cases it is anyway prudent to use one's own python and never some system python on linux distros that is full of random hacks that usually sooner or later break something for the average application developer.

PS: I understand that you guys already pushed this through and everyone is forced to do this(until someone forks setuptools again and we begin another format war) and that at this point I am unlikely to change your mind but I do posit that you fixed a problem in the system that did not cause it, on the wrong level, by turning the wrong dial and you also went against one of the attractive aspects of development in Python, which is its reluctance to force people to do things the ONE way. As long as it does not break things, should not be governed with an iron fist and this did not break things for 63 major versions, from what I am gathering from the explanation people in debian did.

[NumberChiffre]

Seems there's been some ongoing issue with this package, a common solution also exists for https://github.com/pytorch/pytorch/issues/69894#issuecomment-1080635462

I double checked and the above solution works for this issue as well.

[xkszltl]

... Projects should adapt to provide conformant versions (i.e. 0.23+ubuntu1) or environments that rely on such packages should pin to Setuptools < 66. These changes are necessary to keep Setuptools healthy and aligned with the goals of the packaging ecosystem. ...

I would say version pinning should ONLY be suggested as a glass-breaking work around, to cover the gap before an issued is fixed in the near future.

Pinning would make future upgrade even harder to push out. Given the combination of:

• Fast movement of python ecosystem
• Limited backporting, even for security features
• Relatively short support cycle

pinning basically put a time bomb in user code. IMO that's not a good idea for infrastructure like setuptools, which is supposed to be reliable so that people can build their things on top.

[xkszltl]

Seems there's been some ongoing issue with this package, a common solution also exists for pytorch/pytorch#69894 (comment)

I double checked and the above solution works for this issue as well.

There seems to be too many of "aggressive changes" after 59, to the point that I've frequently seen pin-to-59 answers and I personally had them somewhere as well. That's definitely not a good trend.

Also, for breakage like this in distro, most user, especially python dev who don't deal with low level works, simply don't have much connection/feedback to the distro side. It can easily take years before patched on the distro side, during which downstream user has nothing to do but wait. So it won't even help canonicalize the version scheme existing in this world, but just add unnecessary trouble to everyone.

[kozmaz87]

And you know... I hate anaconda with a passion because of its super-slow and buggy package dependency resolution algo and sometimes its overly restrictive or downright bad(pyface pulling in GPL packages in conda but not in pip) dependency structure enforced through and through but not even they prevent people to append whatever they want at the end of the version string.

@xkszltl saying it is a timebomb in user code to pin old version of setuptools is accurate and I would also posit that treating python devs like they were Java devs is indeed a bad trend. We do not need everything prescribed for us. i.e. PEP8 can be enforced by using specific tooling, but python still runs code that isn't PEP8 but is still parseable because it is the way it has been build for decades a social contract and understanding of sorts amongst python devs that we do not stampede over each other unnecessarily as long as things work.