requests
requests copied to clipboard
psf
The latest version of requests (2.29.0) does not support urllib3 2.0.0
The latest version of requests (2.29.0) does not support urllib3 2.0.0
urllib3 2.0.0 was just released: https://github.com/urllib3/urllib3/releases/tag/2.0.0
But currently requests 2.29.0 has a range bound on it: <1.27 and >=1.21.1 for urllib3.
If you try to install a package that has urllib3==2.0.0 as a dependency (while using the latest version of requests), there will be errors:
<PACKAGE> depends on urllib3==2.0.0
requests 2.29.0 depends on urllib3<1.27 and >=1.21.1
Expecting requests to support the latest version of urllib3.
(For Python 3.7 or newer)
Hi @mdmintz, this is intentional as discussed here. We'll move the pin once we get more data points on any issues in the major version bump. We have a responsibility to keep the majority of the Python ecosystem stable during the transition.
@nateprewitt I would like to note that if requests blocks urllib3 2.0.0 and everyone has requests installed for something, most people will not be able to get any new data about urllib3 2.0.0 in practice because it just cannot be installed. Is that picture missing something? Regarding the blast radius mentioned at https://github.com/psf/requests/pull/6430#issuecomment-1522542220 I would like to note that the responsibility is with every project using urllib3 directly, not with requests. Why would it be requests' responsibility?
Many, if not most, projects pull urllib3 into their closure through another dependency. Pinning transitive dependencies, while good practice through something like a lockfile, is not widely observed. We have a decade of issues on this project as empirical data on that. Often times, users aren't even aware they're using Requests, let alone urllib3. Those are the cases where this can go wrong very easily.
We'll likely move the pin within the next week, but pushing a major version update out isn't something we'll do lightly. Giving urllib3 the opportunity release before we pull everyone else in is the safest approach and least amount of pain for operation teams.
Hi @nateprewitt, thanks for your reply, I appreciate it! :+1:
Many, if not most, projects pull urllib3 into their closure through another dependency. Pinning transitive dependencies, while good practice through something like a lockfile, is not widely observed. We have a decade of issues on this project as empirical data on that. Often times, users aren't even aware they're using Requests, let alone urllib3. Those are the cases where this can go wrong very easily.
What I'm reading here is that you are concerned about users that (1) do not use urllib3 directly and (2) use urllib3 indirectly and (3) have urllib3 pulled in without a semver guard like "<2" and (4) the code using urllib3 would break with urllib3 2.0.0. Can you confirm that's the correct reading of your scenario?
Could you name example tickets where I could have a closer look at reports like that? Thanks!
This is something that shouldn't be upgraded in a minor version of requests, because it causes breaking changes.
For example, we saw our elasticsearch client pull in the 2.0.0 version of urllib3 and it caused this failure on our machine. To resolve, we're adding a constraints.txt file, but not something that should be included in a minor release of requests.
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168
Landon, that's ultimately a decision for users of Requests to decide. We're doing a very temporary overlap of this pin during launch to prevent legitimate bugs being pushed to millions of software environments.
We have no intention to release a new major version of Requests. Urllib3 2.0 was painstakingly made largely backwards compatible. We'll provide a prominent notice that we now support this version in the next minor release, and users are free to pin to older versions as needed.
This is something that shouldn't be upgraded in a minor version of requests, because it causes breaking changes.
For example, we saw our elasticsearch client pull in the 2.0.0 version of urllib3 and it caused this failure on our machine. To resolve, we're adding a constraints.txt file, but not something that should be included in a minor release of requests.
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168
To make what @nateprewitt already eloquently stated a bit more explicit:
We will not be requiring urllib3 2.0, we will be adding support for it by raising the version constraint and ensuring requests works correctly with that version.
If you cannot use 2.0 you can continue to prevent it from being installed and we won't break of you use 1.26.x
We'll move the pin once we get more data points on any issues in the major version bump.
most people will not be able to get any new data about urllib3 2.0.0 in practice because it just cannot be installed
Just a thought, one way to gain more information on compatibility in the field would be to intially make the urllib3 version bump in a requests release that is marked alpha/beta/pre-release. Pip won't install pre-releases by default. This way people who want can opt in for testing.
We'll move the pin once we get more data points on any issues in the major version bump.
most people will not be able to get any new data about urllib3 2.0.0 in practice because it just cannot be installed
Just a thought, one way to gain more information on compatibility in the field would be to intially make the urllib3 version bump in a
requestsrelease that is marked alpha/beta/pre-release. Pip won't install pre-releases by default. This way people who want can opt in for testing.
I don't know what I'm missing, but for testing I can upgrade urllib3 to 2.0.1 just fine; I get a warning that requests is incompatible, but the upgrade completes:
$ pip install urllib3 -U
Requirement already satisfied: urllib3 in ./venv/favoptic/lib/python3.8/site-packages (1.26.15)
Collecting urllib3
Downloading urllib3-2.0.1-py3-none-any.whl (123 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.3/123.3 kB 735.2 kB/s eta 0:00:00
Installing collected packages: urllib3
Attempting uninstall: urllib3
Found existing installation: urllib3 1.26.15
Uninstalling urllib3-1.26.15:
Successfully uninstalled urllib3-1.26.15
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
requests 2.29.0 requires urllib3<1.27,>=1.21.1, but you have urllib3 2.0.1 which is incompatible.
Successfully installed urllib3-2.0.1
I don't know what I'm missing, but for testing I can upgrade urllib3
You're not missing anything. Some people like to over complicate things. Every pre-release I've ever done has been tested by too few people to ever ward off the hateful emails. It's always been more work for no benefit
RequestsDependencyWarning: urllib3 (2.0.1) or chardet (None)/charset_normalizer (3.0.1) doesn't match a supported version!
I don't know what I'm missing, but for testing I can upgrade urllib3
Untrue, even if it installs ok, it is completely broken.
As was originally stated, Requests 2.30.0 has been released within a week of the urllib3 launch. Users who wish to stay on urllib3 1.x should ensure they're pinned to urllib<2 prior to upgrading. This may include users on legacy systems with OpenSSL <1.1.1 or those relying on the Google App Engine integration.
I'll resolve this now as it's unblocked, thanks for your patience.
@nateprewitt thank you!
Note to self, related commit is 2ad18e0e10e7d7ecd5384c378f25ec8821a10a29.
For anyone upgrading, we have yanked Requests 2.30.0 as urllib3 appears to be silently dropping data on compressed response bodies. We'll be tracking investigation in urllib3/urllib3#3009.
Take 2 🎬 We've unyanked 2.30.0 now that the regression in urllib3 has been resolved. I'll leave this open for tracking today in the event of other late arising issues.
@nateprewitt Please yank it again. This is causing issues once more. Ansible builds that were failing all week, stating working again 24 hours ago (I'm guessing this is when you yanked the library). These are now failing again.
@andrew-pickin-epi Please open a separate issue with a full description, we don't have much to go off of your comment.
@andrew-pickin-epi this is usually something we'd appreciate being raised sooner than later. Can you please open a separate issue with the failure and preferably links to the builds?
@nateprewitt & @sethmlarson Ansible build using docker module, which in turn uses requests, which it turn ..... I'm build in AWS amazon linux instances. ...
Ansible reports.
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168
fatal: [sl-monitor]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (Docker SDK for Python: docker above 5.0.0 (Python >= 3.6) or docker before 5.0.0 (Python 2.7) or docker-py (Python 2.6)) on sl-monitor.epimorphics.net's Python /usr/bin/python3.7. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` (Python >= 3.6) or `pip install docker==4.4.4` (Python 2.7) or `pip install docker-py` (Python 2.6). The error was: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168"}
These are fresh OS installs that of deployment code that's been working for months but started failing last week, meaning all our deployment had to be doen by hand, but started working approx 2020-05-03 1400 GMT.
I've just done a fresh set of clean builds and they are all faling once more.
Again, please open a new issue. If you're running on AWS, you'll either need to choose an image with OpenSSL 1.1.1 support or pin your environment to urllib3<2. This is not an issue with Requests or urllib3 though.
@sethmlarson & @nateprewitt I'd gladly raise a separate ticket but I'm not close enough to know what the root cause is, or demonstrate how this can be reproduced.
What I can say is that when the library was pulled my installations worked. I came back here to see if there were any updated, only when my orchestration jobs started failing again, only to see a message 3min old saying the library was back in place and all was well. Sadly this isn't true.
Shouldn't this have been a major version bump here due to the introduced incompabilities?
@bastimeyer It was a major version release for the project which had incompatibilities introduced. Requests allows you to install either urllib3 1.26.x or 2.x, that is not backwards incompatible.
@sethmlarson
Previous releases of requests had pinned urllib3 to <1.27, and now in your current non-major-version-bumping release 2.30.0, you've bumped urllib3 to <3, which means everyone who currently has pinned requests to <3 in one of their past releases (in order to not have any breaking changes) will inevitably make users install urllib3 2.x unless they add the version constraint themselves and publish a new release with the updated version constraints. This is a breaking change of requests, unfortunately.
@bastimeyer This is a question of software development hygiene. If you have a system that relies on stability, you should (ideally) be generating a lockfile, or at a minimum reviewing changes prior to deployment. The addition of support is not a breaking change, choosing to pull in a new major version may be breaking which is why we highlighted to carefully review before installing Requests 2.30.0.
Moving the pin was specifically because Requests was incompatible with urllib3 2.0 previously. That constraint is no longer in play, so users are free to choose their versions as needed.
came back here to see if there were any updated, only when my orchestration jobs started failing again, only to see a message 3min old saying the library was back in place and all was well. Sadly this isn't true.
The issue for us was somewhere in ansible itself
Two separate examples
Error connecting: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'
Failed to connect to the host via ssh: /usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (2.0.0) or chardet (3.0.4) doesn't match a supported version!
In our case, the solution was to pip uninstall urllib3 and install it via apt installing python3-requests and python3-urllib3 as we have better control over pinning in apt packages vs pip.
Mixing apt and pip installations of packages is nearly guaranteed to put your system into a broken state, and is basically unsupported (and unsupportable). I'd strongly encourage you to pick a singe package manager and learn about its pinning faculties.
Mixing apt and pip installations of packages is nearly guaranteed to put your system into a broken state, and is basically unsupported (and unsupportable). I'd strongly encourage you to pick a singe package manager and learn about its pinning faculties.
Understood, and we avoid pip whenever possible for exactly this reason. Unfortunately, the wealth of python-env tooling and the tendency of python library developers to lean towards pip releases vs sane distro packaging make this a losing proposition. Going 100% to pip is untenable as well, as we depend on LTS releases of distros to provide some protection against regressions. If one could only convince all python developers that they should learn how to create packages for popular distros....