Requests 2.32.0 Not supported URL scheme http+docker

[joshzcold]

Newest version of requests 2.32.0 has an incompatibility with python lib docker

INTERNALERROR> Traceback (most recent call last):

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/adapters.py", line 532, in send

INTERNALERROR>     conn = self._get_connection(request, verify, proxies=proxies, cert=cert)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/adapters.py", line 400, in _get_connection

INTERNALERROR>     conn = self.poolmanager.connection_from_host(

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host

INTERNALERROR>     return self.connection_from_context(request_context)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context

INTERNALERROR>     raise URLSchemeUnknown(scheme)

INTERNALERROR> urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

INTERNALERROR> 

INTERNALERROR> During handling of the above exception, another exception occurred:

INTERNALERROR> 

INTERNALERROR> Traceback (most recent call last):

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/docker/api/client.py", line 213, in _retrieve_server_version

INTERNALERROR>     return self.version(api_version=False)["ApiVersion"]

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version

INTERNALERROR>     return self._result(self._get(url), json=True)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/docker/utils/decorators.py", line 44, in inner

INTERNALERROR>     return f(self, *args, **kwargs)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/docker/api/client.py", line 236, in _get

INTERNALERROR>     return self.get(url, **self._set_request_timeout(kwargs))

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/sessions.py", line 602, in get

INTERNALERROR>     return self.request("GET", url, **kwargs)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/sessions.py", line 589, in request

INTERNALERROR>     resp = self.send(prep, **send_kwargs)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/sessions.py", line 703, in send

INTERNALERROR>     r = adapter.send(request, **kwargs)

INTERNALERROR>   File "/var/lib/jenkins/workspace/Development_sm_master/gravity/.nox/lib/python3.10/site-packages/requests/adapters.py", line 534, in send

INTERNALERROR>     raise InvalidURL(e, request=request)

INTERNALERROR> requests.exceptions.InvalidURL: Not supported URL scheme http+docker

Expected Result

Normal initalization of docker client

Actual Result

Stack trace posted above

Reproduction Steps

mkvirtualenv debug_issue
pip install docker
pip install 'requests>=2.32.0'
python

import docker
docker.from_env()

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.12"
  },
  "platform": {
    "release": "6.5.0-28-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "30000020"
  },
  "urllib3": {
    "version": "2.2.1"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

[ShawnHardwick]

I believe this was introduce with this line change which started enforcing the URL scheme: https://github.com/psf/requests/commit/c0813a2d910ea6b4f8438b91d315b8d181302356#diff-a4598bf8444e0feca489e60704c809c0dbfc47b90c4364216c931b497b3312d7R512

The issue is also being tracked over in docker-py repo: https://github.com/docker/docker-py/issues/3256

[nateprewitt]

Hi @joshzcold, could you clarify more about the expected functionality with a http+docker scheme vs calling the URI with just http. Is docker-py using a custom Requests adapter or was this just being passed as generated to Requests?

This appears to be a byproduct of fixing CVE-2024-35195. urllib3 is expecting the scheme to be either http or https here in the context construction.

[nateprewitt]

It looks like there's a proposed patch for docker here. Their sub-adapter seems to reasonably have been relying on implementing get_connection. We'll use this issue to track the change in #6655 regarding get_connection and use the docker issue linked about the docker adapter.

[felixfontein]

@nateprewitt do you think the fix in https://github.com/docker/docker-py/pull/3257 is good enough for now, or do you have a suggestion for a better way to work around this problem, at least for now? I guess it would be great if Docker SDK for Python wouldn't need to depend on internal behavior of requests, but changing that in a proper way likely needs a larger rewrite / rearchitecturing of that part of Docker SDK for Python...

[joshzcold]

Since it looks like docker-py needs apply the fix on their end I believe we can close this PR.

[nateprewitt]

We wrote the #6655 fix trying to avoid breaking users that were relying on the existing behavior of get_connection in their custom Adapters but missed the other side where they implemented a custom get_connection but were still using the default send.

I don't think what docker-py was doing is unreasonable, but unfortunately I don't see a clear way to mitigate this in Requests while maintaining the security fix. https://github.com/docker/docker-py/pull/3257 seems like the least invasive option at the moment.

I have a hunch docker-py may not be the only package that hits this issue so I'd like to track these kinds of problems to evaluate if we need to take further action on our side. I'll leave this open for now to gather any other cases.

[mainland]

requests-unixsocket, which implements the http+unix scheme, is also broken by this change.

[sigmavirus24]

I would guess requests-file or whichever fork of that is still maintained would be affected if anyone uses it

[felixfontein]

requests-unixsocket, which implements the http+unix scheme, is also broken by this change.

It implements it very similarly to Docker SDK for Python, i.e. it also overwrites get_connection(). Adding a _get_connection() similar to the one in https://github.com/docker/docker-py/pull/3257 should fix it as well.

[felixfontein]

I would guess requests-file or whichever fork of that is still maintained would be affected if anyone uses it

I don't know requests-file, but the code in https://github.com/dashea/requests-file overwrites send() itself (and doesn't use HTTPAdapter, but BaseAdapter), so it should not be affected. Or at least not by the same change, maybe it's affected by something else ;-)

[nateprewitt]

@felixfontein and everyone else following along, we've iterated through a few alternatives but each has similar impact on a different user group so a "fix" would only be shifting the problem.

We've decided to move _get_connection to a new public API on HTTPAdapter called get_connection_with_tls_context. Going forward this will be the expanded API signature that enables us and other libraries to properly handle TLS settings when fetching connections. The existing get_connection will be marked as deprecated and users will be pointed to the new API.

This will avoid the clunkiness of needing to override a private method and gives a clear path forward for the API. It's likely other custom adapters implementing get_connection may also suffer from something similar to CVE-2024-35195 and will need this behavior. While we didn't intend to suddenly force this shift, we appreciate everyone's patience while we get things stabilized.

I anticipate we'll have a patch staged at some point today and intend to push that out in 2.32.2 asap.

[sergiosmcr]

@felixfontein and everyone else following along, we've iterated through a few alternatives but each has similar impact on a different user group so a "fix" would only be shifting the problem.

We've decided to move _get_connection to a new public API on HTTPAdapter called get_connection_with_tls_context. Going forward this will be the expanded API signature that enables us and other libraries to properly handle TLS settings when fetching connections. The existing get_connection will be marked as deprecated and users will be pointed to the new API.

This will avoid the clunkiness of needing to override a private method and gives a clear path forward for the API. It's likely other custom adapters implementing get_connection may also suffer from something similar to CVE-2024-35195 and will need this behavior. While we didn't intend to suddenly force this shift, we appreciate everyone's patience while we get things stabilized.

I anticipate we'll have a patch staged at some point today and intend to push that out in 2.32.2 asap.

I am still facing the problem with version 2.32.2. Was the patch to fix the issue included in that release?

[nateprewitt]

@sergiosmcr the patch needs to be done in docker-py which is already staged in https://github.com/docker/docker-py/pull/3257. Requests 2.23.2 was just to make the integration point for docker-py easier. As I noted at the top, we cannot fix this in Requests. Any changes will just move the current problem on to another package using Requests.

[nateprewitt]

This should be resolved with docker-py 7.1.0, no longer requiring pinning Requests<2.32. I'll leave this open for a few more hours for comment and then plan to resolve this as completed by the end of today.

[rkkilari]

I am facing same issue with Docker-compose

Can someone help me what are the exact commands I should follow to fix the issue?

Some of the answers from the net says "pinning requests version to 2.31.0" I am a newbie, wondering what is pinning? how to do?

I am using WSL2 , Ubuntu 22 LTS. docker-compose version 1.29.2, build unknown docker 7.1.0 docker-py 1.10.6 docker-pycreds 0.4.0

Error Message I am receiving is below -->

File "/home/rkk/.local/lib/python3.10/site-packages/requests/adapters.py", line 532, in send conn = self._get_connection(request, verify, proxies=proxies, cert=cert) File "/home/rkk/.local/lib/python3.10/site-packages/requests/adapters.py", line 400, in _get_connection conn = self.poolmanager.connection_from_host( File "/home/rkk/.local/lib/python3.10/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host return self.connection_from_context(request_context) File "/home/rkk/.local/lib/python3.10/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context raise URLSchemeUnknown(scheme) urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

[felixfontein]

@rkkilari the old docker-compose Python package (which is End of Life for two years now!) does not work with Docker SDK for Python 7.1.0, which is the first version that works with Requests 2.32.x. You have to stick to docker < 7.0.0 and requests < 2.32.0.

Also please note that you installed both docker-py and docker; these are incompatible and destroy each other. Uninstall both first, and then reinstall only one of them. If you install docker, make sure to install docker < 7.0.0, since Docker SDK for Python 7 is no longer compatible with docker-compose. docker-py shouldn't be used anymore anyway, except if you need Python 2.6 compatibility.

And to your actual question: pinning means that you make sure a fixed version is installed for a package. In this case, make sure that requests is installed with version 2.31.0 (or more generally < 2.32.0). How to do that depends on how you install the Python packages. If you are using a requirements.txt file, simply replace the line with requests by requests < 2.32.0 (if there is no such line, simply add such a line).