Request with chunked encoding doesn't respect timeout

[kchen]

trafficstars

If I use requests to send a POST request and pass in a generator (and therefore the request uses chunked encoding), the timeout parameter is not respected. If I use a fixed bytes object instead of a generator, the timeout is respected.

Expected Result

The request should time out (in the reproduction steps below, after 2 seconds).

Actual Result

The request did not time out. The reproduction steps below print "Connecting to http://localhost:15382/" and then hangs there.

Reproduction Steps

Run "nc -l 15382", and then the program below:

import requests

url = "http://localhost:15382/"

def generator():
    yield b"a"
    yield b"b"

print(f"Connecting to {url}")
try:
    requests.post(url, timeout=2, data=generator())
    print(f"Completed connecting to {url}")
except Exception as exc:
    print(f"Got {exc}")

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.5"
  },
  "platform": {
    "release": "4.15.0-111-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010104f"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": false
}

It appears that HTTPAdapter.send() uses the timeout for non-chunked requests, but not for chunked requests, so that may be the cause of this issue.

(I initially investigated this bug after finding that my POST request of a large file to an HTTPS server was timing out after about 6 minutes (when I had a 10 minute timeout set). I haven't been able to reproduce the 6 minute timeout with a simple test case, but it seems like that may consist of 1 minute to send the large file, and after sending the file, it looks like openssl may have a default 5 minute timeout if not overridden. If this analysis is correct, a fix should also ensure the openssl timeout is updated as well.)

[grove]

We also ran into this issue the other day. The default timeout seems to be infinite, so if you're unlucky the HTTP POST request will end up blocking forever.

[jakermx]

it depends on the http response header

[Uxio0]

We are observing that behaviour too. Did you manage to fix that?

[kchen]

Nope, I worked around it. (My project uses gevent, and so I wrapped the requests call in a gevent timeout.)

For unrelated reasons, though, my project is moving to httpx, and the test case above (replacing requests with httpx) works fine, so that would be another workaround.

[Uxio0]

Nope, I worked around it. (My project uses gevent, and so I wrapped the requests call in a gevent timeout.)

Really nice, we are using gevent too

For unrelated reasons, though, my project is moving to httpx, and the test case above (replacing requests with httpx) works fine, so that would be another workaround.

httpx is not working well for us, though, we can reproduce the same behaviour

Thank you for your suggestions!