streaming logs raises HTTP exception with error in chunk decoding

[TomasTomecek]

The server is docker-1.13.1.

Packages:

$ pip3 freeze | egrep (requests|urllib)
requests==2.13.0
urllib3==1.20

The code is quite simple:

for l in d.logs(c, follow=True, stream=True):
 print(l)

Traceback (most recent call last):                                                                                                            [667/2266]
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 535, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 298, in _error_catcher
    yield
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 590, in read_chunked
    self._update_chunk_length()
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 539, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 719, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 428, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 618, in read_chunked
    self._original_response.close()
  File "/usr/lib64/python3.6/contextlib.py", line 100, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 316, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tt/.dotfiles/bin/test-build", line 68, in <module>
    for l in d.logs(c, follow=True, stream=True):
  File "/usr/lib/python3.6/site-packages/docker/client.py", line 317, in _stream_raw_result
    for out in response.iter_content(chunk_size=1, decode_unicode=True):
  File "/usr/lib/python3.6/site-packages/requests/utils.py", line 389, in stream_decode_response_unicode
    for chunk in iterator:
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 722, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

[cguethle]

@TomasTomecek Did you get anywhere with this? I am actually seeing this same problem running in a container for something totally different. Trying to requests.put a file to a rest interface on another server.

[TomasTomecek]

@cguethle I haven't investigated further.

[tahamr83]

We are also encountering a similar ProtocolError while logging build logs from a remote docker daemon.

for line in docker.build(**build_params):
    if "stream" in line:
        logger.info(line.get("stream"))

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/http/client.py", line 562, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/local/lib/python3.7/http/client.py", line 529, in _read_next_chunk_size
    return int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/http/client.py", line 594, in _readinto_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/local/lib/python3.7/http/client.py", line 564, in _get_chunk_left
    raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/local/lib/python3.7/http/client.py", line 465, in read
    n = self.readinto(b)
  File "/usr/local/lib/python3.7/http/client.py", line 499, in readinto
    return self._readinto_chunked(b)
  File "/usr/local/lib/python3.7/http/client.py", line 610, in _readinto_chunked
    raise IncompleteRead(bytes(b[0:total_bytes]))
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 731, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/builder/services/build.py", line 41, in build_agent_container_from_spec
    build_image(build_params, docker)
  File "/app/builder/services/build.py", line 65, in build_image
    for line in docker.build(**build_params):
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 346, in _stream_helper
    for chunk in json_stream(self._stream_helper(response, False)):
  File "/usr/local/lib/python3.7/site-packages/docker/utils/json_stream.py", line 66, in split_buffer
    for data in stream_as_text(stream):
  File "/usr/local/lib/python3.7/site-packages/docker/utils/json_stream.py", line 22, in stream_as_text
    for data in stream:
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 352, in _stream_helper
    data = reader.read(1)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Can anyone point me on how to catch this exception and simply continue with the build process?

Python Version: 3.7

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9

[tahamr83]

@TomasTomecek any way we can mitigate this? It's been very frequent with us, we have tasks that build our containers and this happens quite often. Only way I see is to restart the task.