docker
APIClient.push() with no auth config results in EOF with Docker 28.3.3
I reported this to moby/moby (https://github.com/moby/moby/issues/50614), but since it also affects Docker SDK for Python, I'd like to report it here as well.
Reproducer from https://github.com/moby/moby/issues/50614#issuecomment-3148412359:
-
Run on shell:
docker run --name registry --publish 5000:5000 --detach registry:2.8.3 docker pull hello-world:latest docker tag hello-world:latest localhost:5000/hello-world:latest -
Run in Python, with Docker SDK for Python available:
import docker list(docker.APIClient().push("localhost:5000/hello-world:latest", stream=True, decode=True))
This results in:
Traceback (most recent call last):
File "/usr/lib/python3.13/http/client.py", line 579, in _get_chunk_left
chunk_left = self._read_next_chunk_size()
File "/usr/lib/python3.13/http/client.py", line 546, in _read_next_chunk_size
return int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.13/http/client.py", line 597, in _read_chunked
while (chunk_left := self._get_chunk_left()) is not None:
~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.13/http/client.py", line 581, in _get_chunk_left
raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 779, in _error_catcher
yield
File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 904, in _raw_read
data = self._fp_read(amt, read1=read1) if not fp_closed else b""
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 887, in _fp_read
return self._fp.read(amt) if amt is not None else self._fp.read()
~~~~~~~~~~~~~^^^^^
File "/usr/lib/python3.13/http/client.py", line 473, in read
return self._read_chunked(amt)
~~~~~~~~~~~~~~~~~~^^^^^
File "/usr/lib/python3.13/http/client.py", line 609, in _read_chunked
raise IncompleteRead(b''.join(value)) from exc
http.client.IncompleteRead: IncompleteRead(0 bytes read)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<python-input-1>", line 1, in <module>
list(docker.APIClient().push("localhost:5000/hello-world:latest", stream=True, decode=True))
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/docker-py/docker/api/client.py", line 360, in _stream_helper
yield from json_stream(self._stream_helper(response, False))
File "/path/to/docker-py/docker/utils/json_stream.py", line 60, in split_buffer
for data in stream_as_text(stream):
~~~~~~~~~~~~~~^^^^^^^^
File "/path/to/docker-py/docker/utils/json_stream.py", line 16, in stream_as_text
for data in stream:
^^^^^^
File "/path/to/docker-py/docker/api/client.py", line 365, in _stream_helper
data = reader.read(1)
File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 980, in read
data = self._raw_read(amt)
File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 903, in _raw_read
with self._error_catcher():
~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.13/contextlib.py", line 162, in __exit__
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "/usr/lib/python3.13/site-packages/urllib3/response.py", line 806, in _error_catcher
raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
A simple workaround is to pass auth_config={}:
list(docker.APIClient().push("localhost:5000/hello-world:latest", stream=True, decode=True, auth_config={}))
This bug can be fixed on Docker SDK for Python's side with the following patch:
diff --git a/docker/api/image.py b/docker/api/image.py
index 85109473..6e28894e 100644
--- a/docker/api/image.py
+++ b/docker/api/image.py
@@ -484,6 +484,8 @@ class ImageApiMixin:
header = auth.get_config_header(self, registry)
if header:
headers['X-Registry-Auth'] = header
+ else:
+ headers['X-Registry-Auth'] = auth.encode_header({})
else:
log.debug('Sending supplied auth config')
headers['X-Registry-Auth'] = auth.encode_header(auth_config)
I had a similar problem, and adding auth_config={} solved it (thanks for the advice). Everything was working correctly until some operating system upgrades. Our team doesn’t know what exactly changed, but we now see this problem on Ubuntu and Fedora.
We know that it worked on Fedora (version from 06.2025) and on Ubuntu 24.04 (until 08.2025) without any problems:
client = docker.from_env()
client.images.push(
repository=repository,
tag=version,
)
But on 26.08.2025 we noticed that our tests stopped working:
> push_details = client.images.push(
repository=remote_tsxlib_testing_package_image_docker_package.image_tag,
tag=remote_tsxlib_testing_package_image_docker_package.version,
)
src/python/integration_tests/running_scenarios/common/infrastructure.py:859:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/models/images.py:478: in push
return self.client.api.push(repository, tag=tag, **kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/api/image.py:491: in push
response = self._post_json(
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/api/client.py:303: in _post_json
return self._post(url, data=json.dumps(data2), **kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/utils/decorators.py:44: in inner
return f(self, *args, **kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/docker/api/client.py:242: in _post
return self.post(url, **self._set_request_timeout(kwargs))
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/sessions.py:637: in post
return self.request("POST", url, data=data, json=json, **kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/sessions.py:589: in request
resp = self.send(prep, **send_kwargs)
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/sessions.py:746: in send
r.content
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/models.py:902: in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def generate():
# Special case for urllib3.
if hasattr(self.raw, "stream"):
try:
yield from self.raw.stream(chunk_size, decode_content=True)
except ProtocolError as e:
> raise ChunkedEncodingError(e)
E requests.exceptions.ChunkedEncodingError: Response ended prematurely
/home/nightfoxer/.cache/pants/named_caches/pex_root/venvs/1/s/a448a1bf/venv/lib/python3.10/site-packages/requests/models.py:822: ChunkedEncodingError
We use a container with a Docker registry, and there were no logs when we tried pushing using the Docker SDK — but everything worked with the Docker CLI.
When I checked the Docker daemon logs, I found error messages http: panic serving @: runtime error: invalid memory address or nil pointer dereference (attached below).
.
At the same time it has been working od the dev containers with the same Ubuntu and docker engine versions that was running on my computer.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.3 LTS
Release: 24.04
Codename: noble
dpkg-query -W libsystemd0
dpkg-query -W libc6
dpkg-query -W libcap2
libsystemd0:amd64 255.4-1ubuntu8.10
libsystemd0:i386 255.4-1ubuntu8.10
libc6:amd64 2.39-0ubuntu8.5
libc6:i386 2.39-0ubuntu8.5
libcap2:amd64 1:2.66-5ubuntu2.2
libcap2:i386 1:2.66-5ubuntu2.2
dpkg-query -W libssl3
libssl3:amd64 3.0.10-1ubuntu2.3
libssl3:i386 3.0.10-1ubuntu2.3
Libraries that uses docker are the same on my system and in my dev container, but as I sad it had been worked correctly only in dev. And the same situation we had with Fedora and dev container that was ranning (container with ubuntu).
I'm also hit by this. The issue of adding auth_config={} to client.images.push() is that it overwrites previous authentication with a registry, so it will only work in the cases that we don't need auth for registries.
Is there a reliable way to check if the client is already authenticated to a registry?
I'm also hit by this. The issue of adding
auth_config={}toclient.images.push()is that it overwrites previous authentication with a registry, so it will only work in the cases that we don't need auth for registries.Is there a reliable way to check if the client is already authenticated to a registry?
In openstack kolla we did like so: https://github.com/openstack/kolla/commit/ddac7ca1ed393609b7bda8753db3598f1ae02648
This was a regression in v28.3.3; a fix was merged and backported https://github.com/moby/moby/pull/50738, which will be included in the v28.4 release of docker (likely to be released this week); release candidates are available in the test channel on download.docker.com; https://github.com/moby/moby/releases/tag/v28.4.0-rc.2
