pypa
Twine upload fails with large file, SSLEOFError, 'EOF occurred in violation of protocol' and ConnectionError: RemoteDisconnected
Problem description
I'm trying to make a test release of my PyTopo project (current git:
https://github.com/akkana/pytopo/tree/05262ed7484bd37efb7aaa4e76c401f523bd1814) but for months, I've been unable to. I try to upload with:
python -m twine upload --repository testpypi dist/*
and first it fails and retries:
WARNING Retrying (Retry(total=9, connect=5, read=None, redirect=None,
status=None)) after connection broken by 'SSLEOFError(8, 'EOF occurred
in violation of protocol (_ssl.c:2406)')': /legacy/
and then it goes to 100% but then sits there for 5-10 minutes with no output before finally failing with
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
(I'll attach the full error log).
I worked with SnoopJ on #python for a long time today, trying a Python-3.12.6 source build to make sure it wasn't a problem with Debian's 3.12.6, and a Python-3.11.2 build because that was the last build before a patch related to OP_IGNORE_UNEXPECTED_EOF went in, but always got the same result.
I wondered if it had to do with the size of the upload. I had one large data file (test/files/Surface_Ownership_10_14_2015.geojson, 36M) in the tarball, so I tried removing it (it's only used for unit tests) and the error went away.
I've had that file in the source tarball for several years, and it used to work fine (I've released several versions of the package during that time), so something has apparently changed that has made testpypi and/or twine less tolerant of large uploads, and maybe of OpenSSL 3.x protocol errors. And the error message gives no hint of what the real problem is.
This may be related to the following other issues: https://github.com/pypa/twine/issues/273 https://github.com/pypa/packaging-problems/issues/696 https://github.com/python/cpython/issues/87960
Full error log here; twineerrs.txt
a patch related to OP_IGNORE_UNEXPECTED_EOF went in
To expand on the OpenSSL parts of this a bit:
OpenSSL changed how they respond to an unexpected EOF between 1.1.1 and 3.x (the observed error is from OpenSSL 3.3.2 3 Sep 2024 specifically), the behavior used to be non-fatal and now it is fatal. See this discussion on the OpenSSL repo: https://github.com/openssl/openssl/discussions/22690#discussioncomment-7534909
I suspect this means that Test PyPI is running a TLS implementation that has some gear-grinding with OpenSSL 3.x. That is, it sometimes sends EOF without first sending close_notify. I'm not sure how the large file interacts with that.
I suggested that @akkana try a build of 3.11.2 because of an upstream change that was present in 3.11.2 but removed in 3.11.3 that enabled the SSL_OP_IGNORE_UNEXPECTED_EOF option, which the OpenSSL discussion seems to suggest would work around the new behavior. Debian sid (the distro at play here) switched from Python 3.11.2 to Python 3.11.4 on 25 June 2023, and she said on IRC that this lines up with the timeline when she started seeing the problem.
We confirmed that her build of 3.11.2 did set this flag in the default SSLContext, but it didn't seem to have any effect on the behavior of twine. Possibly some interaction between twine, requests, urllib3 means that the SSL context does not include this option even when the underlying Python's default context has it.
I think at this point someone from PyPA who knows their TLS better than we do should have a look.
I'm observing the same issue with our package cufinufft https://github.com/flatironinstitute/cufinufft. I've tried this with two configurations:
- Python 3.10.12 (Ubuntu) and twine version 6.0.1 (keyring: 23.5.0, pkginfo: 1.8.3, requests: 2.25.1, requests-toolbelt: 0.9.1, urllib3: 1.26.5)
- Python 3.12.1 (compiled from source) and twine version 6.0.1 (keyring: 25.5.0, pkginfo: 1.12.0, requests: 2.32.3, requests-toolbelt: 1.0.0, urllib3: 2.2.3)
Both give me the following error
$ twine upload --verbose *
Uploading distributions to https://upload.pypi.org/legacy/
INFO cufinufft-2.3.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (159.1 MB)
INFO finufft-2.3.1-py3-none-macosx_13_0_arm64.whl (2.4 MB)
INFO finufft-2.3.1-py3-none-macosx_13_0_x86_64.whl (4.0 MB)
INFO finufft-2.3.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)
INFO finufft-2.3.1-py3-none-win_amd64.whl (5.0 MB)
INFO cufinufft-2.3.1.tar.gz (3.7 MB)
INFO finufft-2.3.1.tar.gz (3.7 MB)
INFO username set by command options
INFO Querying keyring for password
Enter your API token:
INFO username: __token__
INFO password: <hidden>
Uploading cufinufft-2.3.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
WARNING Retrying (Retry(total=9, connect=5, read=None, redirect=None, status=None)) after connection broken by 'SSLEOFError(8, 'EOF
occurred in violation of protocol (_ssl.c:2406)')': /legacy/
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 166.8/166.8 MB • 04:35 • 502.8 kB/s
Traceback (most recent call last):
File "/home/janden/venv/twine312/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/home/janden/venv/twine312/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/home/janden/venv/twine312/lib/python3.12/site-packages/urllib3/connection.py", line 507, in getresponse
httplib_response = super().getresponse()
^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/lib/python3.12/http/client.py", line 1419, in getresponse
response.begin()
File "/opt/python/lib/python3.12/http/client.py", line 331, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/opt/python/lib/python3.12/http/client.py", line 300, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
Full error log attached.
Note that this is also quite a large file, which may be triggering the same issue. However we didn't have this issue when uploading our last release (2024-09-10).
I have also just run into this error while uploading a package to testpypi. Like OP I have a file of about 35 MB in my distribution and when I remove it from the .tar the error goes away. Did anyone figure out what was going on?
Edit: I just tried pushing to pypi and it seems to have worked. The issue seems to occur with testpypi only. Python 3.10.12 twine 6.1.0 setuptools 80.3.1
