internetarchive
internetarchive copied to clipboard
jjjake
Consistent OpenSSL.SSL.SysCallError: (32, 'EPIPE')
I'm always getting this error, after some 300 MiB uploaded at about 110 Mbps:
$ ia upload -S metadata.csv
wikimediacommons-2014-09-21:
Traceback (most recent call last): ] 378/70937 - 01:33:02
File "/usr/local/bin/ia", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/internetarchive/cli/ia.py", line 155, in main
sys.exit(ia_module.main(argv, session))
File "/usr/local/lib/python2.7/dist-packages/internetarchive/cli/ia_upload.py", line 236, in main
session)
File "/usr/local/lib/python2.7/dist-packages/internetarchive/cli/ia_upload.py", line 78, in _upload_files
response = item.upload(files, **upload_kwargs)
File "/usr/local/lib/python2.7/dist-packages/internetarchive/item.py", line 751, in upload
request_kwargs=request_kwargs)
File "/usr/local/lib/python2.7/dist-packages/internetarchive/item.py", line 613, in upload_file
**request_kwargs)
File "/usr/local/lib/python2.7/dist-packages/internetarchive/session.py", line 343, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 423, in send
timeout=timeout
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
chunked=chunked)
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 361, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python2.7/httplib.py", line 1057, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 901, in _send_output
self.send(message_body)
File "/usr/lib/python2.7/httplib.py", line 870, in send
self.sock.sendall(datablock)
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 292, in sendall
sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 281, in _send_until_done
return self.connection.send(data)
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1254, in send
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 1163, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')
uploading 2014-09-21.zip: [################################] 378/70937 - 00:00:30
I have requests 2.10.0-2, urllib3 1.15.1-2, ia 1.1.0 and
ii libgnutls-openssl27:amd64 3.5.3-5ubuntu1 amd64 GNU TLS library - OpenSSL wrapper
ii openssl 1.0.2g-1ubuntu9 amd64 Secure Sockets Layer toolkit - cryptographic utility
ii python-openssl 16.1.0-1 all Python 2 wrapper around the OpenSSL library
Upstream bug should be https://github.com/kennethreitz/requests/issues/2543 / https://github.com/shazow/urllib3/issues/367 , can some retry/resume be added?
I have no such issue with ias3upload...
Thanks for the report and detailed info @nemobis.
I've been seeing reports of this issue from others as well (#176, #80).
It looks like you were eventually able to get it uploaded? I see this exists:
http://archive.org/download/wikimediacommons-2014-09-21/2014-09-21.zip
Did you do anything different to eventually get it uploaded, or did you just keep retrying?
Did you do anything different to eventually get it uploaded, or did you just keep retrying?
I switched back to ias3upload.pl... This issue is currently my only blocker to completely supersede that little wonder of a script. :-)
I'm getting this right now with 2.18.4 ... I initially suspected some sort of timeout by the receiving server, however, similar data loads from different request clients seem to be handling fine...
There's a certain request size that triggers this. I can run the same script on smaller data and it will work fine.
Some aspect of this issue plays out entirely on the client side. I can arbitrarily decrease the size of my request, and upon crossing the magic threshold, I go from multi-minute hang, followed by SysCallError, to nearly instant completion, 200 OK
In this sense I think I've ruled out a potential server timeout as a root cause of this issue.
FWIF, my request data (post) is a handful of 500-2000kb strings.
confirmed switched to a different python lib, same request works fine
Hi, @jasconius we are experiencing the same error, could you please share what is the lib you switched to and what version and if that required some changes in the internetarchive python lib APIs?
Getting a very similar issue here:
Traceback (most recent call last):
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/var/www/courtlistener/cl/corpus_importer/tasks.py", line 352, in upload_pdf_to_ia
'licenseurl': 'https://www.usa.gov/government-works',
File "/var/www/courtlistener/cl/corpus_importer/tasks.py", line 437, in upload_to_ia
queue_derive=False, verify=True)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/internetarchive/item.py", line 842, in upload
request_kwargs=request_kwargs)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/internetarchive/item.py", line 692, in upload_file
**request_kwargs)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/internetarchive/session.py", line 370, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python2.7/httplib.py", line 1017, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1051, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 868, in _send_output
self.send(message_body)
File "/usr/lib/python2.7/httplib.py", line 837, in send
self.sock.sendall(datablock)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 218, in sendall
sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 207, in _send_until_done
return self.connection.send(data)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1630, in send
self._raise_ssl_error(self._ssl, result)
File "/var/www/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1538, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
SysCallError: (32, 'EPIPE')
What version are you on @mlissner?
I'm still trying to get to the bottom of this issue, but it seems there is an archive.org bug where our servers hang up the phone before sending the full response to the client. The library I use for HTTP requests, requests, doesn't seem to have a way to turn off connection pooling. This seems to cause an issue when archive.org hangs up the phone early, and the connection gets returned to the pool in a borked state.
There is a change in v1.7.7 that I was hoping would help get around this issue, until the bug can be addressed on archive.org. I added a Connection:close header to each request, to make sure the connection gets closed before being returned to the pool.
We're on 1.7.6. I'll upgrade and see if the error happens again tonight.
It seems to be intermittent, but we retry failed uploads for three consecutive nights to deal with network weirdness, so if it's repeatable, we should see it again tonight. If it doesn't happen tonight either the version bump worked or the network became un-weird. If you don't hear from me, that means it worked!
Sorry, no, I take that back. The server, where this happened, is on 1.7.7. (I accidentally checked my dev machine's version at first.)
So this is definitely still an issue.
That's very helpful, thanks @mlissner. Sorry for the issues, I'll continue to investigate.
@mlissner Are you able to try with HTTP rather than HTTPS? I'm wondering if this is HTTPS related, or not.
I guess theoretically? How do I turn off HTTPS in the internetarchive library?
@mlissner You would need to add {'general': {'secure': False}} to your config.
here's a patch, for example:
diff --git a/cl/corpus_importer/tasks.py b/cl/corpus_importer/tasks.py
index 812a1357..ff187ba1 100644
--- a/cl/corpus_importer/tasks.py
+++ b/cl/corpus_importer/tasks.py
@@ -392,7 +392,11 @@ def upload_pdf_to_ia(self, rd_pk):
access_key = settings.IA_ACCESS_KEY
secret_key = settings.IA_SECRET_KEY
-session = ia.get_session({'s3': {
+session = ia.get_session({
+'general': {
+ 'secure': False,
+},
+'s3': {
'access': access_key,
'secret': secret_key,
}})```
Easy enough. I'll give that a whirl and will report back tomorrow if I still have errors. Thanks for the diff. Making life easy around here.
Traceback (most recent call last): File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request conn.request(method, url, **httplib_request_kw) File "/data/data/com.termux/files/usr/lib/python3.7/http/client.py", line 1229, in request self._send_request(method, url, body, headers, encode_chunked) File "/data/data/com.termux/files/usr/lib/python3.7/http/client.py", line 1275, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/data/data/com.termux/files/usr/lib/python3.7/http/client.py", line 1224, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/data/data/com.termux/files/usr/lib/python3.7/http/client.py", line 1055, in _send_output self.send(chunk) File "/data/data/com.termux/files/usr/lib/python3.7/http/client.py", line 977, in send self.sock.sendall(data) File "/data/data/com.termux/files/usr/lib/python3.7/ssl.py", line 1015, in sendall v = self.send(byte_view[count:]) File "/data/data/com.termux/files/usr/lib/python3.7/ssl.py", line 984, in send return self._sslobj.write(data) BrokenPipeError: [Errno 32] Broken pipe
What is wrong? Thanks
im on internetarchive 1.8.4, Python 3.7.3, Arch Linux is also getting this error.
File "/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 320, in _send_until_done
return self.connection.send(data)
File "/home/archuser/.local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1729, in send
self._raise_ssl_error(self._ssl, result)
File "/home/archuser/.local/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1631, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1055, in _send_output
self.send(chunk)
File "/usr/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
File "/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 331, in sendall
sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
File "/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 326, in _send_until_done
raise SocketError(str(e))
OSError: (32, 'EPIPE')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1055, in _send_output
self.send(chunk)
File "/usr/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
File "/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 331, in sendall
sent = self._send_until_done(data[total_sent:total_sent + SSL_WRITE_BLOCKSIZE])
File "/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 326, in _send_until_done
raise SocketError(str(e))
urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError("(32, 'EPIPE')"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/archuser/.local/bin/ia", line 10, in <module>
sys.exit(main())
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/cli/ia.py", line 159, in main
sys.exit(ia_module.main(argv, session))
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/cli/ia_upload.py", line 252, in main
session)
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/cli/ia_upload.py", line 86, in _upload_files
response = item.upload(files, **upload_kwargs)
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/item.py", line 890, in upload
request_kwargs=request_kwargs)
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/item.py", line 720, in upload_file
**request_kwargs)
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/session.py", line 373, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(32, 'EPIPE')"))
if i use ia upload with --insecure . I get
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1055, in _send_output
self.send(chunk)
File "/usr/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1055, in _send_output
self.send(chunk)
File "/usr/lib/python3.7/http/client.py", line 977, in send
self.sock.sendall(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/archuser/.local/bin/ia", line 10, in <module>
sys.exit(main())
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/cli/ia.py", line 159, in main
sys.exit(ia_module.main(argv, session))
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/cli/ia_upload.py", line 252, in main
session)
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/cli/ia_upload.py", line 86, in _upload_files
response = item.upload(files, **upload_kwargs)
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/item.py", line 890, in upload
request_kwargs=request_kwargs)
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/item.py", line 720, in upload_file
**request_kwargs)
File "/home/archuser/.local/lib/python3.7/site-packages/internetarchive/session.py", line 373, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
@nemobis I am unable to replicate this with internetarchive 1.8.4 and Python 3.7.3 on Arch Linux. Can you provide the identifier and filename you're using in your command? A timestamp would be useful as well if you have it. I'm assuming this is happening consistently? Feel free to email me if you'd prefer.
I'm on requests version 2.21.0, btw. What are you on (pip freeze | grep requests should show you)?
@jjjake
- identifier is
redumpSonyPlaystation2UsaGames2018Aug01 -
iacommand isia upload --spreadsheet=/home/archuser/Downloads/ps2.csv --retries=10 --no-collection-check --verify -H x-archive-keep-old-version:0 - timestamp just now 13h50 (GMT+7)
- yes, it is happening consistently
- i get error when i run
pip freeze | grep requests
req = FrozenRequirement.from_dist(dist)
File "/usr/lib/python3.7/site-packages/pip/_internal/operations/freeze.py", line 239, in from_dist
req = dist.as_requirement()
File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2836, in as_requirement
return Requirement.parse(spec)
File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3138, in parse
req, = parse_requirements(s)
File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3085, in parse_requirements
yield Requirement(line)
File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3092, in __init__
super(Requirement, self).__init__(requirement_string)
File "/usr/lib/python3.7/site-packages/packaging/requirements.py", line 97, in __init__
requirement_string[e.loc : e.loc + 8], e.msg
packaging.requirements.InvalidRequirement: Parse error at "'-nternet'": Expected W:(abcd...)
-
sudo pip freeze | grep requestsgives merequests==2.21.0
i uninstalled and reinstalled...when i run ia configure i get this error...may be it is my system? not sure what is going on
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3.7/site-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
conn.connect()
File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 301, in connect
conn = self._new_conn()
File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fcb49da5080>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3.us.archive.org', port=443): Max retries exceeded with url: /?check_auth=1 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fcb49da5080>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/ia", line 11, in <module>
load_entry_point('internetarchive==1.8.5.dev1', 'console_scripts', 'ia')()
File "/usr/lib/python3.7/site-packages/internetarchive-1.8.5.dev1-py3.7.egg/internetarchive/cli/ia.py", line 159, in main
sys.exit(ia_module.main(argv, session))
File "/usr/lib/python3.7/site-packages/internetarchive-1.8.5.dev1-py3.7.egg/internetarchive/cli/ia_configure.py", line 72, in main
config_file_path = configure(config_file=session.config_file)
File "/usr/lib/python3.7/site-packages/internetarchive-1.8.5.dev1-py3.7.egg/internetarchive/api.py", line 578, in configure
config_file_path = config_module.write_config_file(username, password, config_file)
File "/usr/lib/python3.7/site-packages/internetarchive-1.8.5.dev1-py3.7.egg/internetarchive/config.py", line 96, in write_config_file
auth_config = get_auth_config(username, password)
File "/usr/lib/python3.7/site-packages/internetarchive-1.8.5.dev1-py3.7.egg/internetarchive/config.py", line 70, in get_auth_config
r = requests.get(u, params=p, auth=auth.S3Auth(access_key, secret_key))
File "/usr/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='s3.us.archive.org', port=443): Max retries exceeded with url: /?check_auth=1 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fcb49da5080>: Failed to establish a new connection: [Errno -2] Name or service not known'))
i uploaded to new identifier redumpSonyPlaystation2UsaGames2018Aug01Part2 and it is uploading fine....seems to be issue on archive.org side...though I'm not sure what the issue is.with redumpSonyPlaystation2UsaGames2018Aug01 ..may be there is total size limit
@thiDucTran So, I take it your issue with ia configure above was transient? Or did you just manually create your config?
I do see a lot of 403s in our logs for that identifier. Still investigating, I'll let you know if we figure anything out. Glad to hear it's working with a new identifier.
@jjjake i was able to run sudo ia configure ...i think the configure issue is related to how python/pip is in arch ..i did not install in virtual env
Hit something similar while running within docker on 1.9.0. Tried upgrading a 6.3gb zip, same problem w/ 2.0.3 on Python 3.9.4
Downloading internetarchive-2.0.3-py2.py3-none-any.whl (80 kB)
Found existing installation: internetarchive 1.9.0
It uploads 4% or so and then fails
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', OSError("(32, 'EPIPE')")), 'https://s3.us.archive.org/book_catalog_archive_2021/2021-06.zip')
Exception ignored in: <generator object tqdm.__iter__ at 0x7fb858b4f3c0>
EDIT: This was a permission error; the wrong .conf/ia.conf was configured. :)
I got this error, with the following output:
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 340, in _send_until_done
return self.connection.send(data)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1737, in send
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1046, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 968, in send
self.sock.sendall(data)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 351, in sendall
sent = self._send_until_done(
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 346, in _send_until_done
raise SocketError(str(e))
OSError: (32, 'EPIPE')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1046, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 968, in send
self.sock.sendall(data)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 351, in sendall
sent = self._send_until_done(
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 346, in _send_until_done
raise SocketError(str(e))
urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError("(32, 'EPIPE')"))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/ia", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/internetarchive/cli/ia.py", line 170, in main
sys.exit(ia_module.main(argv, session))
File "/usr/local/lib/python3.8/dist-packages/internetarchive/cli/ia_upload.py", line 250, in main
for _r in _upload_files(item, files, upload_kwargs):
File "/usr/local/lib/python3.8/dist-packages/internetarchive/cli/ia_upload.py", line 98, in _upload_files
response = item.upload(files, **upload_kwargs)
File "/usr/local/lib/python3.8/dist-packages/internetarchive/item.py", line 1206, in upload
resp = self.upload_file(filepath,
File "/usr/local/lib/python3.8/dist-packages/internetarchive/item.py", line 1057, in upload_file
response = self.session.send(prepared_request,
File "/usr/local/lib/python3.8/dist-packages/internetarchive/session.py", line 559, in send
raise e
File "/usr/local/lib/python3.8/dist-packages/internetarchive/session.py", line 556, in send
reraise_modify(e, e.request.url, prepend=False)
File "/usr/local/lib/python3.8/dist-packages/internetarchive/utils.py", line 324, in reraise_modify
six.reraise(ExceptClass,
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/local/lib/python3.8/dist-packages/internetarchive/session.py", line 553, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', OSError("(32, 'EPIPE')")), 'https://s3.us.archive.org/cygwin_repos_2020-10-24/x86/setup.ini')
Exception ignored in: <generator object tqdm.__iter__ at 0x7fd68d006c80>
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1183, in __iter__
File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1294, in close
File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1472, in display
File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1090, in __repr__
File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1434, in format_dict
TypeError: cannot unpack non-iterable NoneType object
I did do ia configure again to make sure it wasn't a credentials issue.
@hamishmb I'm cleaning up old issues. After 5 years, nobody has been able to resolve this or provide a way to replicate it. I myself have not been able to replicate it. A lot has changed in that time in both Python and this code base. A lot of the information here is outdated and not applicable today. I don't doubt this is still an issue for some, just trying to clean up. Reopened.
Il 04/02/22 00:46, jjjake ha scritto:
After 5 years, nobody has been able to resolve this or provide a way to replicate it. I myself have not been to replicate it.
Thanks for looking at old reports. I'm sorry for not being able to help more with replication. I reported this bug while uploading items to https://archive.org/details/wikimediacommons, I think, but I've not done such uploads in a while for various reasons.
I suspect it's possible to find more people able to replicate by looking for users who upload relatively big files (e.g. above 1 GB) from high latency connections (e.g. from home in Europe).
At some point I will resume such uploads and I can promise to reopen this if I see the bug again (if it gets closed), then I'll answer Jake's questions from 2019...
I'm in the UK and have some big files I can test with if that is useful. I don't know if there are any debugging options to try.
It seems to fail immediately for me. However, that was a little while ago and perhaps it has been fixed in a newer version.
The web uploader (which seems to have been in beta since 2013!) usually works fine for me so it's not a dealbreaker that this doesn't work for me.
I'm having this issue regularly, tried from multiple computers (Mac, Windows, Linux). Can't get past 1% without it dying exactly this way. Is there any fix?
