internetarchive
internetarchive copied to clipboard
Published
20 hours ago •
jjjake
jjjake
Connection reset by peer when uploading large file
Uploading my 56GB ZIP file of jonesmekas.com, a website that was grabbed with grab-site, gives me the error requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', error(104, 'Connection reset by peer')), 'https://s3.us.archive.org/jonasmekas.com-2020-04-28-952295c8/jonasmekas.com-2020-04-28-952295c8.zip'). Using Ubuntu on Windows, attached file containing the full console output below and wrote the code I use for uploading to the Internet Archive below.
#!/bin/bash
ls -d */ | cut -f1 -d'/' > ./websites.txt
input="./websites.txt"
while IFS= read -r line
do
echo "Zipping folder."
zip -r -9 "$line"".zip" "./""$line"
echo "Finished zipping folder."
echo "Uploading zipped folder."
./ia upload "$line" "$line"".zip" --metadata="mediatype:data" --metadata="creator:Coloradohusky" --metadata="description:warc created using grab-site" --metadata="subject:warc;archiveteam" --metadata="language:English"
echo "Upload finished."
done < "$input"
rm websites.txt
Hmm... Is this a consistent issue for you, or is it transient? Are you experiencing it with other files you try to upload?
Happens with any large file, usually restarting the upload once or twice lets it upload successfully, but not this time
I'll try to replicate it, but have not been able to so far. Please let me know if you have any other details that might be helpful in troubleshooting this. Thanks for the report, and my apologies for the issues!
Version 1.9.3, adding --debug gives me
jonasmekas.com-2020-04-28-952295c8:
uploading jonasmekas.com-2020-04-28-952295c8.zip: 0%| | 0/55428.0 [00:00<?, ?MiB/s]Endpoint:
https://s3.us.archive.org/jonasmekas.com-2020-04-28-952295c8/jonasmekas.com-2020-04-28-952295c8.zip
HTTP Headers:
Connection:close
Accept-Encoding:gzip, deflate
Accept:*/*
User-Agent:internetarchive/1.9.3 (Linux x86_64; N; ; C6lMKgGleRQFFoSy) Python/2.7.17
x-archive-size-hint:58121100340
x-archive-keep-old-version:1
Cookie:logged-in-user=[REDACTED]%40gmail.com; expires=Thu, 22-Apr-2021 14:33:35 GMT; Max-Age=31536000; path=/; domain=.archive.org; logged-in-sig=[REDACTED]; expires=Thu, 22-Apr-2021 14:33:35 GMT; Max-Age=31536000; path=/; domain=.archive.org
Content-Length:58121100340
Authorization:LOW C6lMKgGleRQFFoSy:[REDACTED]
uploading jonasmekas.com-2020-04-28-952295c8.zip: 0%| | 0/55428.0 [00:00<?, ?MiB/s]
I don't know if this is related or not, but when uploading using Windows Command Prompt, I get this error:
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)), 'https://s3.us.archive.org/jonasmekas.com-2020-04-28-952295c8/jonasmekas.com-2020-04-28-952295c8-00008.warc.gz')
Using a slightly different command, ia upload jonasmekas.com-2020-04-28-952295c8 jonasmekas.com-2020-04-28-952295c8/ --checksum --verify, so instead of uploading a zip file I'm uploading each item in the folder separately.
I'm have the same issue with a 119MB mp3 at my account/profile here https://archive.org/details/amiga-ireland I reduced quality for smaller file sizes and got these results: 112MB - same result 99MB - same result 90.4MB - same result. I'm using macOS.
Edit/Update: I tried ia on Ubuntu 20.04 and the got the same error.
Traceback (most recent call last):-Viral.mp3: [###### ] 26/120 - 00:01:57
File "/Users/amigaireland/ia/.bootstrap/_pex/pex.py", line 367, in execute
File "/Users/amigaireland/ia/.bootstrap/_pex/pex.py", line 293, in _wrap_coverage
File "/Users/amigaireland/ia/.bootstrap/_pex/pex.py", line 325, in _wrap_profiling
File "/Users/amigaireland/ia/.bootstrap/_pex/pex.py", line 410, in _execute
File "/Users/amigaireland/ia/.bootstrap/_pex/pex.py", line 468, in execute_entry
File "/Users/amigaireland/ia/.bootstrap/_pex/pex.py", line 486, in execute_pkg_resources
File "/Users/amigaireland/.pex/install/internetarchive-1.8.5-py2.py3-none-any.whl.0749beeb0c64b2168053f35fe9c039127438232f/internetarchive-1.8.5-py2.py3-none-any.whl/internetarchive/cli/ia.py", line 159, in main
sys.exit(ia_module.main(argv, session))
File "/Users/amigaireland/.pex/install/internetarchive-1.8.5-py2.py3-none-any.whl.0749beeb0c64b2168053f35fe9c039127438232f/internetarchive-1.8.5-py2.py3-none-any.whl/internetarchive/cli/ia_upload.py", line 224, in main
for _r in _upload_files(item, files, upload_kwargs):
File "/Users/amigaireland/.pex/install/internetarchive-1.8.5-py2.py3-none-any.whl.0749beeb0c64b2168053f35fe9c039127438232f/internetarchive-1.8.5-py2.py3-none-any.whl/internetarchive/cli/ia_upload.py", line 86, in _upload_files
response = item.upload(files, **upload_kwargs)
File "/Users/amigaireland/.pex/install/internetarchive-1.8.5-py2.py3-none-any.whl.0749beeb0c64b2168053f35fe9c039127438232f/internetarchive-1.8.5-py2.py3-none-any.whl/internetarchive/item.py", line 889, in upload
request_kwargs=request_kwargs)
File "/Users/amigaireland/.pex/install/internetarchive-1.8.5-py2.py3-none-any.whl.0749beeb0c64b2168053f35fe9c039127438232f/internetarchive-1.8.5-py2.py3-none-any.whl/internetarchive/item.py", line 718, in upload_file
**request_kwargs)
File "/Users/amigaireland/.pex/install/internetarchive-1.8.5-py2.py3-none-any.whl.0749beeb0c64b2168053f35fe9c039127438232f/internetarchive-1.8.5-py2.py3-none-any.whl/internetarchive/session.py", line 390, in send
raise e
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', error(32, 'Broken pipe')), 'https://s3.us.archive.org/Going_Viral/AmigaIreland-S4E4-Going-Viral.mp3')
uploading AmigaIreland-S4E4-Going-Viral.mp3: [################################] 26/120 - 00:00:32
I get a similar result on Ubuntu 20.04 with their repo version of ia. The items in the stack trace are a little different, but the end result is the same.
Traceback (most recent call last):-Viral.mp3: [############### ] 58/120 - 00:00:35
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1240, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1286, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1235, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1045, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 967, in send
self.sock.sendall(data)
File "/usr/lib/python3.8/ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "/usr/lib/python3.8/ssl.py", line 1173, in send
return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 702, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1240, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1286, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1235, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1045, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 967, in send
self.sock.sendall(data)
File "/usr/lib/python3.8/ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "/usr/lib/python3.8/ssl.py", line 1173, in send
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/ia", line 11, in <module>
sys.exit(main())
File "/usr/lib/python3/dist-packages/internetarchive/cli/ia.py", line 168, in main
sys.exit(ia_module.main(argv, session))
File "/usr/lib/python3/dist-packages/internetarchive/cli/ia_upload.py", line 224, in main
for _r in _upload_files(item, files, upload_kwargs):
File "/usr/lib/python3/dist-packages/internetarchive/cli/ia_upload.py", line 86, in _upload_files
response = item.upload(files, **upload_kwargs)
File "/usr/lib/python3/dist-packages/internetarchive/item.py", line 1007, in upload
resp = self.upload_file(body,
File "/usr/lib/python3/dist-packages/internetarchive/item.py", line 848, in upload_file
response = self.session.send(prepared_request,
File "/usr/lib/python3/dist-packages/internetarchive/session.py", line 492, in send
raise e
File "/usr/lib/python3/dist-packages/internetarchive/session.py", line 489, in send
reraise_modify(e, e.request.url, prepend=False)
File "/usr/lib/python3/dist-packages/internetarchive/utils.py", line 310, in reraise_modify
six.reraise(ExceptClass,
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/internetarchive/session.py", line 486, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', BrokenPipeError(32, 'Broken pipe')), 'https://s3.us.archive.org/Going_Viral/AmigaIreland-S4E4-Going-Viral.mp3')
uploading AmigaIreland-S4E4-Going-Viral.mp3: [################################] 58/120 - 00:00:36
I finally got it to work by uploading everything except the mp3 first (./ia upload My_Item thumb.jpg ...... ) and then a seperate command for the mp3 (./ia upload My_Item mytrack.mp3). I was back on macOS for this.
Hi I also get this error consistently when uploading a large batch of files. The first few files (amounting to around one TB) succeed, but then they start to fail.
uploading /asd/xyz.mp4: 100%|█| 1861/1861 [5:35:13<00:00, 10.
ERROR:internetarchive.session:(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), 'https://s3.us.archive.org/asd-asd/asd/xyz.mp4')
Traceback (most recent call last):
File "C:\Program Files\Python37\lib\site-packages\urllib3\connectionpool.py", line 672, in urlopen
chunked=chunked,
File "C:\Program Files\Python37\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Program Files\Python37\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "C:\Program Files\Python37\lib\http\client.py", line 1344, in getresponse
response.begin()
File "C:\Program Files\Python37\lib\http\client.py", line 306, in begin
version, status, reason = self._read_status()
File "C:\Program Files\Python37\lib\http\client.py", line 275, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python37\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Program Files\Python37\lib\site-packages\urllib3\connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Program Files\Python37\lib\site-packages\urllib3\util\retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Program Files\Python37\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Program Files\Python37\lib\site-packages\urllib3\connectionpool.py", line 672, in urlopen
chunked=chunked,
File "C:\Program Files\Python37\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Program Files\Python37\lib\site-packages\urllib3\connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "C:\Program Files\Python37\lib\http\client.py", line 1344, in getresponse
response.begin()
File "C:\Program Files\Python37\lib\http\client.py", line 306, in begin
version, status, reason = self._read_status()
File "C:\Program Files\Python37\lib\http\client.py", line 275, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "D:\forks\internetarchive\internetarchive\cli\ia.py", line 181, in <module>
main()
File "D:\forks\internetarchive\internetarchive\cli\ia.py", line 170, in main
sys.exit(ia_module.main(argv, session))
File "D:\forks\internetarchive\internetarchive\cli\ia_upload.py", line 245, in main
for _r in _upload_files(item, files, upload_kwargs):
File "D:\forks\internetarchive\internetarchive\cli\ia_upload.py", line 97, in _upload_files
response = item.upload(files, **upload_kwargs)
File "D:\forks\internetarchive\internetarchive\item.py", line 1124, in upload
request_kwargs=request_kwargs)
File "D:\forks\internetarchive\internetarchive\item.py", line 970, in upload_file
**request_kwargs)
File "D:\forks\internetarchive\internetarchive\session.py", line 521, in send
raise e
File "D:\forks\internetarchive\internetarchive\session.py", line 518, in send
reraise_modify(e, e.request.url, prepend=False)
File "D:\forks\internetarchive\internetarchive\utils.py", line 338, in reraise_modify
traceback)
File "C:\Program Files\Python37\lib\site-packages\six.py", line 696, in reraise
raise value
File "D:\forks\internetarchive\internetarchive\session.py", line 515, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "C:\Program Files\Python37\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "C:\Program Files\Python37\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response
')), 'https://s3.us.archive.org/asd-asd/asd/xyz.mp4')
But actually the file was uploaded successfully from the servers point of view. It shows on the webpage and also has the correct hash. So the only problem is that the server closes the connection after the file, but the client expects the connection to remain open to send more files. So in the case the client should accept the connection close and just reopen it again.
I don't know why the archive servers are behaving this way. I found this:
Currently, there is no limit on the size of files nor the number of files. However, from a systems perspective, we do not recommend files larger than 50 GBs to be uploaded or more than 1000 files, per single page.
This is because items can “break” as well as take a very long time to derive and can often timeout and fail. Some users have managed to upload files larger than 50GB’s but there is always a risk that these files will cause problems.
So op might be hitting the 50GB "soft limit" and I could be hitting the 1000 files soft limit.
EDIT: I went through some of the code in internetarchive/requests/urllib3/httplib (which is a mess of abstractions...) and from what I could figure out, the request is send successfully, but afterwards, the server does not return a response which is why the client assumes the upload failed.
My preferred way to handle this, would be to just log the error and tell the user that the upload MIGHT not have succeeded and continue with the rest of the batch. The user could then retry the full batch manually and with skipping existing files enabled, it should still verify correctly.
To do that, the RemoteDisconnectedexception would need be handled.
Also having the issue on ver 2.0.3. Seems to happen pretty randomly, one item is over 50GB the other is not. Neither is over 1000 items. Both have the same issue, but seems slightly different to what others have posted. Gives error 104 connection reset by peer. Seems it can happen at any point during the upload, sometimes immediately, and sometimes it will happen after uploading 10 or so files without issue. My connection is fine in my end, no chance of dropped packets. Hardwired out to fiber.
One of the logfiles:
uploading 20190128_Mon_Knight-HD.mp4: 100%|██████████| 4125/4125 [08:54<00:00, 7.72MiB/s]
uploading 20190129_Tue_Knight-HD.mp4: 100%|██████████| 4114/4114 [10:50<00:00, 6.32MiB/s]
uploading 20190130_Wed_Knight-HD.mp4: 100%|██████████| 4122/4122 [13:10<00:00, 5.21MiB/s]
uploading 20190131_Thu_Knight-HD.mp4: 62%|██████▏ | 2556/4125 [10:02<11:59, 2.18MiB/s]Traceback (most recent call last):
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib/python3.8/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1046, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 968, in send
self.sock.sendall(data)
File "/usr/lib/python3.8/ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "/usr/lib/python3.8/ssl.py", line 1173, in send
return self._sslobj.write(data)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nker150/.pex/installed_wheels/99b7087f0df82e0bbf7da74aa50fc1d07c8c2178/requests-2.25.1-py2.py3-none-any.whl/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/nker150/.pex/installed_wheels/d1105e15407e3e5c47e9ce7a058e7d3b20b29db2/urllib3-1.26.4-py2.py3-none-any.whl/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib/python3.8/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1046, in _send_output
self.send(chunk)
File "/usr/lib/python3.8/http/client.py", line 968, in send
self.sock.sendall(data)
File "/usr/lib/python3.8/ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "/usr/lib/python3.8/ssl.py", line 1173, in send
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nker150/ia/.bootstrap/pex/pex.py", line 483, in execute
File "/home/nker150/ia/.bootstrap/pex/pex.py", line 400, in _wrap_coverage
File "/home/nker150/ia/.bootstrap/pex/pex.py", line 431, in _wrap_profiling
File "/home/nker150/ia/.bootstrap/pex/pex.py", line 539, in _execute
File "/home/nker150/ia/.bootstrap/pex/pex.py", line 655, in execute_entry
File "/home/nker150/ia/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/cli/ia.py", line 170, in main
sys.exit(ia_module.main(argv, session))
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/cli/ia_upload.py", line 249, in main
for _r in _upload_files(item, files, upload_kwargs):
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/cli/ia_upload.py", line 97, in _upload_files
response = item.upload(files, **upload_kwargs)
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/item.py", line 1240, in upload
resp = self.upload_file(body,
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/item.py", line 1057, in upload_file
response = self.session.send(prepared_request,
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/session.py", line 559, in send
raise e
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/session.py", line 556, in send
reraise_modify(e, e.request.url, prepend=False)
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/utils.py", line 324, in reraise_modify
six.reraise(ExceptClass,
File "/home/nker150/.pex/installed_wheels/7aefa60418061ca11491cde48708e4ffb38db070/six-1.15.0-py2.py3-none-any.whl/six.py", line 703, in reraise
raise value
File "/home/nker150/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/session.py", line 553, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "/home/nker150/.pex/installed_wheels/99b7087f0df82e0bbf7da74aa50fc1d07c8c2178/requests-2.25.1-py2.py3-none-any.whl/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/nker150/.pex/installed_wheels/99b7087f0df82e0bbf7da74aa50fc1d07c8c2178/requests-2.25.1-py2.py3-none-any.whl/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), 'https://s3.us.archive.org/knight201901/20190131_Thu_Knight-HD.mp4')
uploading 20190131_Thu_Knight-HD.mp4: 62%|██████▏ | 2556/4125 [10:22<06:22, 4.11MiB/s]
This one and two other log files are attached but it's all the same theme. All files (except for a jpg item image) are ~4GB mp4 files.
This is one of the more interesting ones because it shows it was working fine for several files, then quit. I'm also using --retries=10 as used in the example, but I guess the script crashes so it never retries.
ialog1.txt ialog2.txt ialog3.txt
That happened to me, yet it still showed up on archive.org. Wasn't a huge file, can't remember, I think 66mb?
Happens to me as well. Both on 1.9.3 on Win32, and on 2.0.3 on Cygwin.
Although sometimes I get a "bad status line" error instead. Smells like something fishy is happening on the S3 endpoint, maybe something slightly out of spec or whatever
uploading somefile.zip: 4416MiB [3:21:53, 2.74s/MiB]
Traceback (most recent call last):
File "/cygdrive/d/bin/ia2/.bootstrap/pex/pex.py", line 483, in execute
File "/cygdrive/d/bin/ia2/.bootstrap/pex/pex.py", line 400, in _wrap_coverage
File "/cygdrive/d/bin/ia2/.bootstrap/pex/pex.py", line 431, in _wrap_profiling
File "/cygdrive/d/bin/ia2/.bootstrap/pex/pex.py", line 539, in _execute
File "/cygdrive/d/bin/ia2/.bootstrap/pex/pex.py", line 655, in execute_entry
File "/cygdrive/d/bin/ia2/.bootstrap/pex/pex.py", line 687, in execute_pkg_resources
File "/home/Darkstar/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/cli/ia.py", line 170, in main
sys.exit(ia_module.main(argv, session))
File "/home/Darkstar/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/cli/ia_upload.py", line 249, in main
for _r in _upload_files(item, files, upload_kwargs):
File "/home/Darkstar/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/cli/ia_upload.py", line 97, in _upload_files
response = item.upload(files, **upload_kwargs)
File "/home/Darkstar/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/item.py", line 1256, in upload
request_kwargs=request_kwargs)
File "/home/Darkstar/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/item.py", line 1059, in upload_file
**request_kwargs)
File "/home/Darkstar/.pex/installed_wheels/4f0799d9e2c59aec5855e795772b5b3ef620df65/internetarchive-2.0.3-py2.py3-none-any.whl/internetarchive/session.py", line 559, in send
raise e
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', BadStatusLine("''",)), 'https://s3.us.archive.org/someitem/somefile.zip')
I'm using v2.0.3 on Windows and also getting errors trying to upload a large file (a 41GB .DAT file), but I get 3 exceptions in a row when the error happens:
Traceback (most recent call last):
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 1257, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 1303, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 1252, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 1051, in _send_output
self.send(chunk)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 973, in send
self.sock.sendall(data)
File "C:\Program Files\Python\Python39\lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Program Files\Python\Python39\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\util\retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\packages\six.py", line 769, in reraise
raise value.with_traceback(tb)
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\urllib3\connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 1257, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 1303, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 1252, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 1051, in _send_output
self.send(chunk)
File "C:\Program Files\Python\Python39\lib\http\client.py", line 973, in send
self.sock.sendall(data)
File "C:\Program Files\Python\Python39\lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Program Files\Python\Python39\lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\twig\AppData\Roaming\Python\Python39\Scripts\ia.exe\__main__.py", line 7, in <module>
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\cli\ia.py", line 170, in main
sys.exit(ia_module.main(argv, session))
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\cli\ia_upload.py", line 249, in main
for _r in _upload_files(item, files, upload_kwargs):
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\cli\ia_upload.py", line 97, in _upload_files
response = item.upload(files, **upload_kwargs)
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\item.py", line 1206, in upload
resp = self.upload_file(filepath,
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\item.py", line 1057, in upload_file
response = self.session.send(prepared_request,
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\session.py", line 559, in send
raise e
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\session.py", line 556, in send
reraise_modify(e, e.request.url, prepend=False)
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\utils.py", line 324, in reraise_modify
six.reraise(ExceptClass,
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\six.py", line 719, in reraise
raise value
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\internetarchive\session.py", line 553, in send
r = super(ArchiveSession, self).send(request, **kwargs)
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Users\twig\AppData\Roaming\Python\Python39\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)),
'https://s3.us.archive.org/Dogecoin_Coin/Bootstrap/bootstrap.dat')
Using the WebGUI on the site itself to upload this same file, I'll sporadically get a "There is a network problem" popup with a "Resume" button, and clicking it resumes uploading from where it left off, but sometimes will upload with no issues at all. However, trying to upload it with IA, the upload fails every time.
