s3cmd
s3cmd copied to clipboard
s3tools
ERROR: Download of 'xxxx' failed (Reason: Unknown OsError 0)
Environment: Digital Ocean Space Bucket: 20,000+ files Size: 6gb
Steps to reproduce: s3cmd sync s3://$AWS_BUCKETNAME/ /tmp/$PGDATABASE-$TODAY/
Result:
Apr 01 16:18:07.291 | odoo-backup-20210401-2308-gskbn | Get the filestore from Production
Apr 01 16:18:27.527 | odoo-backup-20210401-2308-gskbn | download: 's3://odoo-customer/000056ce50f18900f00dcec30b6f59963b519d28' -> '/tmp/odoo14-20210401/000056ce50f18900f00dcec30b6f59963b519d28' [1 of 28822]
Apr 01 16:18:28.608 | odoo-backup-20210401-2308-gskbn |
23267 of 23267 100% in 1s 21.03 kB/s
23267 of 23267 100% in 1s 21.03 kB/s done
Apr 01 16:18:28.610 | odoo-backup-20210401-2308-gskbn | download: 's3://odoo-customer/0000b59cd02262eeab9b0f7d90e3afc6ef0dfc4b' -> '/tmp/odoo14-20210401/0000b59cd02262eeab9b0f7d90e3afc6ef0dfc4b' [2 of 28822]
Apr 01 16:18:29.982 | odoo-backup-20210401-2308-gskbn |
4431 of 4431 100% in 1s 3.15 kB/s
4431 of 4431 100% in 1s 3.15 kB/s done
Apr 01 16:18:29.982 | odoo-backup-20210401-2308-gskbn | download: 's3://odoo-customer/000171e8f80d613f4fc98e5a9ea35cdcd06d99f0' -> '/tmp/odoo14-20210401/000171e8f80d613f4fc98e5a9ea35cdcd06d99f0' [3 of 28822]
Apr 01 16:18:31.019 | odoo-backup-20210401-2308-gskbn | ERROR: Download of '000171e8f80d613f4fc98e5a9ea35cdcd06d99f0' failed (Reason: Unknown OsError 0)
Apr 01 16:18:31.020 | odoo-backup-20210401-2308-gskbn | ERROR: Exiting now because of fatal error
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn |
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | An unexpected error has occurred.
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | Please try reproducing the error using
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | the latest s3cmd code from the git master
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | branch found at:
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | https://github.com/s3tools/s3cmd
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | and have a look at the known issues list:
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | If the error persists, please report the
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | following lines (removing any private
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | info as necessary) to:
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | [email protected]
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn |
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn |
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn |
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | Invoked as: /usr/bin/s3cmd sync s3://odoo-customer/ /tmp/odoo14-20210401/
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | Problem: RemoteDisconnected: Remote end closed connection without response
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | S3cmd: 2.0.2
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | python: 3.8.5 (default, Jul 28 2020, 12:59:40)
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | [GCC 9.3.0]
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | environment LANG=C.UTF-8
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn |
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | Traceback (most recent call last):
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/bin/s3cmd", line 3092, in <module>
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | rc = main()
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/bin/s3cmd", line 3001, in main
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | rc = cmd_func(args)
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/bin/s3cmd", line 1887, in cmd_sync
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | return cmd_sync_remote2local(args)
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/bin/s3cmd", line 1475, in cmd_sync_remote2local
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | ret, seq, size_transferred = _download(remote_list, seq, remote_count + update_count, size_transferred, dir_cache)
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/bin/s3cmd", line 1321, in _download
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | response = s3.object_get(uri, dst_stream, dst_file, extra_label = seq_label)
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/lib/python3/dist-packages/S3/S3.py", line 716, in object_get
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | response = self.recv_file(request, stream, labels, start_position)
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/lib/python3/dist-packages/S3/S3.py", line 1655, in recv_file
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | http_response = conn.c.getresponse()
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | response.begin()
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/lib/python3/dist-packages/S3/Custom_httplib3x.py", line 53, in httpresponse_patched_begin
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | version, status, reason = self._read_status()
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | raise RemoteDisconnected("Remote end closed connection without"
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | http.client.RemoteDisconnected: Remote end closed connection without response
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn |
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | An unexpected error has occurred.
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | Please try reproducing the error using
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | the latest s3cmd code from the git master
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | branch found at:
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | https://github.com/s3tools/s3cmd
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | and have a look at the known issues list:
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | If the error persists, please report the
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | above lines (removing any private
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | info as necessary) to:
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | [email protected]
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Apr 01 16:18:31.021 | odoo-backup-20210401-2308-gskbn |
Apr 01 16:18:31.049 | odoo-backup-20210401-2308-gskbn | error: Could not open logfile
Apr 01 16:18:31.049 | odoo-backup-20210401-2308-gskbn | caused by: No such file or directory (os error 2)
Apr 01 16:18:31.049 | odoo-backup-20210401-2308-gskbn |
Apr 01 16:18:31.049 | odoo-backup-20210401-2308-gskbn | Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
Apr 01 16:18:31.049 | odoo-backup-20210401-2308-gskbn | Please attach the full debug log to all bug reports.
Expected Results: All the files get transferred with out any errors.
I'm having the same issue on a large bucket/space:
Traceback (most recent call last):
File "/usr/bin/s3cmd", line 3092, in <module>
rc = main()
File "/usr/bin/s3cmd", line 3001, in main
rc = cmd_func(args)
File "/usr/bin/s3cmd", line 1887, in cmd_sync
return cmd_sync_remote2local(args)
File "/usr/bin/s3cmd", line 1475, in cmd_sync_remote2local
ret, seq, size_transferred = _download(remote_list, seq, remote_count + update_count, size_transferred, dir_cache)
File "/usr/bin/s3cmd", line 1321, in _download
response = s3.object_get(uri, dst_stream, dst_file, extra_label = seq_label)
File "/usr/lib/python3/dist-packages/S3/S3.py", line 716, in object_get
response = self.recv_file(request, stream, labels, start_position)
File "/usr/lib/python3/dist-packages/S3/S3.py", line 1655, in recv_file
http_response = conn.c.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/lib/python3/dist-packages/S3/Custom_httplib3x.py", line 53, in httpresponse_patched_begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
Did you find any workaround @JevinD ?
@rayrutjes You are also encountering the problem with Digital Ocean Space service?
Yes, also with DO Space. When I reduce the number of targeted files it works with no problem, but if I pick folders with too much files it crashes sometimes in the middle or then at the very start and doesn't even manage to process a single file.
I've switched to rclone and it seems to work though 🤔
I had the problem with DO as well. And now even if I try to get only the remaining part, it crashes anyway.
