Upload fails with 408 error

[echamberlain]

trafficstars

My smaller uploads are successful, but this one keeps failing with the following error, any ideas?

Traceback (most recent call last): 295.36 KB/s, average 385.03 KB/s, eta 19:06:42.
File "/usr/local/bin/glacier-cmd", line 8, in load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')() File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 751, in main args.func(args) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 147, in wrapper return fn(_args, *_kwargs) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 300, in upload args.name, args.partsize, args.uploadid, args.resume) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, *_kwargs) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 204, in glacier_connect_wrap return func(_args, *_kwargs) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, *_kwargs) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 257, in sdb_connect_wrap return func(_args, *_kwargs) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 62, in wrapper ret = fn(_args, *_kwargs) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 1086, in upload writer.write(part) File "/Library/Python/2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glaciercorecalls.py", line 129, in write data) File "/Library/Python/2.7/site-packages/boto/glacier/layer1.py", line 625, in upload_part response_headers=response_headers) File "/Library/Python/2.7/site-packages/boto/glacier/layer1.py", line 83, in make_request raise UnexpectedHTTPResponseError(ok_responses, response) boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)

[jqdoumen]

I'm having exactly the same issue. For last night's upload of 3 files, 2 failed with this error.

[ctiwald]

Having the exact same issue with a 13GB upload.

[offlinehacker]

In progress of fixing ;) On Nov 12, 2012 3:24 PM, "Christopher Tiwald" [email protected] wrote:

Having the exact same issue with a 13GB upload.

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/95#issuecomment-10289329.

[SitronNO]

I had the same problem with several files. However, if I changed the partsize, it worked. Can some of you try to change the partsize, and see if the file will upload without the 408-error?

[vsespb]

IMHO It's Amazon problem https://forums.aws.amazon.com/thread.jspa?messageID=399111 I experience it with this my client https://github.com/vsespb/mt-aws-glacier when do high concurrency uploads.

[vsespb]

Seems changing tcp_congestion_control from cubic to westwood helps a lot in my case ( I use ADSL) . Note that different Linux kernels have different default tcp_congestion_control. Also different mode should be used for different internet connection technologies.

[theRemix]

Confirmed multiple attempts failed uploading a 6.6G file using $glacier-cmd upload --partsize 8 my_vault my_large_file.tar.bz2 --partsize 32 also did not work, and same results without the --partsize param

... File "/usr/lib64/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glaciercorecalls.py", line 129, in write data) File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/glacier/layer1.py", line 637, in upload_part response_headers=response_headers) File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/glacier/layer1.py", line 84, in make_request raise UnexpectedHTTPResponseError(ok_responses, response) boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)

does it work with smaller filesizes? i'm going to split it up and test.

[theRemix]

i got something different, socket.error Connection reset by peer

i split up my large files into 100M chunks and uploaded

$glacier-cmd upload --partsize 32 my_vault *

... File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/glacier/layer1.py", line 79, in make_request data=data) File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/connection.py", line 932, in make_request return self._mexe(http_request, sender, override_num_retries) File "/usr/lib64/python2.6/site-packages/boto-2.8.0-py2.6.egg/boto/connection.py", line 894, in _mexe raise e socket.error: [Errno 104] Connection reset by peer

i also tested with --partsize 8 and got this same error boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)

[adammulligan]

FYI: I was getting this problem with pretty much every upload great than 100mb or so, but cloning the latest uskudnik/amazon-glacier-cmd-interface (0b88b042b3) and reinstalling seems to have fixed it entirely and I no longer get 408s. I still think there's an issue on Amazon's end since it's happening with multiple clients, but hopefully this will help some people.

[uskudnik]

Anyone else fixed this problem with https://github.com/uskudnik/amazon-glacier-cmd-interface/commit/0b88b042b3 (although I suspect issue was fixed a couple of commits earlier or Amazon fixed it on its end)?

[kmand]

I just started using this and downloaded the master.zip which is presumably a2d5763f67. I'm using it with bacula, which generates large files to upload. So far I have had a 10GB upload fail, a 50GB succeed and a 38GB fail. The failures give the error message of this thread. The uploads are going at about 1.5GB/hour through my cable modem, so the successful 50 GB took over 30 hours. The 10GB and 38GB failed about half way through.

Any suggestions?

[SitronNO]

I do not understand how a calculation of the clients ETA is suppose to fix the 408-bug? https://github.com/uskudnik/amazon-glacier-cmd-interface/commit/0b88b042b3 - Or am I looking at the wrong code/commit?

And just to make matters worse, I use the Parallel uploads-branch from wvmarle, which does not contain the code.

[SitronNO]

I just tested the latest code (git://github.com/uskudnik/amazon-glacier-cmd-interface.git) on a new client, and I did not get the 408-error as I did with the parallel-uploads-code. I tried over 11 files, the biggest was 37GB. Total of 67GB.

[The-Judge]

I cloned the repo and installed it from a2d5763 . I created a fresh vault and tried to upload a testfile of 785MB to it (I choosed Ubuntu ISO image for it). I'm using this command to do so:

glacier-cmd upload --description "Test Glacier handling with Ubuntu ISO Image" testvault ubuntu-13.04-desktop-amd64.iso

The upload starts and glacier-cmd displays it's progress bar. It uploads fine at first, but at any time this upload crashes with the described error. It already crashed at 30MB, 90MB, 120MB, ... not really foreseeable.

[ken107]

Looks like this problem is with the Amazon API. I'm using the Java API and had to write my own uploader to circumvent this problem. My uploader first downloads the entire list of parts, then it only uploads parts that are missing or has mismatched checksum. This allows to resume the upload from where it got interrupted. EDIT: I found the reason is the SDK doesn't retry on a 408 by default. This can be fixed by using a custom ClientConfiguration.RetryPolicy when constructing the AmazonGlacierClient.

[simonbaudry]

Same issue for me with a file of 1.1 GB !

I solved it by setting partsize parameter to 32.

[mraheel83]

Yesterday I setup glacier-cmd from current source code and I'm still getting this error when I'm uploading 140GB file.

I'm getting this error after 10GB upload out of 140GB. Please advise me how can I fix it.

Error is:

File "/usr/bin/glacier-cmd", line 9, in load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')() File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacie r.py", line 929, in main args.func(args) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacie r.py", line 156, in wrapper return fn(_args, *_kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacie r.py", line 309, in upload args.name, args.partsize, args.uploadid, args.resume) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/Glacie rWrapper.py", line 65, in wrapper ret = fn(_args, *_kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/Glacie rWrapper.py", line 232, in glacier_connect_wrap return func(_args, *_kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/Glacie rWrapper.py", line 65, in wrapper ret = fn(_args, *_kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/Glacie rWrapper.py", line 253, in sdb_connect_wrap return func(_args, *_kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/Glacie rWrapper.py", line 65, in wrapper ret = fn(_args, *_kwargs) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/Glacie rWrapper.py", line 1157, in upload writer.write(part) File "/usr/lib/python2.6/site-packages/glacier-0.2dev-py2.6.egg/glacier/glacie rcorecalls.py", line 129, in write data) File "/usr/lib/python2.6/site-packages/boto-2.38.0-py2.6.egg/boto/glacier/laye r1.py", line 1279, in upload_part response_headers=response_headers) File "/usr/lib/python2.6/site-packages/boto-2.38.0-py2.6.egg/boto/glacier/laye r1.py", line 119, in make_request raise UnexpectedHTTPResponseError(ok_responses, response) boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408,code=RequestTimeoutException, message=Request timed out.)

Thanks

[gburca]

Does the gburca@85ef4aa tip/branch fix this issue for anyone? See also https://github.com/uskudnik/amazon-glacier-cmd-interface/issues/171#issuecomment-101495322.

[SitronNO]

In the last days I have tried to upload 248 different files, all ranging from 13Mb to 37Gb (total of 450GB). After three days of 408 errors every 1-6 hours, I tried the code from @gburca (85ef4aa6dd58a36a77e4406366ddc27c7813cf5e) It has now uploaded 250GB the last 24 hours without a single error. Will try some more, but this is promising!