s3gof3r icon indicating copy to clipboard operation
s3gof3r copied to clipboard

Published 20 hours ago • verified publisher icon rlmcpherson

gof3r fails with exit code 0 if multi-part upload is aborted

Open bmurphy1976 opened this issue 9 years ago • 4 comments

Earlier today we spun up over 100 Amazon EC2 instances to upload a backlog of large files into S3. Because of the volume and size of files, Amazon S3 started to throttle our requests. The request throttling caused gof3r to abort it's multi-part upload and exit with an exit code of 0. Our system believed the upload was successful because of the exit code and subsequently kicked off a number of jobs that then failed because the files were missing.

I believe the proper behavior in this scenario is for gof3r to exit with a non-zero exit code.

I've attached the console output of one of the invalid jobs that clearly shows the problem. The zero in the last line is the exit code of the gof3r process.

[2015-12-14 22:17:58,598] [INFO] [C] EXEC: gof3r put -b BUCKETCENSORED -k /f/c4/23edfd2fe402e993dfc87fc629999/1450130538d48582/mp4/1596/d91a44495da9439bbcecccadd2668eea.mp4 -p /tmp/2ed5141f956447d4825832a0eb42aaa0/d91a44495da9439bbcecccadd2668eea.mp4 --no-md5 --debug -m x-amz-server-side-encryption:AES256 -m x-mf-md5:a2cb8d5b4f4f947986bf3e9357aed138 -m uid:0 -m gid:0 -m mode:33188 -m atime:1450130737 -m ctime:1450131470 -m mtime:1450131470
[2015-12-14 22:17:59,563] [INFO] [E] 2015/12/14 22:17:59 Error on attempt 0: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:17:59,813] [INFO] [E] 2015/12/14 22:17:59 Error on attempt 0: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:17:59,978] [INFO] [E] 2015/12/14 22:17:59 Error on attempt 0: Retrying part: 3, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,077] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 1: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,233] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 0: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,462] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 0: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,587] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 1: Retrying part: 3, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,716] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 2: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,745] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 1: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,819] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 1: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,947] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 0: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:00,986] [INFO] [E] 2015/12/14 22:18:00 Error on attempt 0: Retrying part: 6, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:01,002] [INFO] [E] 2015/12/14 22:18:01 Error on attempt 1: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:01,657] [INFO] [E] 2015/12/14 22:18:01 Error on attempt 2: Retrying part: 3, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:01,721] [INFO] [E] 2015/12/14 22:18:01 Error on attempt 0: Retrying part: 8, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:01,795] [INFO] [E] 2015/12/14 22:18:01 Error on attempt 2: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:02,044] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 2: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:02,057] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 0: Retrying part: 9, Error: Put https://BUCKETCENSORED.s3.amazonaws.com/f/c4/23edfd2fe402e993dfc87fc629999/1450130538d48582/mp4/1596/d91a44495da9439bbcecccadd2668eea.mp4?partNumber=9&uploadId=a.K8_OwKHWx_MOuo6vlI9A66SyJjw9ezrcBHvll16rBT.Yu4D.3NLPYIvbkI4M4FXEWfR9XMXYNY.8NJjPb4f2FgQ3vna_eBJUq.k9r7MnFVYisCrrwHdVXz5ZwG41N8: EOF
[2015-12-14 22:18:02,085] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 1: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:02,132] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 1: Retrying part: 6, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:02,169] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 3: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:02,363] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 0: Retrying part: 10, Error: Put https://BUCKETCENSORED.s3.amazonaws.com/f/c4/23edfd2fe402e993dfc87fc629999/1450130538d48582/mp4/1596/d91a44495da9439bbcecccadd2668eea.mp4?partNumber=10&uploadId=a.K8_OwKHWx_MOuo6vlI9A66SyJjw9ezrcBHvll16rBT.Yu4D.3NLPYIvbkI4M4FXEWfR9XMXYNY.8NJjPb4f2FgQ3vna_eBJUq.k9r7MnFVYisCrrwHdVXz5ZwG41N8: EOF
[2015-12-14 22:18:02,417] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 1: Retrying part: 8, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:02,497] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 2: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:02,848] [INFO] [E] 2015/12/14 22:18:02 Error on attempt 1: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:03,060] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 2: Retrying part: 6, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:03,290] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 3: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:03,295] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 3: Retrying part: 5, Error: Put https://BUCKETCENSORED.s3.amazonaws.com/f/c4/23edfd2fe402e993dfc87fc629999/1450130538d48582/mp4/1596/d91a44495da9439bbcecccadd2668eea.mp4?partNumber=5&uploadId=a.K8_OwKHWx_MOuo6vlI9A66SyJjw9ezrcBHvll16rBT.Yu4D.3NLPYIvbkI4M4FXEWfR9XMXYNY.8NJjPb4f2FgQ3vna_eBJUq.k9r7MnFVYisCrrwHdVXz5ZwG41N8: EOF
[2015-12-14 22:18:03,486] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 2: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:03,557] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 3: Retrying part: 3, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:03,567] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 1: Retrying part: 10, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:03,652] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 2: Retrying part: 8, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:03,787] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 4: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:03,901] [INFO] [E] 2015/12/14 22:18:03 Error on attempt 2: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:04,372] [INFO] [E] 2015/12/14 22:18:04 Error on attempt 3: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:04,491] [INFO] [E] 2015/12/14 22:18:04 Error on attempt 3: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:04,824] [INFO] [E] 2015/12/14 22:18:04 Error on attempt 3: Retrying part: 8, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:04,835] [INFO] [E] 2015/12/14 22:18:04 Error on attempt 4: Retrying part: 3, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:04,936] [INFO] [E] 2015/12/14 22:18:04 Error on attempt 2: Retrying part: 10, Error: Put https://BUCKETCENSORED.s3.amazonaws.com/f/c4/23edfd2fe402e993dfc87fc629999/1450130538d48582/mp4/1596/d91a44495da9439bbcecccadd2668eea.mp4?partNumber=10&uploadId=a.K8_OwKHWx_MOuo6vlI9A66SyJjw9ezrcBHvll16rBT.Yu4D.3NLPYIvbkI4M4FXEWfR9XMXYNY.8NJjPb4f2FgQ3vna_eBJUq.k9r7MnFVYisCrrwHdVXz5ZwG41N8: EOF
[2015-12-14 22:18:04,967] [INFO] [E] 2015/12/14 22:18:04 Error on attempt 4: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:05,016] [INFO] [E] 2015/12/14 22:18:05 Error on attempt 3: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:05,361] [INFO] [E] 2015/12/14 22:18:05 Error on attempt 4: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:05,470] [INFO] [E] 2015/12/14 22:18:05 Error on attempt 0: Retrying part: 11, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:05,976] [INFO] [E] 2015/12/14 22:18:05 Error on attempt 5: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:06,027] [INFO] [E] 2015/12/14 22:18:06 Error on attempt 4: Retrying part: 8, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:06,096] [INFO] [E] 2015/12/14 22:18:06 Error on attempt 4: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:06,384] [INFO] [E] 2015/12/14 22:18:06 Error on attempt 3: Retrying part: 10, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:06,385] [INFO] [E] 2015/12/14 22:18:06 Error on attempt 1: Retrying part: 11, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:06,425] [INFO] [E] 2015/12/14 22:18:06 Error on attempt 4: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:06,887] [INFO] [E] 2015/12/14 22:18:06 Error on attempt 4: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:07,325] [INFO] [E] 2015/12/14 22:18:07 Error on attempt 5: Retrying part: 3, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:07,554] [INFO] [E] 2015/12/14 22:18:07 Error on attempt 5: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:07,794] [INFO] [E] 2015/12/14 22:18:07 Error on attempt 5: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:08,425] [INFO] [E] 2015/12/14 22:18:08 Error on attempt 5: Retrying part: 8, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:08,450] [INFO] [E] 2015/12/14 22:18:08 Error on attempt 0: Retrying part: 12, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:08,465] [INFO] [E] 2015/12/14 22:18:08 Error on attempt 4: Retrying part: 10, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:08,631] [INFO] [E] 2015/12/14 22:18:08 Error on attempt 5: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:08,906] [INFO] [E] 2015/12/14 22:18:08 Error on attempt 5: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:09,251] [INFO] [E] 2015/12/14 22:18:09 Error on attempt 5: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:09,254] [INFO] [E] 2015/12/14 22:18:09 Error on attempt 1: Retrying part: 12, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:10,054] [INFO] [E] 2015/12/14 22:18:10 Error on attempt 6: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:11,203] [INFO] [E] 2015/12/14 22:18:11 Error on attempt 0: Retrying part: 13, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:11,380] [INFO] [E] 2015/12/14 22:18:11 Error on attempt 5: Retrying part: 10, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:11,742] [INFO] [E] 2015/12/14 22:18:11 Error on attempt 6: Retrying part: 3, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:12,259] [INFO] [E] 2015/12/14 22:18:12 Error on attempt 6: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:12,380] [INFO] [E] 2015/12/14 22:18:12 Error on attempt 6: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:12,575] [INFO] [E] 2015/12/14 22:18:12 Error on attempt 1: Retrying part: 13, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:13,025] [INFO] [E] 2015/12/14 22:18:13 Error on attempt 6: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:13,363] [INFO] [E] 2015/12/14 22:18:13 Error on attempt 6: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:13,743] [INFO] [E] 2015/12/14 22:18:13 Error on attempt 6: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:13,998] [INFO] [E] 2015/12/14 22:18:13 Error on attempt 0: Retrying part: 14, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:14,053] [INFO] [E] 2015/12/14 22:18:14 Error on attempt 2: Retrying part: 13, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:15,644] [INFO] [E] 2015/12/14 22:18:15 Error on attempt 3: Retrying part: 13, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:15,786] [INFO] [E] 2015/12/14 22:18:15 Error on attempt 6: Retrying part: 10, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:16,591] [INFO] [E] 2015/12/14 22:18:16 Error on attempt 0: Retrying part: 15, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:17,678] [INFO] [E] 2015/12/14 22:18:17 Error on attempt 4: Retrying part: 13, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:17,758] [INFO] [E] 2015/12/14 22:18:17 Error on attempt 7: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:17,887] [INFO] [E] 2015/12/14 22:18:17 Error on attempt 1: Retrying part: 15, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:19,422] [INFO] [E] 2015/12/14 22:18:19 Error on attempt 7: Retrying part: 3, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:20,057] [INFO] [E] 2015/12/14 22:18:20 Error on attempt 7: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:20,141] [INFO] [E] 2015/12/14 22:18:20 Error on attempt 7: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:20,224] [INFO] [E] 2015/12/14 22:18:20 Error on attempt 5: Retrying part: 13, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:20,256] [INFO] [E] 2015/12/14 22:18:20 Error on attempt 7: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:20,607] [INFO] [E] 2015/12/14 22:18:20 Error on attempt 7: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:20,781] [INFO] [E] 2015/12/14 22:18:20 Error on attempt 0: Retrying part: 16, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:20,903] [INFO] [E] 2015/12/14 22:18:20 Error on attempt 7: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:22,436] [INFO] [E] 2015/12/14 22:18:22 Error on attempt 0: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:23,300] [INFO] [E] 2015/12/14 22:18:23 Error on attempt 7: Retrying part: 10, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:23,732] [INFO] [E] 2015/12/14 22:18:23 Error on attempt 1: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:24,758] [INFO] [E] 2015/12/14 22:18:24 Error on attempt 6: Retrying part: 13, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:25,151] [INFO] [E] 2015/12/14 22:18:25 Error on attempt 2: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:26,776] [INFO] [E] 2015/12/14 22:18:26 Error on attempt 3: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:28,984] [INFO] [E] 2015/12/14 22:18:28 Error on attempt 4: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:31,739] [INFO] [E] 2015/12/14 22:18:31 Error on attempt 8: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:31,758] [INFO] [E] 2015/12/14 22:18:31 Error on attempt 5: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:32,527] [INFO] [E] 2015/12/14 22:18:32 Error on attempt 7: Retrying part: 13, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:34,241] [INFO] [E] 2015/12/14 22:18:34 Error on attempt 8: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:34,245] [INFO] [E] 2015/12/14 22:18:34 Error on attempt 8: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:34,703] [INFO] [E] 2015/12/14 22:18:34 Error on attempt 8: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:34,905] [INFO] [E] 2015/12/14 22:18:34 Error on attempt 8: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:35,253] [INFO] [E] 2015/12/14 22:18:35 Error on attempt 8: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:36,134] [INFO] [E] 2015/12/14 22:18:36 Error on attempt 6: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:37,163] [INFO] [E] 2015/12/14 22:18:37 Error on attempt 0: Retrying part: 19, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:37,238] [INFO] [E] 2015/12/14 22:18:37 Error on attempt 8: Retrying part: 10, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:39,883] [INFO] [E] 2015/12/14 22:18:39 Error on attempt 0: Retrying part: 20, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:42,613] [INFO] [E] 2015/12/14 22:18:42 Error on attempt 0: Retrying part: 21, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:44,022] [INFO] [E] 2015/12/14 22:18:44 Error on attempt 1: Retrying part: 21, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:45,263] [INFO] [E] 2015/12/14 22:18:45 Error on attempt 7: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:45,350] [INFO] [E] 2015/12/14 22:18:45 Error on attempt 2: Retrying part: 21, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:47,019] [INFO] [E] 2015/12/14 22:18:47 Error on attempt 3: Retrying part: 21, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:47,750] [INFO] [E] 2015/12/14 22:18:47 Error on attempt 0: Retrying part: 22, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:49,089] [INFO] [E] 2015/12/14 22:18:49 Error on attempt 1: Retrying part: 22, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:50,155] [INFO] [E] 2015/12/14 22:18:50 Error on attempt 0: Retrying part: 23, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:50,659] [INFO] [E] 2015/12/14 22:18:50 Error on attempt 2: Retrying part: 22, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:52,681] [INFO] [E] 2015/12/14 22:18:52 Error on attempt 3: Retrying part: 22, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:53,519] [INFO] [E] 2015/12/14 22:18:53 Error on attempt 0: Retrying part: 24, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:54,695] [INFO] [E] 2015/12/14 22:18:54 Error on attempt 4: Retrying part: 22, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:55,066] [INFO] [E] 2015/12/14 22:18:55 Error on attempt 0: Retrying part: 25, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:55,904] [INFO] [E] 2015/12/14 22:18:55 Error on attempt 1: Retrying part: 25, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:57,492] [INFO] [E] 2015/12/14 22:18:57 Error on attempt 2: Retrying part: 25, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:58,016] [INFO] [E] 2015/12/14 22:18:58 Error on attempt 0: Retrying part: 26, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:58,344] [INFO] [E] 2015/12/14 22:18:58 Error on attempt 9: Retrying part: 1, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:58,646] [INFO] [E] 2015/12/14 22:18:58 Error on attempt 3: Retrying part: 25, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:59,154] [INFO] [E] 2015/12/14 22:18:59 Error on attempt 8: Retrying part: 17, Error: 503: "Please reduce your request rate."
[2015-12-14 22:18:59,242] [INFO] [E] 2015/12/14 22:18:59 Error on attempt 1: Retrying part: 26, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:00,437] [INFO] [E] 2015/12/14 22:19:00 Error on attempt 4: Retrying part: 25, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:00,792] [INFO] [E] 2015/12/14 22:19:00 Error on attempt 2: Retrying part: 26, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:01,080] [INFO] [E] 2015/12/14 22:19:01 Error on attempt 9: Retrying part: 7, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:01,378] [INFO] [E] 2015/12/14 22:19:01 Error on attempt 9: Retrying part: 5, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:01,563] [INFO] [E] 2015/12/14 22:19:01 Error on attempt 9: Retrying part: 2, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:01,690] [INFO] [E] 2015/12/14 22:19:01 Error on attempt 9: Retrying part: 9, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:02,953] [INFO] [E] 2015/12/14 22:19:02 Error on attempt 9: Retrying part: 4, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:08,342] [INFO] [E] 2015/12/14 22:19:08 Error on attempt 0: Retrying part: 36, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:09,703] [INFO] [E] 2015/12/14 22:19:09 Error on attempt 1: Retrying part: 36, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:10,837] [INFO] [E] 2015/12/14 22:19:10 Error on attempt 0: Retrying part: 38, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:11,348] [INFO] [E] 2015/12/14 22:19:11 Error on attempt 2: Retrying part: 36, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:12,383] [INFO] [E] 2015/12/14 22:19:12 Error on attempt 0: Retrying part: 34, Error: Put https://BUCKETCENSORED.s3.amazonaws.com/f/c4/23edfd2fe402e993dfc87fc629999/1450130538d48582/mp4/1596/d91a44495da9439bbcecccadd2668eea.mp4?partNumber=34&uploadId=a.K8_OwKHWx_MOuo6vlI9A66SyJjw9ezrcBHvll16rBT.Yu4D.3NLPYIvbkI4M4FXEWfR9XMXYNY.8NJjPb4f2FgQ3vna_eBJUq.k9r7MnFVYisCrrwHdVXz5ZwG41N8: net/http: timeout awaiting response headers
[2015-12-14 22:19:12,940] [INFO] [E] 2015/12/14 22:19:12 Error on attempt 3: Retrying part: 36, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:13,813] [INFO] [E] 2015/12/14 22:19:13 Error on attempt 0: Retrying part: 39, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:13,824] [INFO] [E] 2015/12/14 22:19:13 Error on attempt 1: Retrying part: 34, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:15,061] [INFO] [E] 2015/12/14 22:19:15 Error on attempt 4: Retrying part: 36, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:15,269] [INFO] [E] 2015/12/14 22:19:15 Error on attempt 1: Retrying part: 39, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:15,652] [INFO] [E] 2015/12/14 22:19:15 Error on attempt 0: Retrying part: 40, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:16,564] [INFO] [E] 2015/12/14 22:19:16 Error on attempt 1: Retrying part: 40, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:16,866] [INFO] [E] 2015/12/14 22:19:16 Error on attempt 2: Retrying part: 39, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:17,550] [INFO] [E] 2015/12/14 22:19:17 Error on attempt 5: Retrying part: 36, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:17,995] [INFO] [E] 2015/12/14 22:19:17 Error on attempt 3: Retrying part: 39, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:18,669] [INFO] [E] 2015/12/14 22:19:18 Error on attempt 0: Retrying part: 41, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:19,936] [INFO] [E] 2015/12/14 22:19:19 Error on attempt 4: Retrying part: 39, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:21,335] [INFO] [E] 2015/12/14 22:19:21 Error on attempt 0: Retrying part: 42, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:22,160] [INFO] [E] 2015/12/14 22:19:22 Error on attempt 6: Retrying part: 36, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:22,778] [INFO] [E] 2015/12/14 22:19:22 Error on attempt 0: Retrying part: 43, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:23,025] [INFO] [E] 2015/12/14 22:19:23 Error on attempt 1: Retrying part: 43, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:23,520] [INFO] [E] 2015/12/14 22:19:23 Error on attempt 2: Retrying part: 43, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:23,982] [INFO] [E] 2015/12/14 22:19:23 Error on attempt 5: Retrying part: 39, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:24,063] [INFO] [E] 2015/12/14 22:19:24 Error on attempt 3: Retrying part: 43, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:25,022] [INFO] [E] 2015/12/14 22:19:25 Error on attempt 4: Retrying part: 43, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:26,724] [INFO] [E] 2015/12/14 22:19:26 Error on attempt 5: Retrying part: 43, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:29,712] [INFO] [E] 2015/12/14 22:19:29 Error on attempt 7: Retrying part: 36, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:30,039] [INFO] [E] 2015/12/14 22:19:30 Error on attempt 6: Retrying part: 43, Error: 503: "Please reduce your request rate."
[2015-12-14 22:19:54,153] [INFO] [E] 2015/12/14 22:19:54 12 buffers of 20 MB allocated
[2015-12-14 22:19:54,174] [INFO] [E] 2015/12/14 22:19:54 Error aborting multipart upload: 503: "Please reduce your request rate."
[2015-12-14 22:19:54,174] [INFO] [E] duration: 1m55.276254856s
[2015-12-14 22:19:54,197] [INFO] [C] EXEC COMPLETED: 0 (115.60s)

bmurphy1976 avatar Dec 15 '15 02:12 bmurphy1976

This could be due to either shadowing, which should be addressed by #85, or possibly a race condition in the putter error handling. The putter needs to be updated similarly to the getter to ensure p.err can not be set concurrently.

rlmcpherson avatar Dec 30 '15 20:12 rlmcpherson

@bmurphy1976 Do you know whether the zero exit code was happening consistently or intermittently? If it was due to a race condition, I would expect the error to be returned intermittently. If it was happening consistently, the root cause may be different and should be easier to track down as well.

rlmcpherson avatar Jan 02 '16 19:01 rlmcpherson

I asked our devs. The consesus seems to be that it felt more consistent than intermittent. Our system can hit a very high concurrent load when pushed, and as soon as we cross Amazon's limit all hell breaks loose.

bmurphy1976 avatar Jan 03 '16 01:01 bmurphy1976

I've noticed similar behavior trying to do a multi-part upload with SSE-C enabled. All parts of the upload fail with error 400s coming back from s3 due to https://github.com/rlmcpherson/s3gof3r/issues/49, but gof3r exits 0.

brianvans avatar May 09 '17 00:05 brianvans