aws
Recursive copy to S3 intermittently fails midway through with 404 error
Describe the bug
We're experiencing an issue with the cli during a recursive copy to S3. We're copying files from a folder up to a bucket, but the copy fails midway through with a 404 error. The workflow looks like this:
$ bucket=my-bucket-name
$ aws s3 mb s3://$bucket --region us-east-1
make_bucket: my-bucket-name
$ # < loop and wait for `aws s3api head-bucket --bucket $bucket` to return 200 >
$ aws s3 cp --acl public-read --recursive $from_dir s3://$bucket >/dev/null
upload failed: <dir> to s3://my-bucket-name/img/1.png An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist
After the failure, we observe that some files have been copied into the bucket, but not all of the files in $from_dir. Adding a --debug to the cp command shows files successfully being PUT, followed by one that fails due to a 404, which then causes the entire command to fail:
# Several files be successfully copied:
2022-07-20T15:29:12.9319044Z 2022-07-20 15:29:12,931 - ThreadPoolExecutor-0_7 - urllib3.connectionpool - DEBUG - [https://my-bucket-name.s3.amazonaws.com:443 ](https://my-bucket-name.s3.amazonaws.com/) "PUT /nuxt/0c6f8ea.js.map HTTP/1.1" 200 0
2022-07-20T15:29:12.9305832Z 2022-07-20 15:29:12,929 - ThreadPoolExecutor-0_4 - urllib3.connectionpool - DEBUG - [https://my-bucket-name.s3.amazonaws.com:443 ](https://my-bucket-name.s3.amazonaws.com/) "PUT /nuxt/0b559bd.js.map HTTP/1.1" 200 0
....
# Followed by a failure and 404 error:
2022-07-20T15:29:12.9347320Z 2022-07-20 15:29:12,934 - ThreadPoolExecutor-0_6 - urllib3.connectionpool - DEBUG - [https://my-bucket-name.s3.amazonaws.com:443 ](https://my-bucket-name.s3.amazonaws.com/) "PUT /nuxt/0cde623.js.map HTTP/1.1" 404 None
Not sure if this could be a locking issue, or a problem with eventual consistency in S3. We're not sure why we'd see a 404 for a file we're creating, unless it's not able to find the bucket. It happens sporadically, and if we rerun the job it may work on the second try. It is happening quite frequently in our CI, up to several times per day. It's worth noting we have been using this process for a couple years, and this has only become in issue over the past 2-3 weeks.
Expected Behavior
I expect that when aws s3api head-bucket --bucket $bucket returns 200 that we can copy to the bucket without 404 errors.
Current Behavior
The recursive copy fails partway through, copying some, but not all, of the files from the source directory into the bucket. We receive this error:
upload failed: <dir> to s3://my-bucket-name/img/1.png An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist
And the debug logs show a 404 on a PUT:
2022-07-20T15:29:12.9347320Z 2022-07-20 15:29:12,934 - ThreadPoolExecutor-0_6 - urllib3.connectionpool - DEBUG - [https://my-bucket-name.s3.amazonaws.com:443 ](https://my-bucket-name.s3.amazonaws.com/) "PUT /nuxt/0cde623.js.map HTTP/1.1" 404 None
Reproduction Steps
Not consistently reproducible.
$ bucket=my-bucket-name
$ aws s3 mb s3://$bucket --region us-east-1
make_bucket: my-bucket-name
$ # < loop and wait for `aws s3api head-bucket --bucket $bucket` to return 200 >
$ aws s3 cp --acl public-read --recursive $from_dir s3://$bucket >/dev/null
upload failed: <dir> to s3://my-bucket-name/img/1.png An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist
Possible Solution
No response
Additional Information/Context
No response
CLI version used
1.19.105-r0
Environment details (OS name and version, etc.)
docker, alpine:3.15.4
Hi @kflavin thanks for reaching out. If you can't reproduce this consistently is it possible that you're experiencing intermittent network issues? I also recommend updating to a more recent version of the CLI (v2 if possible) and trying this again.
Hi @tim-finnigan, thanks for the response. I don't think it's a connectivity issue, I'm seeing 404's being returned, so there is a response coming back. I'm running these tests from an EC2 in our AWS account. There's also no connectivity errors in the logs.
I was able to replicate using version 1.22.81 of the cli. Here's the (sanitized) error I got back with --debug enabled:
2022-07-21 17:39:12,139 - ThreadPoolExecutor-0_7 - botocore.auth - DEBUG - CanonicalRequest:
PUT
/js/290.9d5b11b58e54eafdb5f2.bundle.js.map
content-md5:BcHvQd+Yd22cXKzbjxZ3yw==
host:mybucket-868f148b.s3.amazonaws.com
x-amz-acl:public-read
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20220721T173912Z
content-md5;host;x-amz-acl;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2022-07-21 17:39:12,139 - ThreadPoolExecutor-0_7 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220721T173912Z
20220721/us-east-1/s3/aws4_request
1111111
2022-07-21 17:39:12,139 - ThreadPoolExecutor-0_7 - botocore.auth - DEBUG - Signature:
1111111
2022-07-21 17:39:12,140 - ThreadPoolExecutor-0_7 - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <function signal_transferring at 0x7f286c5e1bd0>
2022-07-21 17:39:12,140 - ThreadPoolExecutor-0_7 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://mybucket-868f148b.s3.amazonaws.com/js/290.9d5b11b58e54eafdb5f2.bundle.js.map, headers={'x-amz-acl': b'public-read', 'User-Agent': b'aws-cli/1.22.81 Python/3.10.4 Linux/4.14.225-169.362.amzn2.x86_64 botocore/1.21.49', 'Content-MD5': b'BcHvQd+Yd22cXKzbjxZ3yw==', 'Expect': b'100-continue', 'X-Amz-Date': b'20220721T173912Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=credentialremoved/20220721/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-acl;x-amz-content-sha256;x-amz-date, Signature=1111111', 'Content-Length': '493132'}>
2022-07-21 17:39:12,140 - ThreadPoolExecutor-0_7 - botocore.httpsession - DEBUG - Certificate path: /usr/lib/python3.10/site-packages/botocore/cacert.pem
2022-07-21 17:39:12,141 - ThreadPoolExecutor-0_7 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2022-07-21 17:39:12,142 - ThreadPoolExecutor-0_9 - botocore.awsrequest - DEBUG - Received a non 100 Continue response from the server, NOT sending request body.
2022-07-21 17:39:12,142 - ThreadPoolExecutor-0_9 - urllib3.connectionpool - DEBUG - https://mybucket-868f148b.s3.amazonaws.com:443 "PUT /js/290.9d5b11b58e54eafdb5f2.bundle.js HTTP/1.1" 404 None
2022-07-21 17:39:12,143 - ThreadPoolExecutor-0_9 - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': '1111111111111', 'x-amz-id-2': '111', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 21 Jul 2022 17:39:12 GMT', 'Server': 'AmazonS3', 'Connection': 'close'}
2022-07-21 17:39:12,143 - ThreadPoolExecutor-0_9 - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>mybucket-868f148b</BucketName><RequestId>1111111111111</RequestId><HostId>111</HostId></Error>'
2022-07-21 17:39:12,146 - ThreadPoolExecutor-0_9 - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': '1111111111111', 'x-amz-id-2': '111', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Thu, 21 Jul 2022 17:39:12 GMT', 'Server': 'AmazonS3', 'Connection': 'close'}
2022-07-21 17:39:12,146 - ThreadPoolExecutor-0_9 - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>mybucket-868f148b</BucketName><RequestId>1111111111111</RequestId><HostId>111</HostId></Error>'
2022-07-21 17:39:12,146 - ThreadPoolExecutor-0_9 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f286baba050>
2022-07-21 17:39:12,146 - ThreadPoolExecutor-0_9 - botocore.retryhandler - DEBUG - No retry needed.
2022-07-21 17:39:12,146 - ThreadPoolExecutor-0_9 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f286baba110>>
2022-07-21 17:39:12,146 - ThreadPoolExecutor-0_9 - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <function enhance_error_msg at 0x7f286c239240>
2022-07-21 17:39:12,147 - ThreadPoolExecutor-0_8 - botocore.awsrequest - DEBUG - 100 Continue response seen, now sending request body.
2022-07-21 17:39:12,147 - ThreadPoolExecutor-0_5 - urllib3.connectionpool - DEBUG - https://mybucket-868f148b.s3.amazonaws.com:443 "PUT /js/210.fb941c38654975249757.bundle.js.map HTTP/1.1" 200 0
2022-07-21 17:39:12,148 - ThreadPoolExecutor-0_5 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '111', 'x-amz-request-id': '002X2HVA9CZ606WR', 'Date': 'Thu, 21 Jul 2022 17:39:13 GMT', 'ETag': '"f159d1e7ca78ad280d2ddd0fb24bb93e"', 'Server': 'AmazonS3', 'Content-Length': '0'}
2022-07-21 17:39:12,148 - ThreadPoolExecutor-0_5 - botocore.parsers - DEBUG - Response body:
b''
2022-07-21 17:39:12,148 - ThreadPoolExecutor-0_5 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f286baba050>
2022-07-21 17:39:12,148 - ThreadPoolExecutor-0_5 - botocore.retryhandler - DEBUG - No retry needed.
2022-07-21 17:39:12,148 - ThreadPoolExecutor-0_5 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f286baba110>>
2022-07-21 17:39:12,148 - ThreadPoolExecutor-0_5 - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <function enhance_error_msg at 0x7f286c239240>
Completed 2.2 MiB/16.8 MiB (5.5 MiB/s) with 70 file(s) remaining^Mupload: dist/js/210.fb941c38654975249757.bundle.js.map to s3://mybucket-868f148b/js/210.fb941c38654975249757.bundle.js.map
2022-07-21 17:39:12,147 - ThreadPoolExecutor-0_9 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/s3transfer/tasks.py", line 139, in __call__
return self._execute_main(kwargs)
File "/usr/lib/python3.10/site-packages/s3transfer/tasks.py", line 162, in _execute_main
return_value = self._main(**kwargs)
File "/usr/lib/python3.10/site-packages/s3transfer/upload.py", line 757, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/usr/lib/python3.10/site-packages/botocore/client.py", line 386, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3.10/site-packages/botocore/client.py", line 705, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist
Completed 2.2 MiB/16.8 MiB (5.5 MiB/s) with 69 file(s) remaining^Mupload failed: dist/js/290.9d5b11b58e54eafdb5f2.bundle.js to s3://mybucket-868f148b/js/290.9d5b11b58e54eafdb5f2.bundle.js An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist
Completed 2.4 MiB/16.8 MiB (5.5 MiB/s) with 68 file(s) remaining^M2022-07-21 17:39:12,148 - ThreadPoolExecutor-0_5 - s3transfer.utils - DEBUG - Releasing acquire 25/None
Hi @kflavin thanks for following up. How often does this error occur? Is the request you're making in the same region as your bucket? And have you tried updating to v2 of the AWS CLI or a more recent version of v1?
Since we haven’t heard back in a few weeks I’m going to close this issue. If you’re still experiencing this issue please respond to the questions above. Thanks!
⚠️COMMENT VISIBILITY WARNING⚠️
Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Since we haven’t heard back in a few weeks I’m going to close this issue. If you’re still experiencing this issue please respond to the questions above. Thanks!
@tim-finnigan Apologies for the delay. This ended up being a bug on the AWS back end. I opened a support ticket and was told it has been resolved. I haven't been able to replicate the issue for several weeks.
Ok glad to hear that it has been resolved, thanks for letting us know!