Unable to copy/list/ get details of an object in a different region. Getting "Aws::S3::Errors::PermanentRedirect: The bucket you are attempting to access must be addressed"

[ashokap]

Please fill out the sections below to help us address your issue

Issue description

My objective:

1. Have an existing bucket(Eg: "abc-mumbai") in mumbai region.
2. Need to copy all these into a new bucket (Eg: "abc-tokyo") which is in tokyo region. I can copy the entire contents of these from AWS console but need to get "public urls" of objects from the new bucket and update some records in our DB. Hence prefer copying these programatically.
3. Copy_to method is failing with an error "Aws::S3::Errors::PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk version: aws-sdk-2.1.15

Version of Ruby, OS environment

Ruby: 2.1.8, Rails: 4.2

Code snippets / steps to reproduce

bucket = Aws::S3::Bucket.new('abc-mumbai') object = bucket.object('12313123/uploads/a.txt') object.copy_to(bucket: 'abc-tokyo', key: '12313123/uploads/a.txt', multipart_copy: true, content_length: object.content_length, copy_source_region: 'abc-mumbai')

[cjyclaire]

you need to use :copy_from for cross region copy instead https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-s3/lib/aws-sdk-s3/customizations/object.rb#L73

[ashokap]

Thanks but tried copy_from as well and same error received.

bucket = Aws::S3::Bucket.new('abc-tokyo') object = bucket.object('12313123/uploads/a.txt') object.copy_from(bucket:'abc-mumbai', key:'12313123/uploads/a.txt') Got the same error message: Full trace below

Aws::S3::Errors::PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in call' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in call' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in call' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/aws-sdk-core/plugins/param_converter.rb:20:in call' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/seahorse/client/plugins/response_target.rb:21:in call' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/seahorse/client/request.rb:70:in send_request' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-core-2.6.49/lib/seahorse/client/base.rb:207:in block (2 levels) in define_operation_methods' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.6.49/lib/aws-sdk-resources/services/s3/object_copier.rb:33:in copy_object' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.6.49/lib/aws-sdk-resources/services/s3/object_copier.rb:15:in copy_from' from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.6.49/lib/aws-sdk-resources/services/s3/object.rb:66:in copy_from' from (irb):17 from /Users/pana/.rbenv/versions/2.1.4/bin/irb:11:in `

[ashokap]

Infact no matter what operation I do on that target object, I am getting "Aws::S3::Errors::Http301Error: " error.

I just instantiated this new object with key and tried "exists?" method. It throws error 301 !!

irb(main):036:0> bucket = Aws::S3::Bucket.new('sample-tokyo-delete')
=> #<Aws::S3::Bucket name="sample-tokyo-delete">
irb(main):037:0> object = bucket.object('55eec4fba7bf37030b000001/attachments/secondary_backup.sh')
=> #<Aws::S3::Object bucket_name="sample-tokyo-delete", key="55eec4fba7bf37030b000001/attachments/secondary_backup.sh">
irb(main):038:0> object.exists?
Aws::S3::Errors::Http301Error: 
from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.6.49/lib/aws-sdk-resources/resource.rb:134:in `rescue in exists?'
	from /Users/pana/.rbenv/versions/2.1.4/lib/ruby/gems/2.1.0/gems/aws-sdk-resources-2.6.49/lib/aws-sdk-resources/resource.rb:131:in `exists?'
	from (irb):38
	from /Users/pana/.rbenv/versions/2.1.4/bin/irb:11:in `<main>'

[cjyclaire]

Tagging this a feature request to handle 301 http status code

[dee-see]

Very interested in this, I get the same error on all operations I attempted on my bucket

[usamaSaddiq]

+1 experiencing the same issue. Any updates on this?

[alextwoods]

If you're getting the same error on all operations you attempt on the bucket, make sure you have constructed the resource with a client with the correct region. eg:

bucket = Aws::S3::Bucket.new(bucket_other_region, client: Aws::S3::Client.new(region: other_region)

These errors largely exist to prevent unintentional cross region requests - so in code where you are doing this you will need to be explicit about regions.

For the copy_from issue, see the documentation for Object#copy_from. You'll need to set multipart_copy to true to use copy_source_region. However, in general for operations like this I'd recommend using the client#copy_object to make an explicit call. eg:

c_region1 = Aws::S3::Client.new(region: region_1)
c_region2 = Aws::S3::Client.new(region: region_2)

c_region1.put_object(bucket: bucket_region1, key: 'test_copy', body: 'handshake')
# Use a client with the correct region for the destination bucket
c_region2.copy_object(bucket: bucket_region2, copy_source: "#{bucket_region1}/test_copy", key: 'test_copy')

# Use the correct client to access the newly copied object
c_region2.get_object(bucket: bucket_region2, key: 'test_copy').body.read

[github-actions[bot]]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

[github-actions[bot]]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.