Getting an intermittent access denied error when running get_object

[thebkbuffalo]

Describe the bug

When trying to retrieve an object from an s3 bucket using get_object I'm intermittently getting an access denied error from AWS. The only info I'm getting back from AWS is #<Aws::S3::Errors::AccessDenied: Access Denied> which isn't really all that helpful. I'm assuming it's not a credentials problem since I'm also writing to s3 in this process with no issues, and also because get_object works sometimes, but definitely not every time. This process is happening on a Sidekiq worker on an ECS instance.

csv_str = client.get_object(bucket: bucketname, key: filename).body.read
parsed_csv = CSV.parse(csv_str, headers: true, encoding: "ISO8859-1:utf-8")
return parsed_csv

Expected Behavior

I expected get_object to work every time.

Current Behavior

Currently get_object only works sometimes.

Reproduction Steps

instantiate a new s3 client, call get_object on the client.

Possible Solution

No response

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-s3

Environment details (Version of Ruby, OS environment)

Ruby version 3.2.1, AWS ECS Sidekiq worker env.

[mullermp]

Aws::S3::Errors::AccessDenied is most certainly a service side error, so I'm not sure it's specifically a Ruby SDK issue. How are you configuring credentials - is it manual or automatically? I see that you're using ECS. Is it possible you are getting different sets of credentials intermittently? Or those credentials become stale? If you are using ECSCredentials, try configuring them with ECSCredentials.new(http_debug_output: <some logger>) and observe if, after failures, credentials are refreshed?

[RanVaknin]

Hi @thebkbuffalo

To add on top of what @mullermp said, if you are seeing intermittent access denied errors, it is possible that your requests are using an incorrect set of credentials. It can happen for many reasons, and all likely stem from a failure of the SDK's credential provider chain from being able to retrieve credentials for the desired ECS environment. It will then attempt to resolve credentials by invoking other providers in the credential chain, and if one of those is configured to return a set of credentials it might result in the wrong role / principal to be assumed and obtained and therefore might lead to access denied errors.

In an ECS environment, you usually have a token associated with the ECS container. Under the hood, the SDK makes an implicit invisible call to exchange that token for a set of temporary credentials. Configuring a logger would result in those implicit API calls to become visible in your logging infrastructure and you can see what is failing and why.

Let us know if you have any questions.

Thanks, Ran~

[thebkbuffalo]

@mullermp you were totally right. it was a server side error that happening. thank you and @RanVaknin for your help!

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.