S3 Gem firing undefined method rewind within Seahorse library

[dburgess560]

Describe the bug

Utilizing the S3 gem, some file upload requests are getting caught into an unhandled exception from the Seahorse library during the retry handler.

Expected Behavior

The request is retried without causing an exception to be thrown.

Current Behavior

On Ruby 2.7.6 with the following S3 gems:

    aws-eventstream (1.2.0)
    aws-partitions (1.626.0)
    aws-sdk-core (3.140.0)
      aws-eventstream (~> 1, >= 1.0.2)
      aws-partitions (~> 1, >= 1.525.0)
      aws-sigv4 (~> 1.1)
      jmespath (~> 1, >= 1.6.1)
    aws-sdk-kms (1.56.0)
      aws-sdk-core (~> 3, >= 3.127.0)
      aws-sigv4 (~> 1.1)
    aws-sdk-s3 (1.114.0)
      aws-sdk-core (~> 3, >= 3.127.0)
      aws-sdk-kms (~> 1)
      aws-sigv4 (~> 1.4)
    aws-sigv4 (1.5.1)
      aws-eventstream (~> 1, >= 1.0.2)
    jmespath (1.6.1)

Calling Aws::S3::Resource.new(client: client).bucket(aws_bucket_name).object(path).upload_file(element_for_upload, metadata) has been occasionally firing undefined method 'rewind' for #<Seahorse::Client::Plugins::ReadCallbackIO:0x0000561a24493100> out from aws-sdk-core (3.140.0) lib/aws-sdk-core/plugins/retry_errors.rb in retry_request

Unfortunately our reporting software isn't setup to sample external libraries so recreating the full call stack without invoking a large invoice for S3 usage is infeasible.

Reproduction Steps

Theoretically calling just Aws::S3::Resource.new(client: client).bucket(aws_bucket_name).object(path).upload_file(element_for_upload, metadata) with valid arguments will occasionally reproduce the exception. We've only been able to see this about 400 times across the last 2 weeks so the randomness due to what we believe is networking is incredibly hard to track down.

Possible Solution

The Seahorse library's ReadCallbackIO should either have a rewind method, or not be handled as it is being handled within the retry_errors library.

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-core (3.140.0)

Environment details (Version of Ruby, OS environment)

Ruby 2.7.6

[alextwoods]

Sorry you're running into this - I'm investigating but haven't yet been able to reproduce, but have a couple of theories.

What is in the metadata you are passing into upload_file?

[dburgess560]

Looking at our traces the error is only ever is occurring with these options

{
  server_side_encryption: "AES256",
  acl: "private",
  content_encoding: "gzip",
  progress_callback: lambda do |bytes_read, _|
    @bytes_uploaded += bytes_read.sum
  end
}

Could it be something in regards to that callback?

[alextwoods]

yes! It is related to the callback! You can remove the callback and it should resolve the retry issue as a temporary workaround.

I will work on a fix for this, but not yet sure what the best solution is.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.