S3 Object Integrity Algorithms and Presigned URLs

[jessedoyle]

Describe the bug

Hey there - thanks for making a great library!

I'm trying to use the SDK to generate a presigned S3 PUT URL that enforces object integrity as documented using the SHA256 checksum algorithm.

After looking through the source, it appears that the Aws::S3::Presigner does not support generating presigned URLs that specify the x-amz-checksum-{algorithm} header as a signed header value.

At first glance this feels like a defect, therefore submitting as a bug. Happy to treat this as a feature request though - thanks!

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

Calling Aws::S3::Presigner#presigned_url with a checksum_algorithm value generates a URL that specifies the necessary checksum headers as signed headers.

Current Behavior

Calling Aws::S3::Presigner#presigned_url with a checksum_algorithm value generates a URL with the checksum headers as unsigned headers - meaning that the recipient of the URL can't utilize object integrity checks.

Reproduction Steps

Here's a minimal script that demonstrates the issue:

require 'bundler/inline'
require 'net/http'

gemfile do
  source 'https://rubygems.org/'
  gem 'aws-sdk-s3', '~> 1.182.0'
  gem 'marcel', '~> 1.0.4'
  gem 'nokogiri', '~> 1.18.5'
  gem 'rest-client', '~> 2.1.0'
end

class FileAttributes
  attr_reader :bytesize, :checksum, :mime_type

  def initialize(path)
    @bytesize = File.read(path).bytesize
    @checksum = digest(path)
    @mime_type = Marcel::MimeType.for(Pathname.new(path))
  end

  private

  def digest(path)
    Digest::SHA2.new.tap do |sha|
      File.open(path) do |f|
        while chunk = f.read(256)
          sha << chunk
        end
      end
    end.base64digest
  end
end

def upload_file(path, attributes, url, headers: {})
  puts "==> Uploading #{path}..."
  puts "Request: PUT #{url}"

  response = RestClient::Request.execute(
    url: url,
    method: :put,
    headers: { 'Content-Type' => attributes.mime_type }.merge(headers),
    payload: File.new(path)
  )

  puts "Response: #{response.code}"
rescue RestClient::ExceptionWithResponse => e
  puts "Error: #{e.response.body}"
end

attributes = FileAttributes.new('file_one.png')
presigner = Aws::S3::Presigner.new
presigned_url = presigner.presigned_url(
  :put_object,
  bucket: ENV.fetch('BUCKET_NAME'),
  key: 'test',
  checksum_algorithm: 'SHA256',
  checksum_sha256: attributes.checksum,
  expires_in: 3600,
  content_type: attributes.mime_type,
  content_length: attributes.bytesize
)

# Request succeeds as expected
upload_file('file_one.png', attributes, presigned_url)

# Uploading with a _different_ checksum using the same URL succeeds, but SHOULD fail
upload_file('file_two.jpeg', attributes, presigned_url)

# Specifying x-amz-checksum-sha256 fails with 403 (HeadersNotSigned)
upload_file(
  'file_one.png',
  attributes,
  presigned_url, 
  headers: { 'x-amz-checksum-sha256' => attributes.checksum }
)

Save the script as script.rb and provide two different test files in the same directory (file_one.png and file_two.jpeg).

Execute the script as follows:

BUCKET_NAME=bucket ruby script.rb

Possible Solution

Avoid removing the checksum handler and don't hoist the checksum headers when the presigned URL is being generated.

Additional Information/Context

It's worthwhile to note that the Content-MD5 header is treated as a signed header for presigned URLs. This is different than the behaviour when a different algorithm is specified.

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-s3, 1.182.0

Environment details (Version of Ruby, OS environment)

Ruby 3.3.6, MacOS 15.3.2

[mullermp]

Thanks for opening an issue. This was an intentional decision because of backwards compatibility and because s3 doesn't support this feature on presigned urls to my understanding. S3 team is aware of this.

[jessedoyle]

Thanks for the context @mullermp!

From my testing, the S3 PutObject API calls do in fact support object integrity checks via presigned URLs if the X-Amz-SignedHeaders component of the signature includes the necessary header fields (x-amz-checksum-{algorithm} and x-amz-sdk-checksum-algorithm) AND the caller passes these headers along in the request as well.

It feels like a change can be made to the SDK in a non-breaking way to opt in to this behaviour during presigning - for example adding a new optional parameter to Aws::S3::Presigner#presigned_url similar to this:

presigner = Aws::S3::Presigner.new
presigner.presigned_url(
  ...,
  checksum_algorithm: 'SHA256',
  checksum_sha256: 'checksum',
  checksum_sign_headers: true
)

I'd be happy to submit a PR if you think this is an acceptable approach?

[mullermp]

S3 would like to support it directly using query parameters and not by sending headers. I'll talk with them and other teams.

[jessedoyle]

Ah makes sense - please let me know if there's anything I can do to support 👍