add option to disable integrity check for multipart upload

[peng225]

Acknowledgements

• [x] I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• [x] I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

I read the announcement about the integrity check behaviour change, and tried to disable the integrity check by setting request_checksum_calculation and response_checksum_validation to when_required.

However, it seems that the integrity check cannot be disabled for multipart upload by Uploader.Upload function in feature/s3/manager package. I checked the HTTP header of the SDK's requests using tcpdump command. The results were as follows:

feature/s3/manager v1.17.57 (depending on service/s3 v1.76.0) without options:

p.H..._@PUT /5a50a3a7-c751-439f-922d-d7c4f169301a/orig_key?partNumber=1&uploadId=N2U1Y2Q0ZGYtZjFkZS00YTliLWEyMzItMmQxNzNkYzhhYjkzLmNkNWYwOTgyLTFiZjItNGE1ZC04MzZlLTVmYmRkMmQyNmZiNHgxNzM5NDA4ODUwOTg5NjYxMjI5&x-id=UploadPart HTTP/1.1
Host: localhost:35737
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.76.0 ft/s3-transfer m/E,G,Z
Content-Length: 5242880
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: c3e23ad4-e461-4c1c-a58c-0e6fc17cc14e
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=******, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-checksum-crc32;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=15953e6b7d081c3fbdd8537dd5b4898c59fa0dfc5967e5d885f14a7ca60628c7
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Checksum-Crc32: m0lNkw==
X-Amz-Content-Sha256: e34d74e6bb37a93786c16bdb844ca7f465a7dbeba739ca0629ded58cb74d5a8a
X-Amz-Date: 20250213T010730Z
X-Amz-Sdk-Checksum-Algorithm: CRC32

feature/s3/manager v1.17.57 (depending on service/s3 v1.76.0) with options:

X-Amz-Checksum-Crc32 and X-Amz-Sdk-Checksum-Algorithm headers still exist.

p..o....PUT /b107bc8e-3fa1-48cc-b91c-6dc8da2f5a42/orig_key?partNumber=1&uploadId=ZjQwMzA0NzMtOWM4Zi00OGM1LWI5MzktYTE0Yjg5MTRlMzE1LjNjY2M4OWFkLWNjOGYtNGZiYS1hOWE3LTJmYTMzNDAwYTgxZHgxNzM5NDA4NDQxMzEyNjk5OTQ0&x-id=UploadPart HTTP/1.1
Host: localhost:39767
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.76.0 ft/s3-transfer m/E,G,a
Content-Length: 5242880
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: ff0ca337-d00d-43ed-8955-01b2fe85f103
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=******, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-checksum-crc32;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=6548e5a0a38194a7a299d9106e0d3a1fdfadbba0dfeb20dab86aa7f0c8db0997
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Checksum-Crc32: iC+CIQ==
X-Amz-Content-Sha256: 2b4c61869e5b7fa0acc8e94c84d4c7a41a2c1c1c8eecfe6b22b6efa35792a324
X-Amz-Date: 20250213T010041Z
X-Amz-Sdk-Checksum-Algorithm: CRC32

feature/s3/manager v1.17.45 (depending on service/s3 v1.72.0):

p..u....PUT /9bce1f40-a919-4149-bcfa-02bcb33f7d4d/orig_key?partNumber=1&uploadId=MDMxOTMxNjQtYjllNC00YTcyLThhYzktNDAyMzk1Y2ExNmU1LmU0YzdjOTFlLTM1MjMtNDM5Yi04ODY3LTQ0ODg1YzU4YjVmM3gxNzM5NDA5MDIyMTgyNTkzMzg2&x-id=UploadPart HTTP/1.1
Host: localhost:34109
User-Agent: aws-sdk-go-v2/1.36.0 ua/2.1 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.72.0 ft/s3-transfer m/E,G
Content-Length: 5242880
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 9050aa85-a034-4a7b-8280-39339f545276
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=******, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=a69ec41b84526aabbd3248d4bfe7660794886024e0d84d70577f32985c5a3f27
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Content-Sha256: 8644553ab2366153f25ef050e09e558948904724a46b0fece5000562e048fb5a
X-Amz-Date: 20250213T011022Z

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

The integrity check can be disabled for multipart upload by Uploader.Upload() by setting both request_checksum_calculation and response_checksum_validation to when_required.

Current Behavior

The integrity check cannot be disabled for multipart upload by Uploader.Upload() even if both request_checksum_calculation and response_checksum_validation are set to when_required.

Reproduction Steps

I created a client as follows.

	c = s3.NewFromConfig(cfg, s3util.WithBaseEndpoint(conf.EndpointURL), s3util.WithPathStyle(), func(options *s3.Options) {
		options.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
		options.ResponseChecksumValidation = aws.ResponseChecksumValidationWhenRequired
	})

	uploader = manager.NewUploader(c, func(u *manager.Uploader) {
		u.PartSize = partSize
	}, manager.WithUploaderRequestOptions(func(options *s3.Options) {
		options.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired
		options.ResponseChecksumValidation = aws.ResponseChecksumValidationWhenRequired
	}))

Then called uploader.Upload().

	params := &s3.PutObjectInput{
		Bucket:   &bucketName,
		Body:     body,
		Key:      &key,
		Metadata: metadata,
	}

	_, err := c.uploader.Upload(ctx, params)

Possible Solution

Although I'm not entirely sure, it seems that the X-Amz-Sdk-Checksum-Algorithm header is set regardless of the options for UploadPart() which is internally called by Uploader.Upload(). https://github.com/aws/aws-sdk-go-v2/blob/e40a4a187f860dfa9e2a7f7782c4b5339f29d413/feature/s3/manager/upload.go#L736

Additional Information/Context

Uploader.Upload worked fine with small enough objects which resulted in the single-part upload (= PutObject).

feature/s3/manager v1.17.57 (depending on service/s3 v1.76.0) with options (small object):

p.......PUT /2248ca9e-6a3c-41d4-a9dc-4c5dc6af3638/orig_key?x-id=PutObject HTTP/1.1
Host: localhost:35953
User-Agent: aws-sdk-go-v2/1.36.1 ua/2.1 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.76.0 ft/s3-transfer m/E,G,a
Content-Length: 10
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 7500b7f2-8e29-4ed5-a4ee-99062d5784e4
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=******, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=72486789dcd72fbfbb954e86c0df3cc56dd5cd7bddc5abf3708fc5cd341f30f6
Content-Type: application/octet-stream
X-Amz-Content-Sha256: d6cca940af04ec82989a2c9c517f7aef701ae6b233c3fed108cabc26660f7b6d
X-Amz-Date: 20250213T010236Z

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.57 github.com/aws/aws-sdk-go-v2/service/s3 v1.76.0

Compiler and Version used

go version go1.23.2 linux/amd64

Operating System and version

Ubuntu 22.04.5 LTS

[wty-Bryant]

The integrity check was added to current transfer manager's multipart upload by default after flex checksum update to make sure same algorithm is used between CreateMultipartUpload and UploadPart. The config above was omitted and will be added when we have bandwidth.

[benmcclelland]

Is there any possibility in raising the priority to something more than "minor" since this is currently breaking all 3rd party S3 servers that don't (yet) support the integrity checks? This is forcing clients to pin the following versions to be able to support non-AWS S3 endpoints:

require (
	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.49
	github.com/aws/aws-sdk-go-v2/service/s3 v1.72.3
)

[lucix-aws]

Compatibility with arbitrary 3rd-party S3 clones is not going to be a driving factor for prioritization.

[valyala]

FYI, this bug at AWS SDK breaks backups to S3-compatible storage systems at VictoriaMetrics - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8622 . It would be great fixing it or at least providing working workaround, which will allow disabling integrity checks via config options at AWS SDK, so its behavior will be equal to the previous behavior. As a temporary workaround we had to pin the old AWS SDK version, which doesn't include the integrity checks. I don't think this is a good long-term solution, since it will break whenever some new feature / config will be added into new AWS SDK versions, which will be needed for some of our users.

[stanhu]

@wty-Bryant What do you think about https://github.com/aws/aws-sdk-go-v2/pull/3148? I will do some testing with third-party S3 providers, but I wanted to see if the approach was okay with you.

[stanhu]

@wty-Bryant FYI, I tested awscli with the --debug flag and noticed that it uses CRC64NVME by default for both regular and multipart uploads:

x-amz-sdk-checksum-algorithm:CRC64NVME
x-amz-trailer:x-amz-checksum-crc64nvme

I'm curious why CRC32 was used by default here: https://github.com/aws/aws-sdk-go-v2/blob/e40a4a187f860dfa9e2a7f7782c4b5339f29d413/feature/s3/manager/upload.go#L779

awscli also appears to omit those request headers with AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED, so once https://github.com/aws/aws-sdk-go-v2/pull/3148 is merged then both the Go SDK and CLI should behave similarly.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.