AWS CLI uses streaming trailer chunked requests on newer versions

[halkyon]

I upgraded AWS CLI client and started getting MissingContentLength errors. Debugging the request, it seems like it's using STREAMING-UNSIGNED-PAYLOAD-TRAILER or STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER requests, but we don't support these.

PUT
/test-files/testfile.dat

content-encoding:aws-chunked
host:gateway.storjshare.io
transfer-encoding:chunked
x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date:20250121T032638Z
x-amz-decoded-content-length:1048576
x-amz-sdk-checksum-algorithm:CRC64NVME
x-amz-trailer:x-amz-checksum-crc64nvme

An error occurred (MissingContentLength) when calling the PutObject operation: You must provide the Content-Length HTTP header.

AWS CLI version: aws-cli/2.23.2 Python/3.12.6 Linux/6.8.0-51-generic exe/x86_64.ubuntu.24.

I downgraded the client to the previous version (aws-cli/2.15.21 Python/3.11.6 Linux/6.8.0-51-generic exe/x86_64.ubuntu.24 prompt/off) and it worked again. I guess AWS CLI decides to use the streaming payload trailer on newer versions. I couldn't find a workaround to turn it off, short of downgrading the client.

Test command: aws s3 --endpoint https://gateway.storjshare.io cp /tmp/testfile.dat s3://test-files/testfile.dat (1M file), and no config changes in ~/.aws/config. It seems any size file it tries to stream, and also the same for multi-part uploads.

References:

• AWS docs: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming-trailers.html
• recent minio fix: https://github.com/minio/minio/commit/76913a9fd5c6e5c2dbd4e8c7faf56ed9e9e24091
• https://review.dev.storj.io/c/storj/minio/+/6842

[halkyon]

https://raw.githubusercontent.com/aws/aws-cli/v2/CHANGELOG.rst

2.23.0
...
* feature:``s3``: The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if the SDK has not implemented the specified checksum algorithm then this validation is skipped. Checksum validation behavior can be configured using the ``when_supported`` and ``when_required`` options - in the shared AWS config file using ``response_checksum_validation``, and as an env variable using ``AWS_RESPONSE_CHECKSUM_VALIDATION``.
...

Maybe this was the change?

[halkyon]

Artur and I discovered it changed in 2.23.0. It works in 2.22.35, which was the last version before it: https://github.com/aws/aws-cli/compare/2.22.35..2.23.0

[halkyon]

Related: https://github.com/aws/aws-cli/issues/9214

[halkyon]

we also need to support the new checksum types, minio fixed this upstream here: https://github.com/minio/minio/commit/827004cd6d3da8f49a5320c94ae74ae128156ed6

[heunland]

see also https://github.com/boto/boto3/issues/4392 <https://github.com/boto/boto3/issues/4392 We now have at least 2 customers who have run into this issue, workaround is to downgrade boto3 to v1.35.99 for now.

[amwolff]

w.r.t. the MinIO fix: we need a clean room implementation of this and cannot be inspired by or even be looking at the MinIO's implementation. I've created some issues that represent this effort:

• #90
• #91
• #92

We will assign someone who hasn't yet looked at these commits that Sean referenced.

[NiaStorj]

4/15/25: Moving back to backlog, will pick up once the team is ready

[stonematt]

This has now broken my app and refactoring to an old boto3 is an unscheduled effort. (My app is using s3fs/pandas so versioning is tricky.). Please fix.

[bahuma20]

Having this issue too, when trying to use the backup methods of CNPG (CloudNative Postgres) with the newest version.

[storj-gerrit[bot]]

Change images: downgrade awscli to fix builds mentions this issue.

[GodTamIt]

Just a comment here: I ran into this with CockroachDB backups as well.

AWS seems to have turned on full object checksumming on multipart uploads by default in at least a few of their SDKs now. This'll cause all projects using the default settings to start failing with checksum errors.

I posted a patch for Cockroach since they already have a built-in escape hatch for this, but I expect many people will run into this issue.

[jvdwrf]

Running into this issue as well with CloudNative Postgres backups