aws-sdk-rust icon indicating copy to clipboard operation
aws-sdk-rust copied to clipboard

S3 PutObject: Set content length with a stream hyper Body for checksum

Open andresrsanchez opened this issue 2 years ago • 3 comments

Describe the bug

Hi there! We would want to set the content_length (as size_hint) with a streaming hyper:Body to upload an object to S3 with it's checksum.

Expected Behavior

Upload the streaming body with it's checksum to S3.

Current Behavior

Fail with a message like Only request bodies with a known size can be checksum validated coming from there

Reproduction Steps

We are doing something like:


async upload(from: Box<(dyn io::AsyncRead + Unpin + Send + Sync + 'static)>, from_size_bytes: i64)
        let body = hyper::Body::wrap_stream(tokio_util::io::ReaderStream::new(from));
        let bytes_stream = ByteStream::new(SdkBody::from(body));

        self.client
            .put_object()
            .bucket(self.bucket_name.clone())
            .key(to.0.to_owned())
            .set_metadata(metadata.map(|m| m.0))
            .checksum_algorithm(ChecksumAlgorithm::Sha256)
            .content_length(from_size_bytes) <--- a try to size_hint
            .body(bytes_stream)
            .send()
            .await
            .map_err(|e| {
                metrics::inc_put_object_fail();
                e
            })?;

In Rusoto we used new_with_size to specify the size_hint in the body stream.

Possible Solution

No response

Additional Information/Context

No response

Version

├── aws-config v0.51.0
│   ├── aws-http v0.51.0
│   │   ├── aws-smithy-http v0.51.0
│   │   │   ├── aws-smithy-eventstream v0.51.0
│   │   │   │   ├── aws-smithy-types v0.51.0
│   │   │   ├── aws-smithy-types v0.51.0 (*)
│   │   ├── aws-smithy-types v0.51.0 (*)
│   │   ├── aws-types v0.51.0
│   │   │   ├── aws-smithy-async v0.51.0
│   │   │   ├── aws-smithy-client v0.51.0
│   │   │   │   ├── aws-smithy-async v0.51.0 (*)
│   │   │   │   ├── aws-smithy-http v0.51.0 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.51.0
│   │   │   │   │   ├── aws-smithy-http v0.51.0 (*)
│   │   │   │   ├── aws-smithy-types v0.51.0 (*)
│   │   │   ├── aws-smithy-http v0.51.0 (*)
│   │   │   ├── aws-smithy-types v0.51.0 (*)
│   ├── aws-sdk-sso v0.21.0
│   │   ├── aws-endpoint v0.51.0
│   │   │   ├── aws-smithy-http v0.51.0 (*)
│   │   │   ├── aws-smithy-types v0.51.0 (*)
│   │   │   ├── aws-types v0.51.0 (*)
│   │   ├── aws-http v0.51.0 (*)
│   │   ├── aws-sig-auth v0.51.0
│   │   │   ├── aws-sigv4 v0.51.0
│   │   │   │   ├── aws-smithy-eventstream v0.51.0 (*)
│   │   │   │   ├── aws-smithy-http v0.51.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.51.0 (*)
│   │   │   ├── aws-smithy-http v0.51.0 (*)
│   │   │   ├── aws-types v0.51.0 (*)
│   │   ├── aws-smithy-async v0.51.0 (*)
│   │   ├── aws-smithy-client v0.51.0 (*)
│   │   ├── aws-smithy-http v0.51.0 (*)
│   │   ├── aws-smithy-http-tower v0.51.0 (*)
│   │   ├── aws-smithy-json v0.51.0
│   │   │   └── aws-smithy-types v0.51.0 (*)
│   │   ├── aws-smithy-types v0.51.0 (*)
│   │   ├── aws-types v0.51.0 (*)
│   ├── aws-sdk-sts v0.21.0
│   │   ├── aws-endpoint v0.51.0 (*)
│   │   ├── aws-http v0.51.0 (*)
│   │   ├── aws-sig-auth v0.51.0 (*)
│   │   ├── aws-smithy-async v0.51.0 (*)
│   │   ├── aws-smithy-client v0.51.0 (*)
│   │   ├── aws-smithy-http v0.51.0 (*)
│   │   ├── aws-smithy-http-tower v0.51.0 (*)
│   │   ├── aws-smithy-query v0.51.0
│   │   │   ├── aws-smithy-types v0.51.0 (*)
│   │   ├── aws-smithy-types v0.51.0 (*)
│   │   ├── aws-smithy-xml v0.51.0
│   │   ├── aws-types v0.51.0 (*)
│   ├── aws-smithy-async v0.51.0 (*)
│   ├── aws-smithy-client v0.51.0 (*)
│   ├── aws-smithy-http v0.51.0 (*)
│   ├── aws-smithy-http-tower v0.51.0 (*)
│   ├── aws-smithy-json v0.51.0 (*)
│   ├── aws-smithy-types v0.51.0 (*)
│   ├── aws-types v0.51.0 (*)
├── aws-sdk-s3 v0.21.0
│   ├── aws-endpoint v0.51.0 (*)
│   ├── aws-http v0.51.0 (*)
│   ├── aws-sig-auth v0.51.0 (*)
│   ├── aws-sigv4 v0.51.0 (*)
│   ├── aws-smithy-async v0.51.0 (*)
│   ├── aws-smithy-checksums v0.51.0
│   │   ├── aws-smithy-http v0.51.0 (*)
│   │   ├── aws-smithy-types v0.51.0 (*)
│   ├── aws-smithy-client v0.51.0 (*)
│   ├── aws-smithy-eventstream v0.51.0 (*)
│   ├── aws-smithy-http v0.51.0 (*)
│   ├── aws-smithy-http-tower v0.51.0 (*)
│   ├── aws-smithy-types v0.51.0 (*)
│   ├── aws-smithy-xml v0.51.0 (*)
│   ├── aws-types v0.51.0 (*)
├── aws-smithy-http v0.51.0 (*)
├── aws-types v0.51.0 (*)

Environment details (OS name and version, etc.)

Linux localhost 5.15.0-48-generic #54-Ubuntu SMP Fri Aug 26 13:26:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Logs

No response

andresrsanchez avatar Nov 15 '22 06:11 andresrsanchez