aws-sdk-rust
aws-sdk-rust copied to clipboard
S3 PutObject: Set content length with a stream hyper Body for checksum
Describe the bug
Hi there!
We would want to set the content_length
(as size_hint
) with a streaming hyper:Body
to upload an object to S3 with it's checksum.
Expected Behavior
Upload the streaming body with it's checksum to S3.
Current Behavior
Fail with a message like Only request bodies with a known size can be checksum validated
coming from there
Reproduction Steps
We are doing something like:
async upload(from: Box<(dyn io::AsyncRead + Unpin + Send + Sync + 'static)>, from_size_bytes: i64)
let body = hyper::Body::wrap_stream(tokio_util::io::ReaderStream::new(from));
let bytes_stream = ByteStream::new(SdkBody::from(body));
self.client
.put_object()
.bucket(self.bucket_name.clone())
.key(to.0.to_owned())
.set_metadata(metadata.map(|m| m.0))
.checksum_algorithm(ChecksumAlgorithm::Sha256)
.content_length(from_size_bytes) <--- a try to size_hint
.body(bytes_stream)
.send()
.await
.map_err(|e| {
metrics::inc_put_object_fail();
e
})?;
In Rusoto we used new_with_size to specify the size_hint in the body stream.
Possible Solution
No response
Additional Information/Context
No response
Version
├── aws-config v0.51.0
│ ├── aws-http v0.51.0
│ │ ├── aws-smithy-http v0.51.0
│ │ │ ├── aws-smithy-eventstream v0.51.0
│ │ │ │ ├── aws-smithy-types v0.51.0
│ │ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-types v0.51.0
│ │ │ ├── aws-smithy-async v0.51.0
│ │ │ ├── aws-smithy-client v0.51.0
│ │ │ │ ├── aws-smithy-async v0.51.0 (*)
│ │ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ │ ├── aws-smithy-http-tower v0.51.0
│ │ │ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ │ ├── aws-smithy-types v0.51.0 (*)
│ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ ├── aws-smithy-types v0.51.0 (*)
│ ├── aws-sdk-sso v0.21.0
│ │ ├── aws-endpoint v0.51.0
│ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ ├── aws-smithy-types v0.51.0 (*)
│ │ │ ├── aws-types v0.51.0 (*)
│ │ ├── aws-http v0.51.0 (*)
│ │ ├── aws-sig-auth v0.51.0
│ │ │ ├── aws-sigv4 v0.51.0
│ │ │ │ ├── aws-smithy-eventstream v0.51.0 (*)
│ │ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ ├── aws-smithy-eventstream v0.51.0 (*)
│ │ │ ├── aws-smithy-http v0.51.0 (*)
│ │ │ ├── aws-types v0.51.0 (*)
│ │ ├── aws-smithy-async v0.51.0 (*)
│ │ ├── aws-smithy-client v0.51.0 (*)
│ │ ├── aws-smithy-http v0.51.0 (*)
│ │ ├── aws-smithy-http-tower v0.51.0 (*)
│ │ ├── aws-smithy-json v0.51.0
│ │ │ └── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-types v0.51.0 (*)
│ ├── aws-sdk-sts v0.21.0
│ │ ├── aws-endpoint v0.51.0 (*)
│ │ ├── aws-http v0.51.0 (*)
│ │ ├── aws-sig-auth v0.51.0 (*)
│ │ ├── aws-smithy-async v0.51.0 (*)
│ │ ├── aws-smithy-client v0.51.0 (*)
│ │ ├── aws-smithy-http v0.51.0 (*)
│ │ ├── aws-smithy-http-tower v0.51.0 (*)
│ │ ├── aws-smithy-query v0.51.0
│ │ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-smithy-types v0.51.0 (*)
│ │ ├── aws-smithy-xml v0.51.0
│ │ ├── aws-types v0.51.0 (*)
│ ├── aws-smithy-async v0.51.0 (*)
│ ├── aws-smithy-client v0.51.0 (*)
│ ├── aws-smithy-http v0.51.0 (*)
│ ├── aws-smithy-http-tower v0.51.0 (*)
│ ├── aws-smithy-json v0.51.0 (*)
│ ├── aws-smithy-types v0.51.0 (*)
│ ├── aws-types v0.51.0 (*)
├── aws-sdk-s3 v0.21.0
│ ├── aws-endpoint v0.51.0 (*)
│ ├── aws-http v0.51.0 (*)
│ ├── aws-sig-auth v0.51.0 (*)
│ ├── aws-sigv4 v0.51.0 (*)
│ ├── aws-smithy-async v0.51.0 (*)
│ ├── aws-smithy-checksums v0.51.0
│ │ ├── aws-smithy-http v0.51.0 (*)
│ │ ├── aws-smithy-types v0.51.0 (*)
│ ├── aws-smithy-client v0.51.0 (*)
│ ├── aws-smithy-eventstream v0.51.0 (*)
│ ├── aws-smithy-http v0.51.0 (*)
│ ├── aws-smithy-http-tower v0.51.0 (*)
│ ├── aws-smithy-types v0.51.0 (*)
│ ├── aws-smithy-xml v0.51.0 (*)
│ ├── aws-types v0.51.0 (*)
├── aws-smithy-http v0.51.0 (*)
├── aws-types v0.51.0 (*)
Environment details (OS name and version, etc.)
Linux localhost 5.15.0-48-generic #54-Ubuntu SMP Fri Aug 26 13:26:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Logs
No response