Allow non-retryable sources when uploading files to S3

[edevil]

trafficstars

Describe the feature

Ability to build a ByteStream from an implementation of AsyncBufRead.

Use Case

A simple use-case is, we want to read some data from an existing stream (prior S3 download, for example), compress it on the fly, and upload it.

Example code that currently does NOT work:

    let get_resp = aws_client
        .get_object()
        .bucket(AWS_BUCKET_NAME)
        .key(AWS_RBL_PATH)
        .send()
        .await?;

    let read = StreamReader::new(get_resp.body.map_err(convert_smithy_error));
    let z_encoder = ZstdEncoder::with_quality(read, Level::Precise(5));

    let put_resp = s3_client
        .put_object()
        .bucket(S3_BUCKET)
        .key(S3_DESTINATION_FILE)
        .body(z_encoder)
        .send()
        .await?;

If we assume we don't want retries, it should be possible to use a simple AsyncBufRead implementation for the PUT operation.

Proposed Solution

No response

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [X] This feature might incur a breaking change

A note for the community

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue, please leave a comment

[Velfi]

I'm not sure that allowing for the creation of ByteStreams from types implementing AsyncBufRead is enough. I think the root issue here is that S3 requires us to set a content length unless we set the Transfer-Encoding: chunked header. I believe that encoding is not currently supported by the SDK. I'm working on a related issue right now and I may have an answer to this, although I can't give you a specific time estimate.

[Velfi]

You may be able to accomplish this right now with a multipart upload, although you'd have to manually chunk the stream into 5Mb chunks. We don't have an official example of doing this but we've posted about it in a previous issue.

Let me know if you have any questions about how to adapt that example to your use case.