RequestBody/AsyncRequestBody from HTTP source

[singhbaljit]

Describe the feature

There are various RequestBody/AsyncRequestBody types that allow uploading files/buffers/Streams to S3. However, it would be great if the source can be HTTP (and HTTPS). Currently, one can fetch the file with their choice of HTTP client, and send to S3 clients. However, it requires dealing with low-level files/InputStream/OutputStream, and copying response headers, like content-type and content-length to the request headers for S3. This is also highly error prone in async context, i.e. dealing with non-blocking IO. The HTTP source should allow defining headers to pass in auth headers, etc.

Use Case

It is frustrating to stream a file from another server to S3 in a performant way.

Proposed Solution

No response

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

AWS Java SDK version used

2

[debora-ito]

I don't quite understand how this HTTP/HTTPS source would work, can you provide some examples? Like a sample code showing how you expect this interface to work.

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[singhbaljit]

Here's an example usage:

class SampleS3HttpUpload {

  private final S3AsyncClient s3 = S3AsyncClient.create();

  public void uploadHttp() {

    // made-up APIs
    AsyncRequestBody body = AsyncRequestBody.fromHttpSource("https://upload.wikimedia.org/wikipedia/commons/9/93/Amazon_Web_Services_Logo.svg")
            .header("Authorization", "Bearer some-token")
            .build();

    s3.putObject(req -> req.bucket("some-bucket").key("some-key"), body).join();
  }
}

Basically, I would expect Netty (or CRT, or Apache/URL clients for synchronous calls) to download the file, and upload it to S3. All of the low level piping of InputStream/OutputStream is handled by AWS SDK. Moreover, some response headers, such as content-type and content-length are also copied to the S3 object.

[debora-ito]

@singhbaljit thank you for the example, we understand it better now.

If we would add this option to the S3 client, it would probably be in the TransferManager instead of the RequestBody/AsyncRequestBody interface. TransferManager would be a more suitable place to handle additional logic like request headers, and RequestBody would be there just to handle body content.

We are going to keep this feature request open to gather interest and more feedback on other use cases.

Community note: if you are interested in seeing this feature in the SDK, add a 👍 reaction to the original description to help us with prioritization. Please feel free to comment on other details you'd like to see in this.