S3Client: access headers of successful responses

[pitrou]

Describe the feature

Be able to access HTTP headers of successful S3 responses from the returned Outcome instance.

(note: this is a repost of #1466).

Is your Feature Request related to a problem?

When a S3Client method returns with a successful Outcome, neither the outcome nor the wrapped result give you acces to the HTTP response headers. For example, you can't get the value of a hypothetical "x-amz-bucket-region" header. But there are probably other user cases. (conversely, in case of error, the AWSError class gives you access to the HTTP response details)

I tried using SetDataReceivedEventHandler to intercept the HTTP response, but that handler is only called if the response actually has a body. So HEAD requests don't trigger the handler...

Proposed Solution

A possible solution would be to add a const HeaderValueCollection& GetResponseHeaders () const method to Outcome (or to a dedicated subclass of Outcome).

Another possible solution would be to add a void SetHeadersReceivedEventHandler(...), that handler would be called once when all response headers are received (even if there's no body). The handler's type would probably be std::function<void(const HttpRequest*, HttpResponse*>.

Describe alternatives you've considered

Currently, I'm writing a S3Client subclass so that I can access protected members of S3Client and cook the HTTP request manually, call it using MakeRequest and then read the headers on the response. But that's cumbersome and fragile. Also, some thing are not available (for example ComputeEndpointString() is private, so I can't compute the region signer).

Acknowledge

• [X] I may be able to implement this feature request

AWS C++ SDK version used

1.9.219

Compiler and Version used

Ubuntu clang version 12.0.0-3ubuntu1~20.04.5

[pitrou]

Here is our workaround, which is a bit clumsy and probably does not support all authentication options correctly: https://github.com/apache/arrow/blob/778b1772fd20766e52b2bdccbd37668726f67e0c/cpp/src/arrow/filesystem/s3fs.cc#L540-L578

[jmklix]

Thanks for the feature request. This is something that we want to support on this sdk, but I don't have a timeline for when it will get added.

[sbiscigl]

So talked about it and what we're coming to is that if there is a specific header that is not modeled in a response, that is a modeling issue, and we want to push back on service teams to more correctly model the headers they are returning to us. So for this specific example of HeadBucket and x-amz-bucket-region S3 actually added that to their model and we now return it.

If you notice any headers that are in responses that are not modeled that you want, we can chase the team down to model them. This will solve the problem across all sdks as well and not just cpp.

open a ticket if you notice any others headers in response that are un-modeled

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.