ex_aws_s3 icon indicating copy to clipboard operation
ex_aws_s3 copied to clipboard
verified publisher icon ex-aws

feat: SelectObjectContentRequest

Open avinayak opened this issue 2 years ago • 3 comments

This PR Adds support for SelectObjectContentRequest in S3. This let's users query contents of S3 Objects if the yare of type CSV, JSON or Parquet. Explanation for some design choices:

  • The output type of this request is a chunked EventStream(a binary format) encoded octet-stream.
  • There is no Range header for this request. So, we cannot use multiple http requests to stream this.
  • The only way to stream this is at the socket level (http streaming), which I added in https://github.com/ex-aws/ex_aws/pull/1012 @bernardd Can you please check this?

avinayak avatar Nov 05 '23 22:11 avinayak

ping @bernardd

avinayak avatar Nov 09 '23 00:11 avinayak

This is great work, would love to see this merged.

seanhuggins1 avatar Dec 07 '23 17:12 seanhuggins1

Thanks very much for your work on this and the corresponding PR over in ex_aws_s3, @avinayak. Please also accept my apologies for not getting to it sooner. It's been quite a busy end of the year over here. At the moment it looks like there's a bunch of tests that are failing as a result of the changes to the request family of function signatures. I've also made a note about the XML stuff. If you're able to take a look at that stuff and rebase it against the latest main, I'll do my best to get it merged as soon as I can.

bernardd avatar Dec 11 '23 06:12 bernardd