Feature request: support for s3 select

[alexvpickering]

Would be great to see support for the new s3 select. Thank you for the great software!

[n1ru4l]

It seems like this is a very complex feature.

I think unless one of the maintainers uses s3 select, it is very unlikely to be implemented any time soon.

If you are motivated to implement this feature we could accept a pull request 😊.

[ad-m]

@n1ru4l , why do you see it as complex? if I understand correctly the functionality is based on only one endpoint: https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html

Endpoint must read the stored as stream, decompress stream (if apply), transform stream from CSV / JSON / Parquet to XML containing only a subset of data.

Transformation from CSV / JSON / Parquet via SQL is common for query engines. Contrary to appearances, there are query engines that are independent of a specific database engine. See for example https://drill.apache.org/ .

Before introducing a full-fledge query engine, we can use a subset of the SQL language for column selection. For SQL syntax, there are quite a few SQL parsers published in npm.