athena-federation-python-sdk icon indicating copy to clipboard operation
athena-federation-python-sdk copied to clipboard

Partition Support

Open drernie opened this issue 6 months ago • 1 comments

Partitions are not supported, so Athena will not parallelize the query using partitions.

The main reason we wanted to use this SDK was to create Athena tables build from multiple individual file partitions, as in:

https://github.com/quiltdata/quilt-package-metadata-athena/blob/main/04_create_datalake_and_athena_tables.ipynb

In particular, we wanted to use Athena Federation directly rather than having to hack it with symlinks.

This appears possible with Java, though I'm not entirely sure how.

Would you expect that to work? If not, how hard do you think it might be to add full Partition support?

drernie avatar Aug 13 '24 00:08 drernie