parquet2
parquet2 copied to clipboard
Add support for reading files with modular encryption
See https://github.com/apache/parquet-format/blob/master/Encryption.md for motivation and technical details.
A starting point could be to generate an encrypted file using pyarrow e.g. a new parameter of the matrix in https://github.com/jorgecarleitao/parquet2/blob/main/tests/write_pyarrow.py, and try to read it on the integration tests. This should hint us into
- which APIs we need to change
- which parameters need to be passed
- which dependencies we need to add
cc @shicholas , based on https://github.com/pola-rs/polars/issues/3766