parquet2 icon indicating copy to clipboard operation
parquet2 copied to clipboard

Add support for reading files with modular encryption

Open jorgecarleitao opened this issue 2 years ago • 2 comments

See https://github.com/apache/parquet-format/blob/master/Encryption.md for motivation and technical details.

jorgecarleitao avatar Jun 27 '22 06:06 jorgecarleitao

A starting point could be to generate an encrypted file using pyarrow e.g. a new parameter of the matrix in https://github.com/jorgecarleitao/parquet2/blob/main/tests/write_pyarrow.py, and try to read it on the integration tests. This should hint us into

  • which APIs we need to change
  • which parameters need to be passed
  • which dependencies we need to add

jorgecarleitao avatar Jun 27 '22 06:06 jorgecarleitao

cc @shicholas , based on https://github.com/pola-rs/polars/issues/3766

jorgecarleitao avatar Jun 27 '22 06:06 jorgecarleitao