vortex
vortex copied to clipboard
Modular Encryption
This isn't a priority at the moment, but filing a tracking issue so users can upvote if it's important to them, as well as for us to remain aware of the requirements so as to leave the API holes required to implement this in the future.
See Parquet's modular encryption for more information: https://parquet.apache.org/docs/file-format/data-pages/encryption/
The general implementation in Vortex would be to:
- Add a set of encryption schemes (algorithm + key metadata) to the
FileLayoutflatbuffer. - Add a pointer into the encryption schemes vector to the
Segmentflatbuffer, allowing each segment to leverage a different encryption scheme. - Layout writers to be configured to store metadata / data with a specific encryption scheme, e.g. StructLayout could be configured to have different encryption per column.