vortex icon indicating copy to clipboard operation
vortex copied to clipboard

Modular Encryption

Open gatesn opened this issue 11 months ago • 0 comments

This isn't a priority at the moment, but filing a tracking issue so users can upvote if it's important to them, as well as for us to remain aware of the requirements so as to leave the API holes required to implement this in the future.

See Parquet's modular encryption for more information: https://parquet.apache.org/docs/file-format/data-pages/encryption/

The general implementation in Vortex would be to:

  • Add a set of encryption schemes (algorithm + key metadata) to the FileLayout flatbuffer.
  • Add a pointer into the encryption schemes vector to the Segment flatbuffer, allowing each segment to leverage a different encryption scheme.
  • Layout writers to be configured to store metadata / data with a specific encryption scheme, e.g. StructLayout could be configured to have different encryption per column.

gatesn avatar Jan 10 '25 09:01 gatesn