lakeFS icon indicating copy to clipboard operation
lakeFS copied to clipboard

Add support for exporting delta tables min_reader v3 and min_writer v7, including reader/writer features

Open ion-elgreco opened this issue 1 year ago • 1 comments

The delta catalog exporter doesn't support exporting tables with min_reader_version of 3 and min_writer_version of 7 which has reader and writer features.

This means you won't be able to export new tables that follow this new protocol path. Delta-spark and delta-rs and other implementations are moving forward with managing the protocol this way.

ion-elgreco avatar May 07 '24 11:05 ion-elgreco

See here the docs: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#table-features, and example of the structure:


{
  "protocol": {
    "readerVersion":3,
    "writerVersion":7,
    "readerFeatures":["columnMapping"],
    "writerFeatures":["columnMapping","identityColumns"]
  }
}

ion-elgreco avatar May 07 '24 11:05 ion-elgreco