qbeast-spark icon indicating copy to clipboard operation
qbeast-spark copied to clipboard

Unify Table Properties structure and storage location

Open osopardo1 opened this issue 7 months ago • 0 comments

When saving the data as a table (either with CREATE TABLE SQL statement or a saveAsTable method) we save columnsToIndex, cubeSize and columnStats in the Metastore. This metadata might not be retrievable if not stored in the Delta Log, making it unreliable for other sources.

We should discuss and understand:

  1. Which format should save the Properties (columnsToIndex, for example, should contain the column type)?
  2. Where should we store this data? Both in Metastore and Commit Log? Just in the Commit Log?

osopardo1 avatar Jul 24 '24 05:07 osopardo1