qbeast-spark
qbeast-spark copied to clipboard
Unify Table Properties structure and storage location
When saving the data as a table (either with CREATE TABLE SQL statement or a saveAsTable method) we save columnsToIndex
, cubeSize
and columnStats
in the Metastore. This metadata might not be retrievable if not stored in the Delta Log, making it unreliable for other sources.
We should discuss and understand:
- Which format should save the Properties (columnsToIndex, for example, should contain the column type)?
- Where should we store this data? Both in Metastore and Commit Log? Just in the Commit Log?