qbeast-spark
qbeast-spark copied to clipboard
Unify Table Properties structure and storage location
When saving the data as a table (either with CREATE TABLE SQL statement or a saveAsTable method) we save columnsToIndex, cubeSize and columnStats in the Metastore. This metadata might not be retrievable if not stored in the Delta Log, making it unreliable for other sources.
We should discuss and understand:
- Which format should save the Properties (columnsToIndex, for example, should contain the column type)?
- Where should we store this data? Both in Metastore and Commit Log? Just in the Commit Log?