qbeast-spark
qbeast-spark copied to clipboard
Mandatory load of 'columnsToIndex' and 'cubeSize' as properties in a Qbeast Table
When loading the properties in a Qbeast Table, both the Metadata from the Delta Log and the Catalog Properties are loaded.
Since v0.7.0, we added the properties columnsToIndex and cubeSize, not present in any of the mentioned components. For new and older tables, would be important to verify and load these two table properties either from the Log or from the Catalog.
We should modify the code for QbeastTableImpl and:
- Check if those properties are present as 'options.' or with the bare key.
- If not present, output the latest Revision column names and cube size.