qbeast-spark icon indicating copy to clipboard operation
qbeast-spark copied to clipboard

Mandatory load of 'columnsToIndex' and 'cubeSize' as properties in a Qbeast Table

Open osopardo1 opened this issue 1 year ago • 0 comments

When loading the properties in a Qbeast Table, both the Metadata from the Delta Log and the Catalog Properties are loaded.

Since v0.7.0, we added the properties columnsToIndex and cubeSize, not present in any of the mentioned components. For new and older tables, would be important to verify and load these two table properties either from the Log or from the Catalog.

We should modify the code for QbeastTableImpl and:

  • Check if those properties are present as 'options.' or with the bare key.
  • If not present, output the latest Revision column names and cube size.

osopardo1 avatar Nov 26 '24 08:11 osopardo1