qbeast-spark icon indicating copy to clipboard operation
qbeast-spark copied to clipboard

Unify Create Qbeast Table in one step

Open osopardo1 opened this issue 1 year ago • 1 comments

Right now, the code to create a Qbeast Table without data (no SAVE AS), is done in two steps:

  1. Create an empty data frame and save it in the Location.
  2. Convert the Delta Table to the Qbeast Table with ConvertToQbeastCommand.

It is executed like that because we do not have access to methods in the CreateDeltaTableCommand, nor can we delegate the creation to that object. Otherwise (if we call directly CreateDeltaTableCommand), the table would be created in the Catalog as "delta", and the whole operation would fail.

The idea is to do both in the same transaction. Perhaps it is a good moment to build a CreateQbeastTableCommand to centralize the process.

osopardo1 avatar Jul 30 '24 10:07 osopardo1

We need a design document for this issue to understand clearly the implications of the change.

fpj avatar Oct 21 '24 13:10 fpj