overwatch
overwatch copied to clipboard
Data Model - Add Spark Cost Fact
Spark Costing Fact has been developed but it must be developed as a view. The resulting dataset is too large / too expensive to materialize for all tasks.
The code is complete in scala but is being converted to SQL to create the view. It's possible that we will release a function to expose the DF at first but goal is to get this into SQL to produce a view in gold data model.
will review this in the 0.6.2 milestone but it may not be feasible to materialize this within a reasonable cost; thus we need to find a way to create it as a view and provide guidance around filters.