overwatch icon indicating copy to clipboard operation
overwatch copied to clipboard

Data Model - Add Spark Cost Fact

Open GeekSheikh opened this issue 4 years ago • 1 comments

Spark Costing Fact has been developed but it must be developed as a view. The resulting dataset is too large / too expensive to materialize for all tasks.

The code is complete in scala but is being converted to SQL to create the view. It's possible that we will release a function to expose the DF at first but goal is to get this into SQL to produce a view in gold data model.

GeekSheikh avatar Jul 30 '21 10:07 GeekSheikh

will review this in the 0.6.2 milestone but it may not be feasible to materialize this within a reasonable cost; thus we need to find a way to create it as a view and provide guidance around filters.

GeekSheikh avatar May 11 '22 17:05 GeekSheikh