[EPIC] - Publish DBSQL Gold Entities

[GeekSheikh]

Originally, SQL Analytics Endpoints were identified as specialized clusters but they have since been modified into an entirely separate entity. As such, at this time, no SQL analytics data is being captured in the clusters_snapshot table but further investigation is necessary to determine how the audit logs have changed.

Targeting this resolution for next release 0.4.2

Scope and Steps

1. Get the dataframe from API . This will be defined in Workspace.scala. It will be similar to that of clusers_snapshot_bronze and jobs_snapshot_bronze this will act as a lookup for missing audit data in Silver
2. Define a endPointsTarget. Just make sure it's defined as a Bronze target
3. Create a module inside Bronze. Make sure it has an appropriate module ID and name. The schema scrubber implementation should NOT be INSIDE BRONZE -- but rather a function defined inside the trait, BronzeTransforms -- as needed
4. You create a process that implements ETL Definition inside Bronze.scala . You create a process that implements ETL Definition inside Bronze.scala -- this is what will actually call the functions in the BronzeTransformations that perform the schema cleansing, etc
5. Create a pipeline to execute the module and load the Target. Bronze() is a pipeline so all you need to do is check the Overwatch scope to see if sqlAnalytics is present -- if it is, execute the process defined in Step 4, otherwise skip which leads to step 0-ish which is likely add sqlAnalytics scope to the Overwatch Scope enumeration object and validate any requisite dependencies (there may not be any in bronze)

[GeekSheikh]

Below is the code references to the issue and this will follow through to costs by cluster type in the gold layer.

isSQLAnalytics Boolean

[GeekSheikh]

Query History was added in 0.7.0 endpoints and other dbsql dims will be coming soon.