Unable to track the cluster deletions

[Neha-vs123]

Scenario 1: Created a cluster. Later on, the cluster was not found(not sure how it got deleted). Tried to track the user who deleted it and when it got deleted, but the record was not found in the cluster table in the consumer database. The same cluster id is present in the bronze table.

Scenario 2: Reproduced the issue by simply creating a cluster and deleting it very soon without any cluster usage. This particular cluster deleted_by info was also not found in the cluster table.

The following query can be used as a workaround to find out the cluster details for anyone who comes across the same issue.

import com.databricks.labs.overwatch.pipeline.{TransformFunctions, Schema} import TransformFunctions._ import com.databricks.labs.overwatch.utils._ import org.apache.spark.sql.functions._

val auditRaw = table("overwatch_global_etl.audit_log_bronze")
  .verifyMinimumSchema(Schema.auditMasterSchema)
  .selectExpr("*", "requestParams.*").drop("requestParams")

val frontCols = Array("cluster_id", "ts", "actionName", "userIdentity", "response", "userAgent")

display(
  auditRaw
    .filter('serviceName === "clusters")
    .filter('organization_id === "demo")
    .withColumn("cluster_id",
      when('actionName === "create", get_json_object($"response.result", "$.cluster_id"))
        .when('actionName =!= "create" && 'cluster_id.isNull, 'clusterId)
        .otherwise('cluster_id).alias("cluster_id"),
               )
    .withColumn("ts", from_unixtime('timestamp / 1000.0).cast("timestamp"))
    .filter('cluster_id === "0815-152441-i6ppf5y2")
    .cullNull
    .moveColumnsToFront(frontCols)
    .orderBy('timestamp)
)