[DQ] - cluster state detail silver - merge condition

[GeekSheikh]

The merge condition is keyed off of timestamp_start but a new column must be created and used as the key.

When an OW run begins the state of a cluster is captured from where it left off until now. The current_state will not have an end_ts because it hasn't ended yet. To calculate costs through current OW run, the pipeline_snapshot time is used here. The next record on the subsequent OW run will be the end of the current state (from cluster_events_bronze) but it will be keyed in timestamp_state_start; thus the record doesn't get updated to reflect the proper end time resulting in the cluster appearing to be in two states at once until the pipeline_snapshot time of the previous run.

To correct this a new TS key field should be generated to key the merge and update the last record from the previous run. This solution needs to be reviewed and detailed out.

The reason this happens is because cluster_state_detail is driven by cluster_events_bronze. Each state from cluster_events_bronze ONLY has a timestamp the start/end are derived based off a lead/lag resulting in one record per key (i.e. cluster) getting extra time in state for each run.

[GeekSheikh]

@sawankulkarni-db @souravbaner-da -- looks like this solution didn't make it into 0.7.0...can you confirm? This is unfortunate as it was a very MINOR change. I'm not sure where this PR went, I thought it was merged but it looks like the code still isn't updated. Let's be sure to get this into the next dot release coming next week.

[sawankulkarni-db]

@GeekSheikh - This change did not go in 0700. Please let me know of the source branch from which we can create a new branch and push the changes. Thanks!