Job runs with "job_cluster" type are getting filtered out

[sudharshanraja-db]

Customer reported an issue (173812) that some of the run ids are missing in job run gold table.

On further debugging we identified that these are present in bronze but getting filtered out at silverTransforms.

[sudharshanraja-db]

[sudharshanraja-db]

cc @piyushagarwalla-db

[radekhag]

I have noticed in 0.6.1.0 that the problem occurs especially when using a reused job cluster which utilizes node pools in a multi-task job. One task in said job that happens on an interactive cluster created a record in silver and gold layer correctly.

[GeekSheikh]

Note that this will be resolved with #342 and #353 -- we're trying to get this merged and resolved as part of 0.6.1.2 -- at the very least as a workaround. It may be until 0.6.2.0 (when we can make schema changes) that this is fully fixed. Please standby and follow those other tickets. Will begin development on this in the next week or two.