hail
hail copied to clipboard
[batch] Optimize SQL query generated for listing jobs / batches / job groups
What happened?
The current query code uses EXISTS
and IN
to check whether a batch / job / job group has a particular attribute. This is extremely slow especially for queries that have a lot of matches. The fix for this is to use a LEFT JOIN LATERAL
. However, by adding a new optional table join, I believe the code has to be more complicated than it currently is because there's a second WHERE clause. Note that the fix needs to be applied to both v1 and v2 queries.
See https://github.com/hail-is/hail/pull/14282/files#r1490076943 for more discussion.
Version
0.2.128
Relevant log output
No response