hail icon indicating copy to clipboard operation
hail copied to clipboard

[batch] Optimize SQL query generated for listing jobs / batches / job groups

Open jigold opened this issue 3 months ago • 1 comments

What happened?

The current query code uses EXISTS and IN to check whether a batch / job / job group has a particular attribute. This is extremely slow especially for queries that have a lot of matches. The fix for this is to use a LEFT JOIN LATERAL. However, by adding a new optional table join, I believe the code has to be more complicated than it currently is because there's a second WHERE clause. Note that the fix needs to be applied to both v1 and v2 queries.

See https://github.com/hail-is/hail/pull/14282/files#r1490076943 for more discussion.

Version

0.2.128

Relevant log output

No response

jigold avatar Mar 13 '24 19:03 jigold