dask-sql
dask-sql copied to clipboard
[DF] Implement subquery decorrelation optimizer rules
Is your feature request related to a problem? Please describe. PR https://github.com/apache/arrow-datafusion/pull/2885 adds three new optimizer rules for decorrelating subqueries and translating them into joins. This may result in more efficient queries in Dask SQL.
Describe the solution you'd like Temporarily copy these rules into Dark SQL and try them out. We can provide feedback before the PR gets merged into DataFusion.
Once the PR is merged we can remove our temporary copies of the code.
Describe alternatives you've considered None
Additional context None
Looks like these optimizer rules got introduced with #680, though they still don't work for certain queries, for example we see some warnings/errors in https://github.com/dask-contrib/dask-sql/pull/683#pullrequestreview-1085669860