datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard
verified publisher icon apache

Support sort merge join with a join condition

Open viirya opened this issue 1 year ago • 6 comments

What is the problem the feature request solves?

Currently SMJ with join condition is not supported by Comet and falls back to Spark. The feature was added into DataFusion but we've not incorporated it yet. We should do it to add the support.

Describe the potential solution

No response

Additional context

No response

viirya avatar May 07 '24 18:05 viirya

@viirya can I take it as I spent some time with SMJ?

comphead avatar May 31 '24 21:05 comphead

Re-opening this since we discovered that there are some correctness and performance issues and currently have this feature disabled by default

andygrove avatar Sep 20 '24 17:09 andygrove

There is an existing test SortMergeJoin with join filter that we need to enable as part of closing this issue. It was ignored in https://github.com/apache/datafusion-comet/pull/1057

andygrove avatar Nov 05 '24 18:11 andygrove

Thanks @andygrove I'll take if from now

comphead avatar Nov 05 '24 18:11 comphead

Waiting for DataFusion 44.0.0

comphead avatar Nov 18 '24 22:11 comphead

Re-opening this issue because we only enable this feature in tests currently due to poor performance in benchmarks

andygrove avatar Jan 10 '25 15:01 andygrove