Oleks V

Found when fixing SMJ bugs https://github.com/apache/datafusion/pull/10304

cc @korowa you might be interested

@alamb @ozankabak @viirya @mustafasrepo @berkaysynnada @metesynnada appreciate your inputs.

it is used if next conditions met https://github.com/apache/arrow-datafusion/blob/81c96fc3db0ea35638278f32df066be63b745a51/datafusion/core/src/physical_planner.rs#L1136 There is also a small set of tests introduced in `sort_merge_join.slt`. And the plans there shows SMJ To enforce SMJ its needed...

I'm thinking if its enough to add fuzz tests, prob we also need to run benchmarks on top of SMJ? Afaik now benchmarks are on top of the HJ?

we shouldn't be comparing HJ vs SMJ 1:1, but the performance has to be quite close? What I'm trying to solve is to find a strategy to remove the experimental...

I ran TPCH benchmarks for SMJ and got ``` thread 'tokio-runtime-worker' panicked at datafusion/physical-plan/src/joins/sort_merge_join.rs:1357:22: index out of bounds: the len is 0 but the index is 1 note: run with...

> Seems like a good reason to keep it marked as experimental I'll create a separate issue on it. Once TPCH passed we can get back on SMJ status

Check the LeftAnti issue reported in HJ https://github.com/apache/datafusion/issues/10583 to be handled also in SMJ

Thanks @edmondop I'm already working on it, if you have spare time and would like to work on SMJ its really needed to add fuzz tests for SMJ when the...