datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Physical planner makes assumption that join keys are specified in left = right order

Open andygrove opened this issue 2 years ago • 3 comments

Describe the bug

The following code fails for TPC-DS query 10 because the join keys are not in the expected order.

                    let join_on = keys
                        .iter()
                        .map(|(l, r)| {
                            let l = l.try_into_col()?;
                            let r = r.try_into_col()?;
                            Ok((
                                Column::new(&l.name, left_df_schema.index_of_column(&l)?),
                                Column::new(&r.name, right_df_schema.index_of_column(&r)?),
                            ))
                        })
                        .collect::<Result<join_utils::JoinOn>>()?;

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

andygrove avatar Jan 03 '23 02:01 andygrove

It looks like the logical plan builder should always put the columns in the correct order but this is not happening in some cases .. the fix should probably be in the logical plan builder and not the physical planner

andygrove avatar Jan 03 '23 03:01 andygrove

related issue #4688

jackwener avatar Jan 04 '23 16:01 jackwener

Edit: The issue described in this comment is unrelated to the OG issue. Hence I removed the comment and created a new ticket. Please see #4887.

askoa avatar Jan 12 '23 16:01 askoa

I think this has been resolved @andygrove please reopen if this is not the case

Dandandan avatar Sep 09 '24 18:09 Dandandan