apache
Improve performance of TPC-H q16
What is the problem the feature request solves?
Comet is currently slower than Spark for TPC-H q16.
This query does not run fully native yet.
Issues
- BuildRight with LeftAnti is not supported (https://github.com/apache/datafusion-comet/issues/457)
- BroadcastExchange is not supported (confusing error, needs improving)
- BroadcastHashJoin disabled because not all child plans are native
- SortMergeJoin disabled because not all child plans are native
- Disabling coalesce partitions (so that we can enable Comet shuffle) has quite a negative impact on performance - https://github.com/apache/datafusion-comet/issues/387
Describe the potential solution
No response
Additional context
No response
As an experiment, I removed our check where we fall back to Spark for BuildRight with LeftAnti so that the query runs natively (possibly with incorrect results) and performance increased significantly and matched Spark (but was not faster). I no longer saw the "BroadcastExchange is not supported" error.
At the end of the query we are performing a RowToColumnar to move to Comet just for a query stage that performs a sort and then perform a ColumnarToRow. This likely explains why we currently make the query slower than Spark.
This issue is largely out of date, and current issues are noted in https://github.com/apache/datafusion-comet/issues/391