datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

chore: Add `scanImpl` attribute to `CometScanExec`

Open andygrove opened this issue 7 months ago • 1 comments

Which issue does this PR close?

Follows on from https://github.com/apache/datafusion-comet/pull/1744

Diff between this PR and https://github.com/apache/datafusion-comet/pull/1744: https://github.com/andygrove/datafusion-comet/compare/scan-refactor-2...scan-refactor-3

Rationale for this change

This PR adds a scanImpl attribute to CometScanExec and this is populated at planning time in CometScanRule based on CometConf.COMET_NATIVE_SCAN_IMPL. All other references to CometConf.COMET_NATIVE_SCAN_IMPL have been removed from the implementation code.

This is a step toward supporting an "auto" mode in CometScanRule so that we can pick the best scan implementation and potentially have multiple scan implementations active within the same query.

What changes are included in this PR?

Add scanImpl attribute to CometScanExec and propagate this to other code.

How are these changes tested?

Existing tests.

andygrove avatar May 17 '25 13:05 andygrove

Codecov Report

Attention: Patch coverage is 66.66667% with 13 lines in your changes missing coverage. Please review.

Project coverage is 58.60%. Comparing base (f09f8af) to head (f6e6654). Report is 243 commits behind head on main.

Files with missing lines Patch % Lines
...n/scala/org/apache/comet/rules/CometScanRule.scala 53.33% 4 Missing and 3 partials :warning:
.../scala/org/apache/comet/serde/QueryPlanSerde.scala 71.42% 0 Missing and 2 partials :warning:
...ala/org/apache/spark/sql/comet/CometScanExec.scala 75.00% 0 Missing and 2 partials :warning:
.../apache/comet/parquet/CometParquetFileFormat.scala 66.66% 0 Missing and 1 partial :warning:
...n/scala/org/apache/comet/rules/CometExecRule.scala 0.00% 0 Missing and 1 partial :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1746      +/-   ##
============================================
+ Coverage     56.12%   58.60%   +2.47%     
- Complexity      976     1131     +155     
============================================
  Files           119      130      +11     
  Lines         11743    12663     +920     
  Branches       2251     2362     +111     
============================================
+ Hits           6591     7421     +830     
- Misses         4012     4058      +46     
- Partials       1140     1184      +44     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar May 17 '25 14:05 codecov-commenter