datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Enable `spark.comet.sparkToColumnar.enabled` when running Spark SQL tests

Open andygrove opened this issue 1 month ago • 0 comments

What is the problem the feature request solves?

Many (most?) Spark SQL tests do not start with Parquet scan, but use RDDScanExec, LocalTableScanExec, etc. so Comet falls back to Spark very early. This means that we are missing out on test coverage.

Describe the potential solution

  • Make spark.comet.sparkToColumnar.enabled configurable via env var
  • Set that env var in CI when running Spark SQL tests
  • Fix failing tests (or ignore them until we can fix the bugs)

Additional context

No response

andygrove avatar Nov 06 '25 22:11 andygrove