datafusion-comet
datafusion-comet copied to clipboard
Enable `spark.comet.sparkToColumnar.enabled` when running Spark SQL tests
What is the problem the feature request solves?
Many (most?) Spark SQL tests do not start with Parquet scan, but use RDDScanExec, LocalTableScanExec, etc. so Comet falls back to Spark very early. This means that we are missing out on test coverage.
Describe the potential solution
- Make
spark.comet.sparkToColumnar.enabledconfigurable via env var - Set that env var in CI when running Spark SQL tests
- Fix failing tests (or ignore them until we can fix the bugs)
Additional context
No response