datafusion-comet
datafusion-comet copied to clipboard
Add documentation on building with Spark 4
What is the problem the feature request solves?
I can run individual test suites against Spark 3.4 with the following command:
./mvnw test -DwildcardSuites="org.apache.comet.CometExpressionSuite" -P"spark-3.4"
If I change the Spark version to 4.0 then I get compilation errors:
[ERROR] /Users/andy/git/apache/datafusion-comet/common/src/main/java/org/apache/comet/parquet/AbstractColumnReader.java:26:33: error: cannot access DataType
[ERROR] /Users/andy/git/apache/datafusion-comet/common/src/main/java/org/apache/comet/vector/CometVector.java:34:33: error: cannot access Decimal
[ERROR] /Users/andy/git/apache/datafusion-comet/common/src/main/java/org/apache/comet/vector/CometVector.java:36:38: error: cannot access ColumnVector
...
I would like to see some documentation in the installation guide and development guide on how to build and test locally with Spark 4.
Describe the potential solution
No response
Additional context
No response