datafusion-comet
datafusion-comet copied to clipboard
apache
[EPIC] Add support for Spark 4.0.1
What is the problem the feature request solves?
Spark 4 will soon be available, so we should coordinate the work to add support. We already have partial support for Spark 4.0.0-preview1.
Let's use this issue to track the work and create issues as needed.
- https://github.com/apache/datafusion-comet/pull/1830
- https://github.com/apache/datafusion-comet/issues/1974
- https://github.com/apache/datafusion-comet/issues/1846
- https://github.com/apache/datafusion-comet/issues/1942
- https://github.com/apache/datafusion-comet/issues/1941
- https://github.com/apache/datafusion-comet/issues/1947
- https://github.com/apache/datafusion-comet/issues/1948
- https://github.com/apache/datafusion-comet/issues/1949
- https://github.com/apache/datafusion-comet/issues/1967
- https://github.com/apache/datafusion-comet/issues/551
- https://github.com/apache/datafusion-comet/issues/313
- https://github.com/apache/datafusion-comet/issues/2408
- Perform audit of all changes noted in https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-35-to-40 and file issues as needed
- https://github.com/apache/datafusion-comet/issues/1989
Describe the potential solution
No response
Additional context
No response
Investigate Guava duplicated classes https://github.com/apache/datafusion-comet/issues/1968
I can work on some of the low hanging issues @andygrove @kazuyukitanimura
