apache
[WIP] Add spark 35
Which issue does this PR close?
Adds Spark 3.5
Rationale for this change
Spark 3.5 is the latest Spark release.
What changes are included in this PR?
Adds Spark 3.5 support, this introduces a new per-spark version of shims as part of adding interfacing to the 3.5 APIs. This has the advantage of static compile checks over the existing shim approach which does reflection for everything, but I can switch to the runtime reflection model for consistency if that's preferred.
How are these changes tested?
WIP
Some class and function has changed in spark 3.5, maybe we need some more shim classes to support both spark 3.5 and the versions before.
I has a similar commit to do this: https://github.com/apache/arrow-datafusion-comet/commit/b038267eb5e6ddafccf84b0147760f6992f36077
Yup this adds some shims (there are existing shims too for 3.3 etc but they use reflection heavily).
FWIW I created https://github.com/apache/arrow-datafusion-comet/issues/140 to improve the current Spark shim handling.
I love it :)
On Thu, Feb 29, 2024 at 10:45 AM Chao Sun @.***> wrote:
FWIW I created #140 https://github.com/apache/arrow-datafusion-comet/issues/140 to improve the current Spark shim handling.
— Reply to this email directly, view it on GitHub https://github.com/apache/arrow-datafusion-comet/pull/127#issuecomment-1971741645, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAOT5NRC355APXEYYRFIADYV53MJAVCNFSM6AAAAABD5DERESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRG42DCNRUGU . You are receiving this because you authored the thread.Message ID: @.***>
FYI https://github.com/apache/datafusion-comet/pull/318 is just merged
Thanks for getting this started @holdenk. I have now created https://github.com/apache/datafusion-comet/pull/587 to add experimental support for Spark 3.5.1 (there will be follow up PRs to fix some test failures), so will close this one.
