seatunnel
seatunnel copied to clipboard
apache
[Feature]support latest version of Spark 3.2.0
https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-24-to-30
Hi @RickyHuo, Please have a look at the PR when possible and let me know if anything else is required.
Thanks
@ashmeet-kandhari Thanks for your contribution.
I want to know the version compatibility with Spark lower version or Scala because I know Spark 3.X built with Scala 2.12.
If we can use spark 2.4 local with seatunnel built on Spark 3.2 or not?
@RickyHuo From what I see, there are API changes due to which this is not possible without adding some logic to look at the api (Issue comes with JDBCUtils.scala file)
How is it going ,in my opinion it's better to be compatible with both spark2 and spark3，spark version can be passed in maven as a parameter，and I can take some help if needed.
@RickyHuo From what I see, there are API changes due to which this is not possible without adding some logic to look at the api (Issue comes with JDBCUtils.scala file)
Some connectors or class may need have different implement for spark2 and spark3.
How is it going ,in my opinion it's better to be compatible with both spark2 and spark3，spark version can be passed in maven as a parameter，and I can take some help if needed.
This method can‘t come true, because spark api have change between 2.x and 3.x. We can‘t maintain two version api in one project
How is it going? @ashmeet-kandhari If you don't have enough time, I will do this recently.
Hi @BenJFan,
I missed the updates recently. Yes, please can you look into it, as am currently pre-occupied.
Thanks for offering :)
support spark3.2.0, spark3.1.2 and spark2.4.7, please see https://github.com/nishuihanqiu/incubator-seatunnel with branch 2.1.3-spark3, that is 2.1.3-spark3
