apache
[Feature] Update Flink dependency to 2.0
Search before asking
- [x] I searched in the issues and found nothing similar.
Motivation
Flink 2.0 has been released recently, and Paimon's Flink version should be updated to 2.0.0 to allow adapting to Flink's new features.
Solution
No response
Anything else?
Most updating work has been done in #4442.
Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Paimon Flink CDC relies on Flink Kafka connector, which has not provided a version compatible with Flink 2.0 yet. There has been an PR(link) about it, and this issue is supposed to start shortly after the PR is completed.
I am getting this error when trying to use FlinkSinkFactory for paimon. I think the references to SinkFunction need to be removed entirely, not just set to null
java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/functions/sink/SinkFunction
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022)
...
The kafka connector is now updated to support 2.0!
Sorry for the late reply and thanks for the reminder! I'm working on the update and as you said, the change fron SinkFunction to v2 sink is bigger than expected.
Apart from issues related to Kafka connector, Paimon also depends on Flink CDC connectors like MySql cdc, PostgresSql cdc and MongoDB cdc. Flink CDC has not updated to Flink 2.x yet. Thus we may still need to withhold the update of Paimon CDC to Flink 2.0.
Flink CDC has a dependency on Paimon as well, which has led to a situation where both components are expecting the other to be updated first, resulting in a temporary update standstill.
Flink CDC has a dependency on Paimon as well, which has led to a situation where both components are expecting the other to be updated first, resulting in a temporary update standstill.
There seems to be a plan to migrate the CDC implementation of Paimon sink to this repo, and Flink CDC would just reply on a dependency of Paimon. Part of this job is being worked on in #6358.