beam
beam copied to clipboard
apache
[Feature Request]: Support for Apache Flink 1.18 in Beam Runners
What would you like to happen?
[Flink Runner] Question
Description
Our organization is in the process of updating our Flink clusters to the latest version, Flink 1.18. We currently use beam-runners-flink-1.16 to run Apache Beam pipelines on our Flink 1.16 cluster. We are concerned that our Beam pipelines will fail after the upgrade due to incompatibility with Flink 1.18. This is based on the information provided at https://beam.apache.org/documentation/runners/flink/: Flink Version Compatibility section.
Ask We would like to inquire about the timeline and plans for supporting Apache Flink 1.18 in Apache Beam. Is there an ongoing effort to add compatibility for Flink 1.18, or is it on the roadmap for a future release?
Additional Context
- Our pipelines are critical to our business operations, and we need to ensure a smooth transition during the Flink upgrade.
- We are willing to assist or contribute to the effort of adding Flink 1.18 support if guidance is provided on what is needed.
Potential Impact
- Without support for Flink 1.18, we may face significant challenges in maintaining our data processing workflows.
- Early information on support timelines will help us plan our upgrade strategy and resource allocation more effectively.
Questions
- Is there a planned release date for a version of Beam that supports Flink 1.18?
- Are there any known workarounds or interim solutions for running Beam pipelines on Flink 1.18?
- How can the community contribute to accelerating support for Flink 1.18 in Beam?
We appreciate any information or guidance the Apache Beam community can provide on this matter.
Thank you for your time and support.
eBay
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [X] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
+1 to this issue, my org is in the same boat (long story short: company-wide migration to Java 17, we need Flink 1.18, but we can't use it because of Beam's lack of support).
Some fun extra context: https://github.com/apache/beam/pull/30197
This MR to support Flink 1.17 is currently blocked because the upgrade would break compatibility with checkpoints created in older versions.
+1 to this issue, we are also in the similar situation at my org as well.
Hi all, looks like this is highly wanted. Could Flink runner user help testing #30197 for their use cases?
As of timeline, as an open source project the Flink runner component has been community driven. Any contribution is welcome.
I forked from that PR to add support for 1.18. It's a trivial change, too: https://github.com/thebozzcl/beam/commit/432643bfd70f1b8a6052c33a04774b93da5a7bae
So far this is working fine in our tests, with a caveat - in its current status, there's some incompatibilities with the publicly released libraries (mostly around SerializablePipelineOptions). For our tests, I had to use custom-built versions of a few libraries based on the repo:
- beam-runners-core-java
- beam-runners-java-fn-execution
- beam-runners-java-job-service
- beam-sdks-java-core
I'm hopeful that once the PR for 1.17 is merged, adding support for 1.18 will be trivial. We're going to try using these libraries in one of our more complex pipelines, see if everything still works.
Also, please note that the codebase in #30197 has diverged from the main branch quite a bit, so I'm not even sure my changes will apply once that's rebased.
@thebozzcl looks like 1.17 support was just merged in https://github.com/apache/beam/pull/30197
