[Feature] submit Flink CDC pipeline job to k8s cluster.

[lvyanquan]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

Currently, there is no a clear description to tell how to run a pipeline on k8s. So If there is no need to modify the code, please add some docs to guide users how to submit job to a Kubernetes cluster. If it's necessary to modify the code, you can submit a pr and add doc to fix it.

Solution

No response

Alternatives

No response

Anything else?

No response

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[czy006]

I think it can be regarded as an ordinary Flink application and can be submitted to Flink Operator or Flink Cluster. I am willing to take on this work. Can you assign it to me? @lvyanquan

[lvyanquan]

This is a comment requirement from many users, looking forward to your code 😊 @czy006

[czy006]

After discussion, we produced a version of relevant design documents : https://docs.google.com/document/d/1HlvRS5aVNqPtOx14mynEL3RFaZORuTFkWZAjvl4k998/edit?usp=sharing

[yeezychao]

I don’t see any discussion on yarn deployment in the document. Are you planning to support yarn mode? @czy006

[liuxinjie5211990]

After discussion, we produced a version of relevant design documents : https://docs.google.com/document/d/1HlvRS5aVNqPtOx14mynEL3RFaZORuTFkWZAjvl4k998/edit?usp=sharing

hello,i have the same problem,is there any plans to support yarn mode?All of our flink jobs are running on yarn flink session. @czy006

[edhe1988]

After discussion, we produced a version of relevant design documents : https://docs.google.com/document/d/1HlvRS5aVNqPtOx14mynEL3RFaZORuTFkWZAjvl4k998/edit?usp=sharing

Flink on YARN is the main deployment mode, why is there no discussion about the YARN deployment mode? @czy006

[herunkang2018]

@czy006 @lvyanquan Compared with submitting using flink-cdc.sh, I think reuse the flink client to submit flink cdc job is more reasonable and easier to maintain, while flink client already supports all kinds of submitting way, including k8s session/application mode and yarn session/application mode, and widely used in production environments.

[czy006]

After discussion, we produced a version of relevant design documents : https://docs.google.com/document/d/1HlvRS5aVNqPtOx14mynEL3RFaZORuTFkWZAjvl4k998/edit?usp=sharing

Flink on YARN is the main deployment mode, why is there no discussion about the YARN deployment mode? @czy006

YARN will be implemented at a later date and is not discussed in this thread.

[czy006]

@czy006 @lvyanquan Compared with submitting using flink-cdc.sh, I think reuse the flink client to submit flink cdc job is more reasonable and easier to maintain, while flink client already supports all kinds of submitting way, including k8s session/application mode and yarn session/application mode, and widely used in production environments.

From another perspective, when we use Flink CDC as an application, we need to support deployment target in the script, which essentially uses flink-client to submit jobs

[PatrickRen]

Closing this issue as it has been migrated to Apache Jira.