apache
[DSIP-][Flink task plugin] Supports submitting flink tasks to k8s
Search before asking
- [X] I had searched in the DSIP and found no similar DSIP.
Motivation
There are more and more ways to submit flink tasks to run on the k8s, so I recommend that the ds support this feature
Design Detail
No response
Compatibility, Deprecation, and Migration Plan
Compatible with the following versions :dolphinshceduler3.2.2,flink1.17,kubernetes 1.20
Test Plan
increase unit-test coverage
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Ok, I see.I expect to have a more detailed design by the end of the week 好的，我明知道了。 我预计在这周末会有一份更详细的设计
Here's a proven idea, subject to change, and the final design I'll present this weekend: Add parameter checksum support to the flink task type k8 by modifying the init and getScript methods of the current flink task type to complete ds scheduling flink commit tasks to k8. 以下是一个经过验证的想法，可能会有变动，最终设计我会在这周末给出：通过修改当前flink任务类型的init和getScript方法，为flink任务类型k8添加参数校验和支持，从而完成ds调度flink提交到k8的任务。
I hope you will let me know if there are any inadequacies in this approach or if it does not conform to community norms, and I will work hard to achieve this goal 如果这个方式有不足之处或者不符合社区规范希望您告知我，我将为完成这个目标而努力
Design detail:
The current FlinkTask has shell and flink api submissions, and I plan to change shell type FlinkTask to support k8s
Execution process analysis:
FlinkTask inherits the AbstractYarnTask and implements its own getScript method. AbstractYarnTask invokes FlinkTask's initSctipt method to generate a script, which is executed by ShellCommandExecutor
当前的FlinkTask 有shell和flink api两种提交方式，我预计改动shell类型的FlinkTask，以使得其支持k8s 执行流程分析： FlinkTask继承了AbstractYarnTask，并实现了自己的getScript方法。AbstractYarnTask会调用FlinkTask的initSctipt方法生成脚本，该脚本会由ShellCommandExecutor执行。
Maybe you have some deviations in your understanding of design details. The design details you provided are the implementation process of current code. What we need is what you will do.
Compatible with the following versions :dolphinshceduler3.2.2,flink1.17,kubernetes 1.20
May I ask how to be compatible with different versions of flink and k8s. And In what ways to support different deployment mode of flink on k8s in ds's backend and frontend.
Hello, I looked up a lot of information and code, here is how to submit flink tasks to k8s cluster on ds and how to handle compatibility replies. Compatibility processing, through different command parameters to support different flink versions. I'm currently planning to support flink session clusters on kubernetes in this release.
As for the k8s front and back end you talked about, I am not particularly clear about what is the k8s front end, is it by exposing the k8s objects applied in the k8s cluster, such as NodePort, Ngpress or other, I hope you can inform me We all know that external applications cannot access k8s internal applications directly, and the same is true of dolphinscheduler in non-K8S deployment environments. If we have a dolphinscheduler deployed on a host outside of k8s, to schedule flink applications inside k8s on that dolphinscheduler, we need to expose the cluster inside k8s, either by opening a port per machine, or through something like an nginx proxy This exposed location allows dolphinscheduler to access the flink cluster inside the k8s and deploy applications. That's what I think, too. I refer to this flink article: https://nightlies.apache.org/flink/flink-docs-release-1.20/zh/docs/deployment/resource-providers/standalone/kubernetes/
If the community has any questions and wants to let me know, I'd appreciate it.
您好，我查阅了很多资料和代码，这是怎么让在ds上提交flink任务到k8s集群和怎么处理兼容性的回复。 兼容性处理，通过不同的命令参数支持不同flink版本。目前我打算在这个版本上支持kubernetes上的flink session集群。
对于您说说的k8s前后端，我不是特别清楚什么是k8s前端，是通过暴露k8s集群内应用的k8s对象吗，如NodePort，Ngpress还是其他的，希望您能告知我 我们都知道外部应用无法直接访问k8s内部应用，同理dolphinscheduler在非k8s部署环境下也是如此。 如果我们在一个k8s外的主机部署了dolphinscheduler，要在这个dolphinscheduler上调度k8s内的flink应用，我们需要把k8s内部的集群暴露出来，或是在每个机器开放一个端口，或是通过类似nginx代理的方式 dolphinscheduler可以通过这个暴露出来的地方去访问k8s内部的flink集群，并部署应用。 这也是我的想法。我参考了flink的这篇文章：https://nightlies.apache.org/flink/flink-docs-release-1.20/zh/docs/deployment/resource-providers/standalone/kubernetes/
如果社区有什么疑问希望能告知我，不胜感激。