flink-on-k8s-operator icon indicating copy to clipboard operation
flink-on-k8s-operator copied to clipboard

Published 20 hours ago • verified publisher icon GoogleCloudPlatform

Will operator be updated to support Native Kubernetes Integration of Flink?

Open chethanuk opened this issue 5 years ago • 15 comments

Will the operator be updated to support Native Kubernetes Integration of Flink?

Check out more details on #FLINK-9953 or https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/native_kubernetes.html#background--internals

chethanuk avatar Feb 12 '20 19:02 chethanuk

I think this is a very importent issue.

Mrart avatar Feb 18 '20 04:02 Mrart

@fu @hongyegong ，I think we can have a discuss about 'supported flink 1.10 is needed ?' and 'how to implement this function',also have some reconstruct code idea.

Mrart avatar Feb 19 '20 02:02 Mrart

We definitely want to support native k8s integration, but currently we don't have enough resource to put into this feature, so there is no timeline for that. We welcome contributions from the community, thanks!

functicons avatar Feb 20 '20 18:02 functicons

@functicons we can discuss some detail things about this。 Or can you give some design？

  1. flink on k8s native need create component by fiink resourcemanagement. So how to recocile the component？
  2. some design about the crds need change。 and ....

Mrart avatar Feb 21 '20 09:02 Mrart

I want to add this feature, but I want some detail design, Can you help me ? @functicons

Mrart avatar Mar 04 '20 06:03 Mrart

Sure, I'll spend some time on this feature maybe next week.

functicons avatar Mar 04 '20 15:03 functicons

OK,Thank you. I will spend more time for the community.

Mrart avatar Mar 05 '20 02:03 Mrart

@functicons I think it is a hard feature, Any progress？

Mrart avatar Apr 01 '20 03:04 Mrart

The new model is more complicated that what it is now. The major problem is that it uses a dedicated cli to launch the cluster instead of applying a set of YAML files with kubectl. I think eventually we need to bypass the cli and directly generating YAML files.

In addition, it is currently in beta, not final yet, I want to wait until it is stable before proposing our design.

functicons avatar Apr 02 '20 23:04 functicons

I have running in our product env. and I desired the feature , because my consumer want it. Ok , we can start the discuss, I think we can add a meshine to convert the cli to yaml, or convert the yaml to cli in our operator. now ,I think convert yaml to cli is more rationalization. If we do this, We can reach the result nearly, but if we use flink 1.10, We must add some feature we have now to put it to product, because flink 1.10 not support so many feature.

Mrart avatar Apr 03 '20 09:04 Mrart

I am from the Flink community and the main maintainer of native Kubernetes integration of Flink. Maybe I could provide more information about the roadmap of native K8s integration.

So far we keep developing the native K8s integration for three major release cycle in Flink.

  • Flink 1.10
    • Beta version
    • Session mode only
  • Flink 1.11
  • Flink 1.12(Will release in December 2020 soon)
    • Security improvement(kerberos, K8s secret)
    • PyFlink support
    • K8s HA

After reviewing the current supported features in flink-on-k8s-operator, I find that we still have some few works to do in the following Flink release cycle. Maybe they could be archived via introducing a common pod template support or some dedicated config options. Before that using a Kubernetes Mutating Admission Webhook could help customization of Flink pods.

  • Init containers and sidecar containers
  • Arbitrary volume(PVC, hostpath, emptyDir, ConfigMap) mount

All in all, I believe the native K8s integration of Flink now is almost production ready. And I am glad to help to get it supported in this flink-on-k8s-operator.

wangyang0918 avatar Dec 07 '20 06:12 wangyang0918

  • Init containers and sidecar containers

Support for Python Beam is planned with Native K8s? and Wat about autoscaling with custom Metrics or just normal autoscaling for start?

chethanuk avatar Dec 11 '20 17:12 chethanuk

Hi @wangyang0918,

I want to say huge thank you for creating native Kubernetes integration in Flink.

Looks like https://github.com/spotify/flink-on-k8s-operator is successor of current operator. Do you plan to help this new project with K8s native deployment?

serjN avatar Jan 12 '22 13:01 serjN

@wangyang0918 I believe lot's of flink-operator user have been expecting native Kubernetes integration in operator. I logged the issue #438 about half year ago. I also want to know the plan for this :).

william-wang avatar Jan 12 '22 14:01 william-wang

@serjN @william-wang FYI, Flink community is now having an active discussion about developing a new Flink K8s operator under the Apache Flink umbrella.

https://lists.apache.org/thread/l1dkp8v4bhlcyb4tdts99g7w4wdglfy4

wangyang0918 avatar Jan 15 '22 10:01 wangyang0918