flinkk8soperator icon indicating copy to clipboard operation
flinkk8soperator copied to clipboard

Published 20 hours ago • verified publisher icon lyft

Enable Reactive Scaling by supporting Application mode along with Session Cluster

Open JayGhiya opened this issue 2 years ago • 8 comments

To manage cloud costs effectively for customers, it is important to have reactive scaling across all sort of services residing in kubernetes. We have been active users of lyft's flink operator from the start and are extremely grateful for the operator's current features.

To enable reactive scaling in task managers from 0...n: a) The operator has to support application mode cluster along with session mode. b) The keda scaled object file has to be deployed by the operator and the trigger has to be externalized in the crd.

i and some members would love to contribute to this feature and some other pressing issues as well.

@anandswaminathan please let me know how to take this forward in terms of contribution .

JayGhiya avatar Mar 14 '22 07:03 JayGhiya

thank you @JayGhiya for opening this req.

vipinshreyaskumar avatar Mar 14 '22 09:03 vipinshreyaskumar

I believe we looked into this. There were several simpler options. Will need to dig a bit.

anandswaminathan avatar Mar 14 '22 17:03 anandswaminathan

Okay. Can you please share those options? As flink operator creates a session cluster and apache flink does not support Reactive scaling on session cluster as of now.

On Mon, 14 Mar, 2022, 11:02 pm Anand Swaminathan, @.***> wrote:

I believe we looked into this. There were several simpler options. Will need to dig a bit.

— Reply to this email directly, view it on GitHub https://github.com/lyft/flinkk8soperator/issues/249#issuecomment-1067098316, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF5IO3LYB2ZM4UFWCMR7ZSDU75ZZVANCNFSM5QUWO7ZA . You are receiving this because you were mentioned.Message ID: @.***>

JayGhiya avatar Mar 14 '22 17:03 JayGhiya

@JayGhiya Can you share or add relevant links as well? Let me take a look

anandswaminathan avatar Mar 14 '22 19:03 anandswaminathan

We are actually thinking of the same. Happy to collaborate on !

maghamravi avatar Mar 15 '22 02:03 maghamravi

@anandswaminathan sure Here is the link for reactive scaling on flink through hpa -> Flink Reactive Scaling. Here is the link to the poc that was done with keda. Flink Keda Poc. Keda Reference

JayGhiya avatar Mar 15 '22 05:03 JayGhiya

Hello @JayGhiya Can you share some feed back about your Flink Keda Poc, I am searching for this solution also. And I am in flink operator deployment, hope to find a ways to coexists flink operator autoscaler with keda 🙏

Yang-LI-CS avatar Oct 03 '23 09:10 Yang-LI-CS

sure @Yang-LI-CS . Here is the repo where I had run a experiment with keda using flink's official Kubernetes operator (https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/) ->https://github.com/JayGhiya/Flink-AutoScale-Poc . The experience is consistent and stable. what exactly are you looking for in terms of type of scalers as it depends on sources of Flink pipeline

JayGhiya avatar Oct 03 '23 17:10 JayGhiya