amoro icon indicating copy to clipboard operation
amoro copied to clipboard

Published 20 hours ago • verified publisher icon apache

[Feature]: Support Flink optmizer run in session cluster

Open huyuanfeng2018 opened this issue 11 months ago • 4 comments

Description

Currently Amoro already supports running tasks in application-mode and yarn-per-job. Both methods can run optmizer very well, but I would like to point out that maybe running tasks in session mode is also a good choice.

Use case/motivation

  1. Faster startup: session cluster is a pre-started Flink Cluster, which can be k8s, yarn or Standalone. At this time, we save the time required to deploy the cluster and start it directly in the session cluster.

  2. Lower resource consumption: amoro's flink optmizer is essentially not a flink task in the traditional sense. It is a distributed service packaged by flink. In fact, it does not consume much resources on the jobmanager. If it is each flink optmizer Both require a cluster, so each job needs to start a jm, but in fact one jobmanager running multiple jobmasters is completely sufficient for flink optmizer, so it can save jm resources.

  3. Faster demo: For example, like other frameworks, if the user wants to simply run amoro's flink optmizer, he can directly and quickly deploy a Flink Standalone cluster without configuring complex environments such as yarn k8s, and checkpoint storage media such as hdfs are also no required. No need (because essentially flink optmizer does not have checkpoint)

  4. Automatic expansion and contraction? I think session mode may be used in amoro's optmizer to expand the capacity better. When expansion is needed, call the API to add a Taskmanager and submit the task. The same goes for shrinking. (Here you can use flink’s adapter scheduling mode？Perhaps further discussion is needed)

so, I think it is a nice feature to support running flink optmizer in session mode

Describe the solution

The user configures an existing flink session cluster. Amoro can submit tasks through flink rest client, just like flink k8s operator.

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

  • [x] Yes I am willing to submit a PR!

Code of Conduct

huyuanfeng2018 avatar Mar 19 '24 12:03 huyuanfeng2018