[Feature]: Serverless process: automatic scaling for process resources

[zhoujinsong]

Description

Currently, all resources used by Amoro processes are pre-deployed. Automatic scaling supports the automatic increase or decrease of process resources based on changes in system load. This ensures sufficient resources to maintain the table performance under high loads and reduces resource usage to save costs in low-load situations.

Use case/motivation

When there are more tables in the system requiring resources, such as an excess of tables in a pending state, automatically scaling resource resources enables the accumulated tables to quickly recover health. When a resource has not received tasks for too long, release the corresponding resource resources to reduce unnecessary resource expenses.

Throughout the entire process, users only need to configure the upper and lower limits of resource usage, and the system should be able to automatically perform resource scaling.

Describe the solution

• Support configuring the upper and lower limits of resources for a specific group
• The system automatically assesses resource usage requirements and performs automatic scaling of resources.

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct