[Feature]: Refactor AMS using function and process api

[majin1102]

Description

Three main goals for this feature:

1. Implementing a pluggable and extensible processing mechanism for lake table formats based on action and process api, after which users could define external or custom optimizing, syncing, expiring processes managed by AMS.
2. Optimize scheduling logic based on quota, which could better control cost and scheduling fairness for optimizing and other processes.
3. Migrate all task running logic from AMS to optimizers, including optimizing plan, commit and maintenance tasks, which will improve AMS stability greatly.
4. Use future model inside of process lifecycle management instead of table status watcher model for better readability and extensibility.

This top issue will be splitted into serveral sub issues.

Use case/motivation

Use action and process model for AMS and improve steability, readibility and extensibility

Describe the solution

No response

Subtasks

• [ ] #2597
• [ ] #3237
• [ ] #3258
• [ ] #3265

No response

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions[bot]]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'