[Feature]: Refactor TableRuntime and Process implementation to support optimization for more table formats

[baiyangtx]

Description

Expand AMS’s optimization capabilities for data lake tables so that they are no longer limited to Iceberg / Mixed-Iceberg.

Use case/motivation

1. Support optimization management for additional table formats, such as Paimon and Hudi.
2. Allow optimization tasks that are not executed by the built-in optimizer—e.g., Spark applications or Kyuubi jobs.
3. Provide an extensible, plug-in–based scheduling framework for optimization tasks.

Describe the solution

1. Refactor the structure of TableRuntime and TableProcess to provide higher-level abstractions that decouple them from their current implementations; AMS will manage against these abstract TableRuntime and TableProcess interfaces.
2. Using SPI to allow pluggable implementations of TableRuntime and TableProcess, ultimately enabling customized task-scheduling logic.

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

• [x] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct