[Improvement]: Integrate multiple formats in a pluggable way

[baiyangtx]

Search before asking

• [X] I have searched in the issues and found no similar issues.

What would you like to be improved?

Currently, Amoro includes multiple formats(iceberg, mixed-iceberg, and paimon) during compilation.

With the integration of more formats such as hudi, implementing the integration of different formats directly in the amoro-core module and the amoro-ams-server module will make the final distribution package more bloated.

This issue hopes to make the integration of the table format in a pluggable way.

This issue hopes to make the integration of table format in a pluggable way, which will help reduce the size of the binary package and also avoid the risk of introducing unnecessary code in the production environment.

How should we improve?

The finnally modules will look like this:

• amoro-ams :
• amoro-api : Amoro core api cross different table formats
• amoro-iceberg: Iceberg related logic
• amoro-formats-integrations
  • amoro-paimon-integration : Integrate paimon to amoro
  • amoro-hudi-integration : Integrate hudi to amoro
  • amoro-iceberg-integration : Integrate iceberg and mixed-iceberg to amoro
• amoro-mixed-format
  • amoro-mixed-format-core : Core api of mixed-format cross different compute engines
  • amoro-mixed-format-spark : Connector of mixed-format for spark
  • amoro-mixed-format-flink : Connector of mixed-format for flink

Related

• #2694

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Subtasks

• #3117
• #3118

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[baiyangtx]

The full design docs

https://docs.google.com/document/d/1pUK1WpW1d1gvF5qTdFJwIseYlOxc02YkFw3QiGplNm4/edit

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions[bot]]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'