[Feature]: Support managing tables with multiple formats under one Catalog

Open baiyangtx opened this issue 2 years ago • 1 comments

Currently, Table formats are strongly correlated with Catalog type, which makes the Arctic hard to use in some production situations.

We should support managing all types of table formats if the catalog type is able to create the table.

Catalog(type=hive)support TableTypes:

Hive: native hive table
Iceberg: native iceberg table which uses org.apache.iceberg.hive.HiveCatalog as implement.
Mixed-Hive: Mixed Streaming format. the base store is a Hive table.
Mixed-Iceberg: Mixed Streaming format, the base store and change store are iceberg tables created by HadoopTables.

Catalog(type=arctic) support TableTypes:

Iceberg: native iceberg table which uses specific iceberg catalog class as implement.
Mixed-Iceberg: Mixed Streaming format, the base store and change store are iceberg tables created by HadoopTables.

Introduce the concept of the InternalCatalog and ExternalCatalog

Implement the Mixed-Iceberg format above on the Iceberg format in any catalog type

Make AMS as an Iceberg Catalog provider via implement RestCatalogAPI

support mixed-hive and mixed-iceberg in same catalog(Milestone 0.5.0 later)

Support all types of table formats in the same internal catalog.

Make engines support unified catalog

No response

Feb 07 '23 02:02 baiyangtx

The design doc is here. https://docs.google.com/document/d/1Q5O2WAlHqGiWM43GbJoDvmoiejAy5DZUpXmIliYLP0k/edit#

Apr 24 '23 08:04 baiyangtx