Sync table formats metadata to multiple catalogs.

[vinishjail97]

Feature Request / Improvement

Context

Users of Apache XTable (Incubating) today can translate metadata across table formats (iceberg, hudi, and delta) and use the tables in different platforms depending on their choice. Today there's still some friction involved in terms of usability because users need to explicitly register the tables in the catalog of their choice (glue, HMS, unity, bigLake etc.) and then use the catalog in the platform of their choice to do DDL, DML queries.

XTable is built on the principle of omni directional interoperability and I'm proposing an interface which allows syncing metadata of table formats to multiple catalogs in a continuous and incremental manner.

Why do we need this feature ?

1. Reduce friction for XTable users - XTable sync will register the tables in the catalogs of their choice after metadata generation. If users are using a single format, they can still use XTable to sync the metadata across multiple catalogs.
2. Avoid catalog lock-in - There's no reason why data/metadata in storage should be registered in a single catalog, users can register the table across multiple catalogs depending on the use-case, ecosystem and features provided by the catalog.

Implementation

I have submitted a PR with the interface for CatalogSync and CatalogSyncOperations, open to the feedback on the feature request.

DISCUSS thread in [email protected] https://lists.apache.org/thread/654n5t5dvysf5bxph2cdm79s9tmz30vg

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[vinishjail97]

Submitted the PR for interfaces -> https://github.com/apache/incubator-xtable/pull/591

[purnachandergit]

This is a good feature, this will help alot

[vinishjail97]

Next steps:

1. Update docs in xtable website.
2. Add sub-tasks for other catalogs, people in the community who are interested to contribute can pick them up.