Iceberg Import and Export

[paulcichonski]

Apache Iceberg is an Open Table Format (OTF) that can be used for storing data in object storage with a well defined schema.

The Iceberg spec defines its own data schema representation (see Table Spec). Tha spec also defines a json serialization of the schema that can be used for schema management in the iceberg tables. Using the Iceberg json schema definition it is possible to call Iceberg Catalog APIs (popular implementations include Glue, Hive, REST, Dynamo, Sql) to create tables or to evolve the schema on existing tables.

This issue is to request that the datacontract tool support an Iceberg schema export that adheres to the above json encoding. This way, models defined in a datacontract yml could be used for managing Iceberg tables. For example, the cli command could look like:

datacontract export --model orders --format iceberg https://datacontract.com/examples/orders-latest/datacontract.yaml > orders.json

Ideally it could support exporting all models at once, but the current Iceberg json definition seems to support one model per json document, so the above could be a good start.