airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Destination Iceberg: add AWS Glue Catalog support

Open rnonnon opened this issue 2 years ago • 2 comments

Tell us about the problem you're trying to solve

The Iceberg connector is currently supporting Hive/Jdbc/Hadoop catalog. I'd like this connector to support the AWS Glue Catalog.

Describe the solution you’d like

Regarding the Iceberg documentation (https://iceberg.apache.org/docs/latest/aws/) and what the current connector code looks like, I guess we already include the mandatory dependencies :

implementation 'org.apache.iceberg:iceberg-spark-runtime-3.3_2.13:1.0.0'
implementation "software.amazon.awssdk:bundle:2.17.131"
implementation "software.amazon.awssdk:url-connection-client:2.17.131"

We probably should only have to add a new entry into CatalogType + adding an Implementation of IcebergCatalogConfig interface

Are you willing to submit a PR?

I'm not yet enough experienced about Airbyte and Apache Iceberg to do so. But will do it if nobody does 😄

rnonnon avatar Feb 08 '23 10:02 rnonnon

Do you have any plans to offer support for this?

GalVishi avatar Feb 19 '24 10:02 GalVishi

Would be really interested as well!

kahlua-kol avatar May 14 '24 18:05 kahlua-kol

We are also very interested, looking forward to this support!

sidharrth2002 avatar May 28 '24 03:05 sidharrth2002

Very interested in this. Looking forward to having this use case supported!

saifmahamood avatar Jul 02 '24 17:07 saifmahamood

Fixed in https://github.com/airbytehq/airbyte/pull/45707

gisripa avatar Sep 20 '24 23:09 gisripa