sling-cli icon indicating copy to clipboard operation
sling-cli copied to clipboard

Published 20 hours ago • verified publisher icon slingdata-io

Iceberg support

Open toddy86 opened this issue 1 year ago • 4 comments

Hi,

Are there any considerations to add Apache Iceberg as a destination table format (on S3, GCS etc)?

Similar to the request that I've seen for Delta here

All the best

Todd

toddy86 avatar Dec 05 '23 16:12 toddy86

Thanks, adding some links:

  • https://github.com/apache/iceberg-go
  • https://pkg.go.dev/github.com/apache/arrow/go/[email protected]
  • https://github.com/apache/arrow/tree/main/go

flarco avatar Dec 06 '23 10:12 flarco

The easiest way may be to use DuckDB with Iceberg and executing it behind the scenes. https://duckdb.org/docs/extensions/iceberg.html (only does reads).

StarRocks does reads and writes to Iceberg.

alberttwong avatar Feb 09 '24 17:02 alberttwong

Alternatively, iceberg support can be achieved through AWS Athena support, the way dtl supports it https://dlthub.com/docs/dlt-ecosystem/destinations/athena#additional-destination-options

parsingeye avatar Mar 05 '24 13:03 parsingeye

The problem is that it's not enough to support iceberg format. You also need to register those files/data into HMS so that projects like Trino and StarRocks can be a SQL query engine to the data within those formats.

alberttwong avatar Mar 28 '24 17:03 alberttwong

Added in #357

See https://blog.slingdata.io/reading-apache-iceberg-data-with-sling

Closing.

flarco avatar Aug 28 '24 11:08 flarco