[CT-1492] [Feature] Add support for Iceberg Tables

[teej]

Is this your first time submitting a feature request?

• [X] I have read the expectations for open source contributors
• [X] I have searched the existing issues, and I could not find an existing issue for this feature
• [X] I am requesting a straightforward extension of existing dbt-snowflake functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Snowflake has announced support for Iceberg Tables, an external table format but with full interoperability as a 1st class snowflake table.

Once this feature goes live, dbt-snowflake should support materializing a model as an external iceberg table.

Describe alternatives you've considered

No response

Who will this benefit?

• dbt users who want their models available as parquet files in storage they control
• dbt users who want an easy way to interoperate with non-dbt systems that use the iceberg table format
• dbt users who want direct access and control over the underlying storage of their data

Are you interested in contributing this feature?

yes

Anything else?

Iceberg Tables are in private preview as of 2022-11-10

[erniedeferia]

+1

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[jaklan]

Any chance to put more light on that? We really need a Snowflake equivalent of https://github.com/dbt-athena/dbt-athena to push forward the adoption of open table format (which, I believe, would be also pretty crucial for dbt to support in the long term).

[multiprocessingh]

+1 here! This would be super useful to support for enterprise sas use cases!

[WangCHEN9]

+1 here!

[DLewisEndeavor]

+1 here! Looking into Iceberg for enterprise sharing across platforms

[apd-bscally]

+1, Iceberg is now in public preview on Snowflake

[mourtallah]

+1 here!

[VSchmidts]

+1 here!

[ksdaly]

+1

[ZergRocks]

+1

[sfc-gh-ripu]

+1

[jimvekemans]

+1, will work on this a bit in the coming weeks and, if successful, open a pull request

[dennysrega1ado]

+1

[waitsfornone]

+1 here. We are adopting Iceberg tables as our data lake format and I would love to generate the tables out of Snowflake with dbt.

[mikezusev]

+1

[zewang-sanofi]

+1

[sfc-gh-wriley]

+1 :-)

[njrogers]

+1

[Fleid]

Hey all! Thanks for "upvoting", please continue to do so. We are wrapping up all the internals work to support 1.8, plus a few more things to do for quality of the plugins and how we test them, and then Iceberg will be at the top of the list!

[sfc-gh-ftrejo]

+1

[sfc-gh-elethier]

+1

[sfc-gh-amdeshpande]

+1

[sfc-gh-mpowell]

+1

[joatkh]

+1

[Jemsley0]

+1

[ncapeta]

Any ETA on 1.8 ?

[jonask137]

Any ETA on 1.8 ?

According to the documentation, it should be released as of April 2024: https://docs.getdbt.com/docs/dbt-versions/core#latest-releases

[shayms8]

i'm getting errors when trying to query a MAP iceberg object from dbt:

  002040 (42601): SQL compilation error:
  Unsupported data type 'map(VARCHAR(16777216), VARCHAR(16777216))'.
  compiled Code at target/run/dbt_snowflake_1/models/L1_iceberg/L1__client_events_iceberg.sql

any idea why it happens? can i somehow solve it? didn't find a documentation suggesting MAP columns are not supported on dbt_snowflake integration

[ShahNewazKhan]

+1