airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Add Iceberg provider

Open romsharon98 opened this issue 10 months ago • 5 comments

Following https://github.com/apache/airflow/pull/32786#issuecomment-1768284013 since Tabular provider doesn't have any specific tabular logic it has been suggested to move Tabular logic to new Apache Iceberg provider


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

romsharon98 avatar Apr 21 '24 13:04 romsharon98

@Fokko Do we want to keep Tabular provider (if there might be future tabular specific behavior?) or should we deprecate the whole tabular provider in favor of apache-iceberg provider? (difference between deprecate specific classes vs the whole provider)

eladkal avatar Apr 21 '24 14:04 eladkal

@Fokko Do we want to keep Tabular provider (if there might be future tabular specific behavior?) or should we deprecate the whole tabular provider in favor of apache-iceberg provider? (difference between deprecate specific classes vs the whole provider)

No, let's deprecate it in favor of the Iceberg provider. Everything in here is just Iceberg, and nothing Tabular specific, so the current naming is confusion

Fokko avatar Apr 29 '24 16:04 Fokko

@romsharon98 Sorry for being late to the party here, but this looks great! Thanks for picking this up 🎉

Fokko avatar Apr 29 '24 16:04 Fokko

@eladkal That makes sense. It was not my intent to merge this without your approval, but just to indicate that there are no further comments on the Operator and Hook itself.

Fokko avatar May 01 '24 07:05 Fokko

@eladkal That makes sense. It was not my intent to merge this without your approval

I know :) This is just to avoid merge by another committer. We are 99% ready here. it's just for the procedure of release due to https://github.com/apache/airflow/pull/39240 for simplicity I will handle it as ad-hoc release

eladkal avatar May 01 '24 09:05 eladkal

🎉🎉🎉

eladkal avatar May 10 '24 05:05 eladkal

Wooohooo!

potiuk avatar May 10 '24 05:05 potiuk

Nice! Thanks @romsharon98 for working on this, and thanks @eladkal and @potiuk for the review 🎉

Fokko avatar May 10 '24 05:05 Fokko