[EPIC] Iceberg-rust Write support

[Fokko]

Iceberg-rust Write support

I've noticed a lot of interest in write support in Iceberg-rust. This issue aims to break this down into smaller pieces so they can be picked up in parallel.

Appetizers

If you're not super familiar with the codebase, feel free to pick up one of the appetizers below. They are, or are not, related to the write path, but are good things to get in, and are good to get to know the code base:

• https://github.com/apache/iceberg-rust/issues/723
• https://github.com/apache/iceberg-rust/issues/720
• https://github.com/apache/iceberg-rust/issues/726
• https://github.com/apache/iceberg-rust/issues/734

Commit path

The commit path entails writing a new metadata JSON.

• [ ] Applying updates to the metadata Updating the metadata is important both for writing a new version of the JSON in case of a non-REST catalog, but also to keep an up-to-date version in memory. It is recommended to re-use the Updates/Requirement objects provided by the REST catalog protocol. PyIceberg uses a similar approach.
  • [x] REST Catalog serialize the updates and requirements into JSON which is dispatched to the REST catalog. Done in https://github.com/apache/iceberg-rust/pull/97.
  • [ ] Other catalogs For the other catalogs, instead of dispatching the updates/requirements to the catalog. There are additional steps:
    • [x] Logic to validate the requirements against the metadata, to detect commit conflicts. A lot of this logic is already being implemented by https://github.com/apache/iceberg-rust/pull/587.
    • [ ] Writing a new version of the metadata.json to the object store. Taking into account the naming as mentioned in the spec.
    • [ ] Provide locking mechanisms within the commit (Glue, Hive, SQL, ..) so the atomic swap happens safely.
      • [ ] SQL Looks like conflict detection is missing. I was expecting logic there to see if rows are being affected (if not, another process has altered the table).
      • [ ] Glue Not yet implemented
• [ ] Commit semantics
  • [ ] MergeAppend appends new manifest list entries to existing manifest files. Reduces the amount of metadata produced, but takes some more time to commit since existing metadata has to be rewritten, and retries are also more costly: https://github.com/apache/iceberg-rust/issues/736
    • [ ] Initial defaults This makes it easier when merge-appending V1 metadata: https://github.com/apache/iceberg-rust/issues/737
  • [x] FastAppend Generates a new manifest per commit, which allows fast commits, but generates more metadata in the long run. PR by @ZENOTME in https://github.com/apache/iceberg-rust/pull/349
• [ ] Snapshot generation manipulation of data within a table is done by appending snapshots to the metadata JSON.
  • [x] APPEND Only data files were added and no files were removed. Similar to add_files.
  • [ ] REPLACE Data and delete files were added and removed without changing table data; i.e., compaction, changing the data file format, or relocating data files.
  • [ ] OVERWRITE Data and delete files were added and removed in a logical overwrite operation.
  • [ ] DELETE Data files were removed and their contents were logically deleted and/or delete files were added to delete rows.
• [ ] Add files to add existing Parquet files to a table. Issue in https://github.com/apache/iceberg-rust/issues/932
  • [ ] Name mapping in case the files don't have field-IDs set.
• [x] Summary generations Part of the snapshot that indicates what's in the snapshot: https://github.com/apache/iceberg-rust/issues/724
• [ ] Metrics collection There are two situations:
  • [x] Collect metrics when writing ~~This is done with the Java API where during writing the upper, lower bound are tracked and the number of null- and nan records are counted.~~. Most of this is in, except the NaN counts: https://github.com/apache/iceberg-rust/issues/417

Related operations

These are not on the critical path to enable writes, but are related to it:

• [x] Update table properties Sets properties on the table. Probably the best to start with since it doesn't require a complicated API.
• [ ] Schema evolution API to update the schema, and produce new metadata.
  • [ ] Having the SchemaUpdate API to evolve the schema without a user have to worry about field-IDs: https://github.com/apache/iceberg-rust/issues/697
  • [ ] Add the unionByName to easily union two schemas to provide easy schema evolution: https://github.com/apache/iceberg-rust/issues/698
• [ ] Partition spec evolution API to update the partition spec, and produce new metadata: https://github.com/apache/iceberg-rust/issues/734.
• [ ] Sort order evolution API to update the schema, and produce new metadata: https://github.com/apache/iceberg-rust/issues/732.

Metadata tables

Metadata tables are used to inspect the table. Having these tables also allows easy implementation of the maintenance procedures since you can easily list all the snapshots, and expire the ones that are older than a certain threshold.

Integration Tests

Integration tests with other engines like spark.

Contribute

If you want to contribute to the upcoming milestone, feel free to comment on this issue. If there is anything unclear or missing, feel free to reach out here as well 👍

[barronw]

Could I work on supporting table property updates? I'm also interested in working on the commit path, but it's not clear to me which tasks can be started on in parallel with the ones that are currently in progress.

[flaneur2020]

i am also interested in the write path, any suggestion about is there any issue that independent with the ongoing works and could be worked in parallel?

i'd happy to take one of these issues too.

[ZhiHanZ]

so do I! rewrite rust is the new trending, community and contributors are truly interested in write support progresses!

[Fokko]

@barronw Sure thing! I've created a separate ticket for it: https://github.com/apache/iceberg-rust/issues/730 Feel free to comment on it so I can assign it to you.

@flaneur2020 Sounds good, what do you think of the summary generation? See https://github.com/apache/iceberg-rust/issues/724 for details

[c-thiel]

@ZhiHanZ if you want an easy start, how about https://github.com/apache/iceberg-rust/issues/723? If you want something a bit more challenging, how about https://github.com/apache/iceberg-rust/issues/732?

[barronw]

@c-thiel @ZhiHanZ Can I pick up one of #723 or #732? #730 is already done.

[c-thiel]

@barronw sure! Just comment in the one you prefer, then we can assign you

[flaneur2020]

sorry came up late after a business trip, if there's still open task available, please assign to me 🙏

[c-thiel]

sorry came up late after a business trip, if there's still open task available, please assign to me 🙏

Sure :) How about https://github.com/apache/iceberg-rust/issues/734 for a start? Feel free to pick the one I mentioned or any other you like and simply post a short comment in the issue that you are working on it!

[mkarbo]

Hey everyone :) It's a bit hard to follow the progress here - would it make sense to create a github project or maybe convert this to a tracker issue with child issues?

[ZENOTME]

Hi @Fokko, CMIIW, I think for Add files to add existing Parquet files to a table, #345 is not the corresponding issue? I create a new issue #932 to track appending existing data file.

[Fokko]

@ZENOTME You're completely right, I've updated the post. Thanks for creating the new ticket

[takaebato]

Hi! I’m really interested in getting involved. If there’s an available issue, could you assign it to me?

[jonathanc-n]

@takaebato Feel free to comment under an issue that you want to work on that hasn't been touched yet, and a maintainer can assign it to you!

[takaebato]

@jonathanc-n Thanks!!

[ion-elgreco]

I'm interested in helping with the project, is there a clear roadmap of python bindings and state of current project?

Having worked on delta-rs for sometime now, I can properly apply some learning from there in iceberg rs and python

[kevinjqliu]

@ion-elgreco great to see you here! big fan of delta-rs and its python bindings :)

This issue just tracks write support for the rust project.

is there a clear roadmap of python bindings and state of current project?

Not yet! But there are a couple things we're working on. I can start an issue to track it. For context, pyiceberg is more feature rich than iceberg-rust. But we're trying to use the iceberg-rust python bindings in a few places. For example, currently we expose partition transformation functions There are plans to

• expose Catalog trait as python binding (#604)
• expose datafusion's IcebergTableProvider to python (#865)
• push down parquet reading (#1144)

Happy to chat more about other areas you might be interested in

[ion-elgreco]

@kevinjqliu happy to hear that :)

Yes, I think it would be easier to build the bindings from the ground up instead of trying to smash it into pyiceberg. One of the issues with delta-rs/deltalake was the long lasting dependency on pyarrow for the writer, and from what I see in pyiceberg it's mostly pyarrow.

I need to read a bit more about what Iceberg supports, and then I can see where to start from. But I would like to work on write support with focus to expose it in python package, maybe (iceberg-rs) this would allow you to at least have distuingishment of pyiceberg.

For the writer I think some datafusion code can be re-used from delta-rs, especially the lazy table providers for streamed writes

[DerGut]

Hi 👋 wanted to mention that I started to look into implementing the Catalog::update_table trait method for the remaining catalogs which IIUC builds the first part of the commit path mentioned above. I mostly looked at the MemoryCatalog for now, but already found some abstractions that could be reused for other catalogs as well, so I might just continue to work on them.

Please share if there are things we need to sort out first. I can also create a dedicated ticket for more in-depth discussions if preferred! :)

---

wrt common abstractions, I think this was at least partially expressed by this issue https://github.com/apache/iceberg-rust/issues/61. Could be worth revisiting a MetaStore as it is implemented in pyiceberg and even referred to in the spec.

[Xuanwo]

Hi @liurenjie1024, does this epic issue reflect the most recent status? Could you update it and add any PRs that are awaiting review? I’m happy to help this move forward.

[jonathanc-n]

@Fokko I made a list of issues + pull requests that are helping with transaction support here to help newer contributors out. Would you be able to add that list to this epic to allow people to have a list to look at or use it to help update this epic?

[liurenjie1024]

Hi @liurenjie1024, does this epic issue reflect the most recent status? Could you update it and add any PRs that are awaiting review? I’m happy to help this move forward.

It's a little outdated. Let me create a plan for write support.

[liurenjie1024]

Hi, @Fokko I feel that the scope of this issue is a little to large. Write support is a somehow large topic to implement since it involves too many things. I plan to open an epic issue with a smaller scope, which implements append data to iceberg table. This will help us to build the framework of missing parts such as transaction commit, WDYT?

[Fokko]

@liurenjie1024 Thanks, that makes sense. Let's close this one for now 👍

[liurenjie1024]

New tracking issue here: https://github.com/apache/iceberg-rust/issues/1382