apache
Support writing to a branch
Feature Request / Improvement
Right now we hardcoded that we write to the main branch all the time, would be great to make this configurable.
The API to write to a branch should look something like
def append(self, df: pa.Table, branch: str = MAIN_BRANCH)
...
def overwrite(self, df: pa.Table, overwrite_filter: BooleanExpression = ALWAYS_TRUE, branch: str = MAIN_BRANCH)
...
But in order to write to a branch, the branch needs to be created first.
From https://iceberg.apache.org/docs/latest/spark-writes/#writing-to-branches:
the branch must exist before performing the write. The operation does not create the branch if it does not exist.
We first need a create branch API.
Then update places currently gated by MAIN_BRANCH.
- https://github.com/apache/iceberg-python/blob/9e0394939bab8d6b26cdde6f71173bdd42b55b2e/pyiceberg/table/metadata.py#L124
- https://github.com/apache/iceberg-python/blob/9e0394939bab8d6b26cdde6f71173bdd42b55b2e/pyiceberg/table/init.py#L592
And finally add a test for writing to a branch
also note dev/provision.py which is used for integration tests already have statements to create tags and branchs
https://github.com/apache/iceberg-python/blob/9e0394939bab8d6b26cdde6f71173bdd42b55b2e/dev/provision.py#L146-L148
I had an offline chat with @kevinjqliu , I shall work on this to build off of the PR created by kevin.
Hello, can I hop back on this train if no one else is actively working on this (again building off of kevin's work)? @kevinjqliu @Fokko
@Gowthami03B definitely, I've assigned the issue to you!
Opening up this to the community as I am gonna be out for the next month! @kevinjqliu
@kevinjqliu @Fokko I can take this forward if no one is actively working on this.
Hi @vinjai thank you very much for working on this issue. I'm just working through the list of open items to check if they are still actively being worked on. Are you still interested in contributing this feature to PyIceberg? 🧊
Hey @sungwy I am working on this at the moment Will open the PR for review by next week