Ability to pickle the `Catalog`

[Fokko]

Feature Request / Improvement

This allows distribution of the Catalog object within Ray.

[kevinjqliu]

Copied from the other thread: https://github.com/apache/iceberg-python/issues/513#issuecomment-2009875953

I was looking at Pydantic docs and it looks like there's a helpful function which supports pickling and unpickling https://docs.pydantic.dev/latest/concepts/serialization/#pickledumpsmodel

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[kevinjqliu]

@dev-goyal I see that you've merged ray-project/ray#46889 curious if you think pickling the Catalog or Table is still valuable to the Ray project

[kevinjqliu]

More context: https://github.com/ray-project/ray/pull/42235#discussion_r1520929199

[dev-goyal]

@dev-goyal I see that you've merged ray-project/ray#46889 curious if you think pickling the Catalog or Table is still valuable to the Ray project

I think it could help, but for me - I did not find recreating the catalog each time to add much overhead. So, it will likely just make the code cleaner is all

[kevinjqliu]

So, it will likely just make the code cleaner is all

Makes sense, the difference is directly pickling the catalog object so that it can be distributed to ray workers versus the current implementation which pass the catalog_kwargs to reconstruct the catalog at the worker node