Feature request: list namespaces and tables in stream style

[dentiny]

Is your feature request related to a problem or challenge?

Hi team, I plan to use list_namespaces and list_tables on recovery.

For example, when a server goes back, created catalog calls list operations to

• Get what tables to recover;
• For each table, recover states before crash by walking through version hint, metadata, etc to data files.

One feature I would like to have is to stream-style list operation, so recovery process (which is also async via tokio) could follow through.

Current interface only supports to get all entities in one time. https://github.com/apache/iceberg-rust/blob/b4bc6dd15ae4f7a4b1c9ce555e3458089a83e228/crates/iceberg/src/catalog/mod.rs#L42-L44

https://github.com/apache/iceberg-rust/blob/b4bc6dd15ae4f7a4b1c9ce555e3458089a83e228/crates/iceberg/src/catalog/mod.rs#L73-L74

Related issue:

• Support pagination for list operations: https://github.com/apache/iceberg-rust/issues/1251
  • I would like to spend some time on it, which should unblock my work

Describe the solution you'd like

Opendal supports stream-style list, if I were to implement the feature, I would very likely mimic the same interface: https://github.com/apache/opendal/blob/faf12983a0f46a5c75641d1a0b0eaa5d0600b972/core/src/types/operator/operator.rs#L1337-L1371

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[Xuanwo]

I think that's a great idea. We can add APIs like list_namespaces_stream and refactor list_namespaces to use list_namespaces_stream internally.