Feature request: Allow pagination on list operations

[dentiny]

Is your feature request related to a problem or challenge?

It's possible to have large number of namespace and tables, with query taking long time listing these large entities. I'm wondering if it's possible and good to provide pagination functionality to list operations.

Interface proposal:

/// Pagination parameters for listing namespaces
#[derive(Debug, Clone)]
pub struct PaginationOptions {
    /// Maximum number of items to return
    pub max_results: usize,
    /// Token for continuing a previous listing
    pub page_token: Option<String>,
}

/// Result of a paginated namespace listing
#[derive(Debug)]
pub struct PaginatedList<T> {
    /// The actual items returned
    pub items: Vec<T>,
    /// Token for retrieving the next page of results
    pub next_page_token: Option<String>,
}

/// List namespaces with pagination support
async fn list_namespaces_paginated(
    &self,
    parent: Option<&NamespaceIdent>,
    pagination: PaginationOptions,
) -> IcebergResult<PaginatedList<NamespaceIdent>>;

/// List tables with pagination support
async fn list_tables_paginated(
    &self,
    namespace: &NamespaceIdent,
    pagination: PaginationOptions,
) -> IcebergResult<PaginatedList<TableIdent>>;

Describe the solution you'd like

No response

Willingness to contribute

I can contribute to this feature independently

[liurenjie1024]

Thanks @dentiny , this looks good to me. Rest catalog spec has something similar, so I think it's reasonable to add this functionality to RestCatalog implementation.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions[bot]]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'