iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard
verified publisher icon apache

Implement limit push down for `IcebergTableProvider `

Open krinart opened this issue 3 months ago • 2 comments

Which issue does this PR close?

N/A

What changes are included in this PR?

Previously _limit was ignored in IcebergTableProvider::scan:

https://github.com/apache/iceberg-rust/blob/aad9e2e3439aeea786741c0af46f8efffba4f747/crates/integrations/datafusion/src/table/mod.rs#L149-L163

This PR propagates limit all the way to the ArrowReaderBuilder.

Note: limit push down is only applied to each batch which means that IcebergTableProvider::scan may potentially return more records than specified by limit.

Which is OK according to TableProvider::scan documentation:

If limit is specified, must only produce at least this many rows, (though it may return more).

Are these changes tested?

Unit tests

krinart avatar Sep 15 '25 17:09 krinart

Happy to help @ZENOTME and thanks for the feedback! Any suggestions on how to proceed and merge this PR?

krinart avatar Sep 19 '25 14:09 krinart

Hi. Anything else I can do on my side to get this merged?

krinart avatar Oct 12 '25 19:10 krinart

hi, @krinart please allow me to edit this PR so we can resolve the conflicts and merge it.

Xuanwo avatar Dec 05 '25 10:12 Xuanwo