iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard

Plan file scan task according scan file size.

Open liurenjie1024 opened this issue 2 years ago • 4 comments

liurenjie1024 avatar Dec 19 '23 07:12 liurenjie1024

Can I work on this ? Additionally is the expectation here is to add multiple FileScanTask based on the split size and the scan file size

rahull-p avatar Aug 10 '24 09:08 rahull-p

Can I work on this ? Additionally is the expectation here is to add multiple FileScanTask based on the split size and the scan file size

Hi, @rahull-p Thanks for your interest, and welcome to contribute! In the basic version, we need to take the split size into account. In advanced version, we also need to take the file open cost into account. Java's reference implementation is here: https://github.com/apache/iceberg/blob/ab6391767e75357a9a12691513d9f83a4180f7e4/core/src/main/java/org/apache/iceberg/BaseTableScan.java#L43

liurenjie1024 avatar Aug 13 '24 14:08 liurenjie1024