paimon-rust icon indicating copy to clipboard operation
paimon-rust copied to clipboard
verified publisher icon apache

reader: Allow reading paimon data into arrow format.

Open Aitozi opened this issue 1 year ago • 3 comments

Aitozi avatar Sep 06 '24 05:09 Aitozi

Arrow format has been supported since Paimon 0.9. Do we need to support it in Paimon Rust 0.1.0?

camilesing avatar Apr 26 '25 08:04 camilesing

This ticket is meant to allow to read the data file through the arrow library

Aitozi avatar Apr 27 '25 01:04 Aitozi

I have read the relevant code (ArrowBatchReader), which uses the InternalRowSequencer class, which references some methods of the BinaryRowWriter and involves operations on the underlying memory. I plan to implement it in Rust using arena modes like bumpalo.

Do you think this is a good idea? @Aitozi

camilesing avatar Apr 30 '25 08:04 camilesing