Should PyDataFrame.collect() return a Table?

[wjones127]

trafficstars

Right now it returns List[pa.RecordBatch], but it might be more natural to return a pa.Table. For one thing, they have a better repr provided by PyArrow.

[matthewmturner]

Asides from repr, do you see any other advantages?

[houqp]

This is to keep the signature in sync with what we have in the Rust core. Perhaps it would be better to add a new method to return a pa.Table instead.

[wjones127]

Asides from repr, do you see any other advantages?

Mostly was just surprised coming from PyArrow, but it sounds like Rust usually just represents results as a sequence of record batches.

Perhaps it would be better to add a new method to return a pa.Table instead.

Yeah perhaps that's a better path. A to_table() method is common in PyArrow. If we eventually get the C Streaming data interface implemented in arrow-rs, we could also provide a to_reader().