[feat] Unify implementation of `to_arrow` and `to_arrow_batch_reader`

[kevinjqliu]

Feature Request / Improvement

Currently, these two functions are performing similar tasks; the only difference is the output format.

Let's unify the implementations so they don't diverge. For example, to_arrow_batch_reader has limit pushed down to the file level, but to_arrow does not.

For more details, see https://github.com/apache/iceberg-python/issues/1032#issuecomment-2282819711

[sungwy]

Thank you for raising this @kevinjqliu . I think this will be a good improvement to reduce the duplication of code.

Have you started working on this already? If not, would it be alright to get this assigned to myself? 🙂

[kevinjqliu]

I have not! Please feel free to take a look.

[sungwy]

closed through: https://github.com/apache/iceberg-python/pull/1043