torcharrow
torcharrow copied to clipboard
Support apache-datafusion
Datafusion https://github.com/apache/arrow-datafusion is the subproject of apache arrow
I can do ta.from_arrow(pa.Table.from_batches( datafusion_dataframe.collect())
for now to
- convert datafusion's dataframe to arrow batch record
- convert arrow batch record to pyarrow table
- use ta.from_arrow to load it
wonder whether this is interesting for you to support datafusion's dataframe directly via
ta.from_datafusion( datafusion_dataframe )