torcharrow icon indicating copy to clipboard operation
torcharrow copied to clipboard

Support apache-datafusion

Open waitingkuo opened this issue 2 years ago • 0 comments

Datafusion https://github.com/apache/arrow-datafusion is the subproject of apache arrow

I can do ta.from_arrow(pa.Table.from_batches( datafusion_dataframe.collect()) for now to

  1. convert datafusion's dataframe to arrow batch record
  2. convert arrow batch record to pyarrow table
  3. use ta.from_arrow to load it

wonder whether this is interesting for you to support datafusion's dataframe directly via

ta.from_datafusion( datafusion_dataframe )

waitingkuo avatar Jul 01 '22 07:07 waitingkuo