pymapd
pymapd copied to clipboard
Support pyarrow RecordBatch and Table in main load_table function
trafficstars
Currently, load_table doesn't support passing a pyarrow RecordBatch. RecordBatch can be converted to a pandas dataframe using .to_pandas(), so it seems trivial to support this as well.
Related #185 : In theory, this is supposed to work with load_table_arrow, but it doesn't work with load_table(method='arrow')
data = <pyarrow.lib.RecordBatch object at 0x7f6bfa17f098>, preserve_index = False
def build_row_desc(data, preserve_index=False):
if not isinstance(data, pd.DataFrame):
# Once https://issues.apache.org/jira/browse/ARROW-1576 is complete
# we can support pa.Table here too
raise TypeError("Create table is not supported for type {}. "
"Use a pandas DataFrame, or perform the create "
> "separately".format(type(data)))
E TypeError: Create table is not supported for type <class 'pyarrow.lib.RecordBatch'>. Use a pandas DataFrame, or perform the create separately
pymapd/_pandas_loaders.py:197: TypeError