pymapd icon indicating copy to clipboard operation
pymapd copied to clipboard

Support pyarrow RecordBatch and Table in main load_table function

Open randyzwitch opened this issue 6 years ago • 1 comments
trafficstars

Currently, load_table doesn't support passing a pyarrow RecordBatch. RecordBatch can be converted to a pandas dataframe using .to_pandas(), so it seems trivial to support this as well.

Related #185 : In theory, this is supposed to work with load_table_arrow, but it doesn't work with load_table(method='arrow')

randyzwitch avatar Apr 09 '19 23:04 randyzwitch

data = <pyarrow.lib.RecordBatch object at 0x7f6bfa17f098>, preserve_index = False

    def build_row_desc(data, preserve_index=False):
    
        if not isinstance(data, pd.DataFrame):
            # Once https://issues.apache.org/jira/browse/ARROW-1576 is complete
            # we can support pa.Table here too
            raise TypeError("Create table is not supported for type {}. "
                            "Use a pandas DataFrame, or perform the create "
>                           "separately".format(type(data)))
E           TypeError: Create table is not supported for type <class 'pyarrow.lib.RecordBatch'>. Use a pandas DataFrame, or perform the create separately

pymapd/_pandas_loaders.py:197: TypeError

randyzwitch avatar Apr 09 '19 23:04 randyzwitch