polars icon indicating copy to clipboard operation
polars copied to clipboard

0.15.16 made pyarrow a requirement when using pl.from_pandas

Open rigogsilva opened this issue 2 years ago • 1 comments

Polars version checks

  • [X] I have checked that this issue has not already been reported.

  • [X] I have confirmed this bug exists on the latest version of Polars.

Issue description

It seems like pyarrow is required now when using pl.from_pandas. 2 days ago this was not true. I believe it is due to this change, but I am not 100% sure:

  • default to pyarrow for writing parquet (#6313)

I could add pyarrow to my dependencies but I am not sure if you intended to make it a requirement as I see a lot of python code to check if pyarrow exists.

Reproducible example

python
import polars as pl
import pandas as pd

polars_df = pl.from_pandas(pd.DataFrame([{"foo": "bar"}]))

print(polars_df)

Expected behavior

Use default writer and not pyarrow.

Installed versions

0.15.16

rigogsilva avatar Jan 24 '23 00:01 rigogsilva

For convertion to and from pandas pyarrow was already a requirement.

This is the code we run:

    """Construct a PyDataFrame from a pandas DataFrame."""
    length = data.shape[0]
    arrow_dict = {
        str(col): _pandas_series_to_arrow(
            data[col], nan_to_none=nan_to_none, min_len=length
        )
        for col in data.columns
    }
    arrow_table = pa.table(arrow_dict)
    return arrow_to_pydf(
        arrow_table, schema=schema, schema_overrides=schema_overrides, rechunk=rechunk
    )

ritchie46 avatar Jan 24 '23 07:01 ritchie46

I will close this as this has always been a requirement.

ritchie46 avatar Jan 25 '23 09:01 ritchie46

Makes sense. Thanks for taking a look at this @ritchie46 !

rigogsilva avatar Jan 25 '23 23:01 rigogsilva