narwhals icon indicating copy to clipboard operation
narwhals copied to clipboard

test: use `dask` with pyarrow dtype backend for testing

Open MarcoGorelli opened this issue 1 month ago • 1 comments

There's odd behaviour in dask when using the classical types, which is quite a pain to work around in tests. I think we should just test using their pyarrow backend reduce the amount of gymnastics we have to do in tests

MarcoGorelli avatar Nov 04 '25 13:11 MarcoGorelli

Tried the simplest change:

def dask_lazy_p2_constructor(obj: Data) -> NativeDask:  # pragma: no cover
     import dask.dataframe as dd
+    import pandas as pd
 
-    return cast("NativeDask", dd.from_dict(obj, npartitions=2))
+    pd_frame = pd.DataFrame(obj).convert_dtypes(dtype_backend="pyarrow")
+    return cast("NativeDask", dd.from_pandas(pd_frame, npartitions=2))

There is one xpass, and 44 failures 😭

FBruzzesi avatar Nov 05 '25 08:11 FBruzzesi