datasets icon indicating copy to clipboard operation
datasets copied to clipboard

largelisttype not supported (.from_polars())

Open Modexus opened this issue 1 year ago • 0 comments

Describe the bug

The following code fails because LargeListType is not supported. This is especially a problem for .from_polars since polars uses LargeListType.

Steps to reproduce the bug

import datasets
import polars as pl

df = pl.DataFrame({"list": [[]]})
datasets.Dataset.from_polars(df)

Expected behavior

Convert LargeListType to list.

Environment info

  • datasets version: 2.19.1.dev0
  • Platform: Linux-6.8.7-200.fc39.x86_64-x86_64-with-glibc2.38
  • Python version: 3.12.2
  • huggingface_hub version: 0.22.2
  • PyArrow version: 16.0.0
  • Pandas version: 2.1.4
  • fsspec version: 2024.3.1

Modexus avatar Apr 24 '24 11:04 Modexus