datasets
datasets copied to clipboard
largelisttype not supported (.from_polars())
Describe the bug
The following code fails because LargeListType is not supported. This is especially a problem for .from_polars since polars uses LargeListType.
Steps to reproduce the bug
import datasets
import polars as pl
df = pl.DataFrame({"list": [[]]})
datasets.Dataset.from_polars(df)
Expected behavior
Convert LargeListType to list.
Environment info
datasetsversion: 2.19.1.dev0- Platform: Linux-6.8.7-200.fc39.x86_64-x86_64-with-glibc2.38
- Python version: 3.12.2
huggingface_hubversion: 0.22.2- PyArrow version: 16.0.0
- Pandas version: 2.1.4
fsspecversion: 2024.3.1