datasets icon indicating copy to clipboard operation
datasets copied to clipboard

encode nifti correctly when uploading lazily

Open CloseChoice opened this issue 1 month ago • 0 comments

When trying to upload nifti datasets lazily I got the error:

from pathlib import Path
from datasets import load_dataset

nifti_dir = Path("<local_path>")

dataset = load_dataset(
        "niftifolder",
        data_dir=str(nifti_dir.absolute()),
        streaming=True,
)

dataset.push_to_hub(repo_id="TobiasPitters/test-nifti-papaya-testdata")
pyarrow.lib.ArrowInvalid: Could not convert <datasets.features.nifti.Nifti1ImageWrapper object at 0x77633407af90> with type Nifti1ImageWrapper: did not recognize Python value type when inferring an Arrow data type

This PR fixes that by encoding the Nifti1ImageWrappers correctly to bytes.

CloseChoice avatar Dec 01 '25 16:12 CloseChoice