datasets
datasets copied to clipboard
encode nifti correctly when uploading lazily
When trying to upload nifti datasets lazily I got the error:
from pathlib import Path
from datasets import load_dataset
nifti_dir = Path("<local_path>")
dataset = load_dataset(
"niftifolder",
data_dir=str(nifti_dir.absolute()),
streaming=True,
)
dataset.push_to_hub(repo_id="TobiasPitters/test-nifti-papaya-testdata")
pyarrow.lib.ArrowInvalid: Could not convert <datasets.features.nifti.Nifti1ImageWrapper object at 0x77633407af90> with type Nifti1ImageWrapper: did not recognize Python value type when inferring an Arrow data type
This PR fixes that by encoding the Nifti1ImageWrappers correctly to bytes.