Alex

@lhoestq im no longer sure my specific concern about with_format(None) was well-founded - I didn't appreciate that the python formatter tries to do nothing to python objects including numpy arrays,...

@lhoestq I've updated to make compatible with latest changes on main, and think the current with_format None behaviour is probably fine - please let me know if there's anything else...

thanks for the reviews and extensions, happy to see this merged :)

possibly due to this logic: ```python def _arrow_array_to_numpy(self, pa_array: pa.Array) -> np.ndarray: if isinstance(pa_array, pa.ChunkedArray): if isinstance(pa_array.type, _ArrayXDExtensionType): # don't call to_pylist() to preserve dtype of the fixed-size array zero_copy_only...

(the download seems to complete successfully despite the error i think?)

Thanks

I was curious about the same - since map is applied on the fly I was assuming that setting num_workers>1 in DataLoader would effectively do the map in parallel, have...

perhaps concatenate_datasets can already be used to achieve almost the same effect?

I've updated the most straightforward failing test cases - lmk if you agree with those. Might need some help / pointers on the remaining new failing tests, which seem a...

@lhoestq I've had a go at fixing a few more test cases but getting quite uncertain about the remaining ones (as well as about some of the array writing ones...