zarr-python icon indicating copy to clipboard operation
zarr-python copied to clipboard

does zarr support big-endian

Open martindurant opened this issue 8 months ago • 2 comments

Describe the issue linked to the documentation

https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/core/metadata/v3.py#L687 seems to imply that where a (v2) data type is defined as big-endian, zarr cannot read it. Is this the case?

File ~/code/zarr/src/zarr/core/metadata/v3.py:704, in DataType.from_numpy(cls, dtype)
    686     return DataType.string
    687 dtype_to_data_type = {
    688     "|b1": "bool",
    689     "bool": "bool",
   (...)
    702     "<c16": "complex128",
    703 }
--> 704 return DataType[dtype_to_data_type[dtype.str]]

KeyError: '>i4'

(reached via virtualizarr reading a kerchunked netCDF3 file) cc @TomNicholas perhaps you've come across this kind of thing.

Suggested fix for documentation

No response

martindurant avatar Apr 22 '25 15:04 martindurant

Zarr v3 currently does not, but will do after the extension dtypes are available. Which is why we in virtualizarr have been eagerly awaiting adding these data types back in #2874.

(This is also why virtualizarr hasn't issued a release in a while - we now depend on zarr-python>=3 internally, but would have to drop support for netCDF3 if we released right now.)

TomNicholas avatar Apr 22 '25 15:04 TomNicholas

OK, thanks Tom - sorry to duplicate.

martindurant avatar Apr 22 '25 15:04 martindurant