future of zarr.numcodecs

[d-v-b]

Now that we moved the numcodecs.zarr3 codecs to their own module in this repo, we should decide how we want those codecs to evolve in the future. I think there are a few issues with those codecs that need to be addressed before we can widely recommend them to users:

codec identifiers

The codecs defined in numcodecs.zarr3 used identifiers like "numcodecs.gzip", "numcodecs.blosc", etc but this "numcodecs"` prefix is of questionable value now that these codecs are defined in zarr python instead of numcodecs. In fact I think it's likely to confuse people if there are two separate codecs named "gzip" and "numcodecs.gzip" (which are identical except the name).

For codecs like gzip, my recommendation is that we accept "gzip" and "numcodecs.gzip" as valid identifiers when reading, but when writing we only emit "gzip".

If necessary we could define a LegacyGzip codec that emits metadata with the "numcodecs.gzip" identifier, for users who need to write data that will be consumed by older versions of zarr python.

nearly-compatible codecs

The BloscCodec defined in zarr.codecs.blosc uses different parameters from zarr.codecs.numcodecs.Blosc -- the types of the shuffle parameter are different, and zarr.codecs.blosc.BloscCodec takes a typesize parameter which zarr.codecs.numcodecs.Blosc does not take.

This is very unfortunate, and I think we should take steps to prevent anyone from using zarr.codecs.numcodecs.Blosc. We should also widen the accepted input types of zarr.codecs.blosc.BloscCodec to accept the parameters used for zarr.codecs.numcodecs.Blosc.

completely wrong codecs

Some codecs have parameters that describe array data types, like astype and delta. For Zarr V3, these codecs should use Zarr V3 data type identifiers. Instead, the codecs defined in numcodecs.zarr3 use Zarr V2 data type identifiers, which is very incorrect. We need to fix this for the zarr.codecs.numcodecs codecs by using Zarr V3 data type identifiers, but this would break forward compatibility -- older Zarr Python versions would not be able to open these newer versions of the astype or delta codecs. I think we have to accept this breakage, because it makes no sense to use Zarr V2 data type identifiers in codecs designed for Zarr V3.

Curious to hear other folks' thoughts on this. I don't see how we can effect some of these necessary changes without forwards compatibility breakage, but I also think the changes outlined here are necessary.