zarr-python Error when using attribute keys with mixed types

trafficstars

Minimal, reproducible code sample, a copy-pastable example if possible

import zarr
z1 = zarr.open('data/example.zarr', mode='w', shape=(10, 10), dtype='i4')
z1.attrs.put({1: "bar"})  # OK
z1.attrs.put({"foo": "bar"})  # OK
z1.attrs.put({1: "bar", "foo": "bar"})  # Error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [14], in <cell line: 1>()
----> 1 z1.attrs.put({1: "bar", "foo": "bar"})

File ~/mambaforge/envs/xarray/lib/python3.9/site-packages/zarr/attrs.py:109, in Attributes.put(self, d)
    106 def put(self, d):
    107     """Overwrite all attributes with the key/value pairs in the provided dictionary
    108     `d` in a single operation."""
--> 109     self._write_op(self._put_nosync, d)

File ~/mambaforge/envs/xarray/lib/python3.9/site-packages/zarr/attrs.py:73, in Attributes._write_op(self, f, *args, **kwargs)
     71 # synchronization
     72 if self.synchronizer is None:
---> 73     return f(*args, **kwargs)
     74 else:
     75     with self.synchronizer[self.key]:

File ~/mambaforge/envs/xarray/lib/python3.9/site-packages/zarr/attrs.py:112, in Attributes._put_nosync(self, d)
    111 def _put_nosync(self, d):
--> 112     self.store[self.key] = json_dumps(d)
    113     if self.cache:
    114         self._cached_asdict = d

File ~/mambaforge/envs/xarray/lib/python3.9/site-packages/zarr/util.py:38, in json_dumps(o)
     36 def json_dumps(o: Any) -> bytes:
     37     """Write JSON in a consistent, human-readable way."""
---> 38     return json.dumps(o, indent=4, sort_keys=True, ensure_ascii=True,
     39                       separators=(',', ': ')).encode('ascii')

File ~/mambaforge/envs/xarray/lib/python3.9/json/__init__.py:234, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    232 if cls is None:
    233     cls = JSONEncoder
--> 234 return cls(
    235     skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    236     check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    237     separators=separators, default=default, sort_keys=sort_keys,
    238     **kw).encode(obj)

File ~/mambaforge/envs/xarray/lib/python3.9/json/encoder.py:201, in JSONEncoder.encode(self, o)
    199 chunks = self.iterencode(o, _one_shot=True)
    200 if not isinstance(chunks, (list, tuple)):
--> 201     chunks = list(chunks)
    202 return ''.join(chunks)

File ~/mambaforge/envs/xarray/lib/python3.9/json/encoder.py:431, in _make_iterencode.<locals>._iterencode(o, _current_indent_level)
    429     yield from _iterencode_list(o, _current_indent_level)
    430 elif isinstance(o, dict):
--> 431     yield from _iterencode_dict(o, _current_indent_level)
    432 else:
    433     if markers is not None:

File ~/mambaforge/envs/xarray/lib/python3.9/json/encoder.py:353, in _make_iterencode.<locals>._iterencode_dict(dct, _current_indent_level)
    351 first = True
    352 if _sort_keys:
--> 353     items = sorted(dct.items())
    354 else:
    355     items = dct.items()

TypeError: '<' not supported between instances of 'str' and 'int'

Problem description

I'm not able to assign attributes using keys with different types. Looks like the issue is coming from json that is trying to sort the keys. I guess there are 3 options:

Only strings are supported: Raise an error if unsupported types are used?
Mixed types are not supported: Raise an error if mixed types are used?
It's a bug: Do not sort the attributes by default?

Version and installation information

Value of zarr.__version__: 2.11.3
Value of numcodecs.__version__: 0.9.1
Version of Python interpreter: 3.9.12
Operating system (Linux/Windows/Mac): Mac
How Zarr was installed (e.g., "using pip into virtual environment", or "using conda"): conda

conda env export

name: xarray channels:

conda-forge dependencies:
affine=2.3.1=pyhd8ed1ab_0
aiobotocore=2.3.2=pyhd8ed1ab_0
aiohttp=3.8.1=py39h63b48b0_1
aioitertools=0.10.0=pyhd8ed1ab_0
aiosignal=1.2.0=pyhd8ed1ab_0
antlr-python-runtime=4.7.2=py39h6e9494a_1003
appdirs=1.4.4=pyh9f0ad1d_0
appnope=0.1.3=pyhd8ed1ab_0
asciitree=0.3.3=py_2
asttokens=2.0.5=pyhd8ed1ab_0
async-timeout=4.0.2=pyhd8ed1ab_0
attrs=21.4.0=pyhd8ed1ab_0
backcall=0.2.0=pyh9f0ad1d_0
backports=1.0=py_2
backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
blosc=1.21.1=h97e831e_3
boost-cpp=1.74.0=hdbf7018_7
boto3=1.21.21=pyhd8ed1ab_0
botocore=1.24.21=pyhd8ed1ab_1
bottleneck=1.3.4=py39h86b5767_1
brotli=1.0.9=h5eb16cf_7
brotli-bin=1.0.9=h5eb16cf_7
brotlipy=0.7.0=py39h63b48b0_1004
bzip2=1.0.8=h0d85af4_4
c-ares=1.18.1=h0d85af4_0
ca-certificates=2022.5.18.1=h033912b_0
cached-property=1.5.2=hd8ed1ab_1
cached_property=1.5.2=pyha770c72_1
cairo=1.16.0=h9e0e54b_1010
cartopy=0.20.2=py39hd74d15e_4
cdat_info=8.2.1=pyhd8ed1ab_2
cdms2=3.1.5=py39ha4b5fb4_14
cdtime=3.1.4=py39hef3cfc1_7
certifi=2022.5.18.1=py39h6e9494a_0
cf-units=3.0.1=py39hc89836e_2
cffi=1.15.0=py39he338e87_0
cfgrib=0.9.10.1=pyhd8ed1ab_0
cfgv=3.3.1=pyhd8ed1ab_0
cfitsio=4.1.0=h2c97ad1_0
cftime=1.6.0=py39h86b5767_1
charset-normalizer=2.0.12=pyhd8ed1ab_0
click=8.1.3=py39h6e9494a_0
click-plugins=1.1.1=py_0
cligj=0.7.2=pyhd8ed1ab_1
cloudpickle=2.1.0=pyhd8ed1ab_0
coverage=6.4=py39h701faf5_0
cryptography=37.0.2=py39h9c2a9ce_0
curl=7.83.1=h372c54d_0
cycler=0.11.0=pyhd8ed1ab_0
cytoolz=0.11.2=py39h63b48b0_2
dask-core=2022.5.1=pyhd8ed1ab_0
decorator=5.1.1=pyhd8ed1ab_0
distarray=2.12.2=pyhd8ed1ab_2
distlib=0.3.4=pyhd8ed1ab_0
distributed=2022.5.1=pyhd8ed1ab_0
eccodes=2.26.0=hf772d56_0
esmf=8.2.0=nompi_h92c19ad_0
esmpy=8.2.0=nompi_py39h8ffa011_1
execnet=1.9.0=pyhd8ed1ab_0
executing=0.8.3=pyhd8ed1ab_0
expat=2.4.8=h96cf925_0
fasteners=0.17.3=pyhd8ed1ab_0
filelock=3.7.0=pyhd8ed1ab_0
findlibs=0.0.2=pyhd8ed1ab_0
flox=0.5.3=pyhd8ed1ab_0
font-ttf-dejavu-sans-mono=2.37=hab24e00_0
font-ttf-inconsolata=3.000=h77eed37_0
font-ttf-source-code-pro=2.038=h77eed37_0
font-ttf-ubuntu=0.83=hab24e00_0
fontconfig=2.14.0=h676cef8_0
fonts-conda-ecosystem=1=0
fonts-conda-forge=1=0
fonttools=4.33.3=py39h701faf5_0
freetype=2.10.4=h4cff582_1
freexl=1.0.6=h0d85af4_0
frozenlist=1.3.0=py39h63b48b0_1
fsspec=2022.5.0=pyhd8ed1ab_0
future=0.18.2=py39h6e9494a_5
g2clib=1.6.3=h7284f3b_1
geos=3.10.2=he49afe7_0
geotiff=1.7.1=had63758_1
gettext=0.19.8.1=hd1a6beb_1008
giflib=5.2.1=hbcb3906_2
h5netcdf=1.0.0=pyhd8ed1ab_0
h5py=3.6.0=nompi_py39hbc6cb89_100
hdf4=4.2.15=hefd3b78_3
hdf5=1.12.1=nompi_ha60fbc9_104
heapdict=1.0.1=py_0
hypothesis=6.46.8=pyhd8ed1ab_0
icu=69.1=he49afe7_0
identify=2.5.1=pyhd8ed1ab_0
idna=3.3=pyhd8ed1ab_0
importlib-metadata=4.11.4=py39h6e9494a_0
importlib_metadata=4.11.4=hd8ed1ab_0
importlib_resources=5.7.1=pyhd8ed1ab_1
iniconfig=1.1.1=pyh9f0ad1d_0
ipython=8.3.0=py39h6e9494a_0
iris=3.2.1=pyhd8ed1ab_0
jasper=2.0.33=h013e400_0
jedi=0.18.1=py39h6e9494a_1
jinja2=3.1.2=pyhd8ed1ab_0
jmespath=1.0.0=pyhd8ed1ab_0
jpeg=9e=h5eb16cf_1
json-c=0.16=h01d06f9_0
jsonschema=4.5.1=pyhd8ed1ab_0
jupyter_core=4.10.0=py39h6e9494a_0
kealib=1.4.14=ha22a8b1_3
kiwisolver=1.4.2=py39h7248d28_1
krb5=1.19.3=hb49756b_0
lazy-object-proxy=1.7.1=py39h63b48b0_1
lcms2=2.12=h577c468_0
lerc=3.0=he49afe7_0
libaec=1.0.6=he49afe7_0
libblas=3.9.0=14_osx64_openblas
libbrotlicommon=1.0.9=h5eb16cf_7
libbrotlidec=1.0.9=h5eb16cf_7
libbrotlienc=1.0.9=h5eb16cf_7
libcblas=3.9.0=14_osx64_openblas
libcdms=3.1.2=h7d8e03a_117
libcf=1.0.3=py39h8d2e2c5_114
libcurl=7.83.1=h372c54d_0
libcxx=14.0.3=hc203e6f_0
libdap4=3.20.6=h3e144a0_2
libdeflate=1.10=h0d85af4_0
libdrs=3.1.2=h102a9a4_118
libdrs_f=3.1.2=h2534672_114
libedit=3.1.20191231=h0678c8f_2
libev=4.33=haf1e3a3_1
libffi=3.4.2=h0d85af4_5
libgdal=3.4.3=h6474519_0
libgfortran=5.0.0=9_3_0_h6c81a4c_23
libgfortran5=9.3.0=h6c81a4c_23
libglib=2.70.2=hf1fb8c0_4
libiconv=1.16=haf1e3a3_0
libkml=1.3.0=h8fd9edb_1014
liblapack=3.9.0=14_osx64_openblas
libllvm10=10.0.1=h009f743_3
libnetcdf=4.8.1=nompi_h6609ca0_102
libnghttp2=1.47.0=h942079c_0
libopenblas=0.3.20=openmp_hb3cd9ec_0
libpng=1.6.37=h7cec526_2
libpq=14.2=hea3049e_0
librttopo=1.1.0=hec60dd8_9
libspatialite=5.0.1=hadde3e2_15
libssh2=1.10.0=h52ee1ee_2
libtiff=4.3.0=hfca7e8f_4
libuuid=2.32.1=h35c211d_1000
libwebp=1.2.2=h28dabe5_0
libwebp-base=1.2.2=h0d85af4_1
libxcb=1.13=h0d85af4_1004
libxml2=2.9.12=h7e28ab6_1
libxslt=1.1.33=h1acebb3_3
libzip=1.8.0=h8b0c345_1
libzlib=1.2.11=h6c3fc93_1014
llvm-openmp=14.0.3=ha654fa7_0
llvmlite=0.36.0=py39h798a4f4_0
locket=1.0.0=pyhd8ed1ab_0
lxml=4.8.0=py39h63b48b0_2
lz4-c=1.9.3=he49afe7_1
markupsafe=2.1.1=py39h63b48b0_1
matplotlib-base=3.5.2=py39h64a0072_0
matplotlib-inline=0.1.3=pyhd8ed1ab_0
msgpack-python=1.0.3=py39h7248d28_1
multidict=6.0.2=py39h63b48b0_1
munkres=1.1.4=pyh9f0ad1d_0
nbformat=5.4.0=pyhd8ed1ab_0
nc-time-axis=1.4.1=pyhd8ed1ab_0
ncurses=6.3=h96cf925_1
netcdf-fortran=4.5.4=nompi_h9ed14b0_100
netcdf4=1.5.8=nompi_py39he7d1c46_101
nodeenv=1.6.0=pyhd8ed1ab_0
nspr=4.32=hcd9eead_1
nss=3.78=ha8197d3_0
numba=0.53.1=py39h32e38f5_1
numcodecs=0.9.1=py39h9fcab8e_2
numexpr=2.8.0=py39hbd61c47_2
numpy=1.22.4=py39h677350a_0
numpy_groupies=0.9.16=pyhd8ed1ab_0
openblas=0.3.20=openmp_h5ad848b_0
openjpeg=2.4.0=h6e7aa92_1
openssl=1.1.1o=hfe4f2af_0
packaging=21.3=pyhd8ed1ab_0
pandas=1.4.2=py39hbd61c47_1
parso=0.8.3=pyhd8ed1ab_0
partd=1.2.0=pyhd8ed1ab_0
patsy=0.5.2=pyhd8ed1ab_0
pcre=8.45=he49afe7_0
pexpect=4.8.0=pyh9f0ad1d_2
pickleshare=0.7.5=py_1003
pillow=9.1.1=py39h579eac4_0
pint=0.19.2=pyhd8ed1ab_0
pip=22.1.1=pyhd8ed1ab_0
pixman=0.40.0=hbcb3906_0
platformdirs=2.5.1=pyhd8ed1ab_0
pluggy=1.0.0=py39h6e9494a_3
pooch=1.6.0=pyhd8ed1ab_0
poppler=22.04.0=h39497b0_0
poppler-data=0.4.11=hd8ed1ab_0
postgresql=14.2=he8fe76e_0
pre-commit=2.19.0=py39h6e9494a_0
proj=9.0.0=h2364a93_1
prompt-toolkit=3.0.29=pyha770c72_0
pseudonetcdf=3.2.2=pyhd8ed1ab_0
psutil=5.9.1=py39h701faf5_0
pthread-stubs=0.4=hc929b4f_1001
ptyprocess=0.7.0=pyhd3deb0d_0
pure_eval=0.2.2=pyhd8ed1ab_0
py=1.11.0=pyh6c4a22f_0
pycparser=2.21=pyhd8ed1ab_0
pygments=2.12.0=pyhd8ed1ab_0
pyopenssl=22.0.0=pyhd8ed1ab_0
pyparsing=3.0.9=pyhd8ed1ab_0
pyproj=3.3.1=py39hfe197cf_0
pyrsistent=0.18.1=py39h63b48b0_1
pyshp=2.3.0=pyhd8ed1ab_0
pysocks=1.7.1=py39h6e9494a_5
pytest=7.1.2=py39h6e9494a_0
pytest-cov=3.0.0=pyhd8ed1ab_0
pytest-env=0.6.2=py_0
pytest-forked=1.4.0=pyhd8ed1ab_0
pytest-xdist=2.5.0=pyhd8ed1ab_0
python=3.9.12=h8b4d769_1_cpython
python-dateutil=2.8.2=pyhd8ed1ab_0
python-eccodes=1.4.0=py39h86b5767_1
python-fastjsonschema=2.15.3=pyhd8ed1ab_0
python-xxhash=3.0.0=py39h63b48b0_1
python_abi=3.9=2_cp39
pytz=2022.1=pyhd8ed1ab_0
pyyaml=6.0=py39h63b48b0_4
rasterio=1.2.10=py39hc9cfaa8_5
readline=8.1=h05e3726_0
requests=2.27.1=pyhd8ed1ab_0
s3transfer=0.5.2=pyhd8ed1ab_0
scipy=1.8.1=py39hfa1a3ab_0
seaborn=0.11.2=hd8ed1ab_0
seaborn-base=0.11.2=pyhd8ed1ab_0
setuptools=62.3.2=py39h6e9494a_0
shapely=1.8.2=py39hd471b09_1
six=1.16.0=pyh6c4a22f_0
snappy=1.1.9=h6e38e02_1
snuggs=1.4.7=py_0
sortedcontainers=2.4.0=pyhd8ed1ab_0
sparse=0.13.0=pyhd8ed1ab_0
sqlite=3.38.5=hd9f0692_0
stack_data=0.2.0=pyhd8ed1ab_0
statsmodels=0.13.2=py39hc89836e_0
tblib=1.7.0=pyhd8ed1ab_0
tiledb=2.8.3=hf17dc58_1
tk=8.6.12=h5dbffcc_0
toml=0.10.2=pyhd8ed1ab_0
tomli=2.0.1=pyhd8ed1ab_0
toolz=0.11.2=pyhd8ed1ab_0
tornado=6.1=py39h63b48b0_3
traitlets=5.2.1.post0=pyhd8ed1ab_0
typing-extensions=4.2.0=hd8ed1ab_1
typing_extensions=4.2.0=pyha770c72_1
tzcode=2022a=h5eb16cf_0
tzdata=2022a=h191b570_0
udunits2=2.2.28=h06ef574_0
ukkonen=1.0.1=py39h7248d28_2
unicodedata2=14.0.0=py39h63b48b0_1
urllib3=1.26.9=pyhd8ed1ab_0
virtualenv=20.14.1=py39h6e9494a_0
wcwidth=0.2.5=pyh9f0ad1d_2
wheel=0.37.1=pyhd8ed1ab_0
wrapt=1.14.1=py39h701faf5_0
xerces-c=3.2.3=h6564042_4
xorg-libxau=1.0.9=h35c211d_0
xorg-libxdmcp=1.1.3=h35c211d_0
xxhash=0.8.0=h35c211d_3
xz=5.2.5=haf1e3a3_1
yaml=0.2.5=h0d85af4_2
yarl=1.7.2=py39h63b48b0_2
zarr=2.11.3=pyhd8ed1ab_0
zict=2.2.0=pyhd8ed1ab_0
zipp=3.8.0=pyhd8ed1ab_0
zlib=1.2.11=h6c3fc93_1014
zstd=1.5.2=ha9df2e0_1
pip:
- numbagg==0.2.1 prefix: /Users/mattia/mambaforge/envs/xarray

May 25 '22 11:05 malmans2

see:

https://bugs.python.org/issue25457
http://json-schema.org/understanding-json-schema/reference/object.html "in JSON all the keys must be strings"

I think I'd tend towards "Only strings are supported" but another sub-option would be to print a warning and automatically stringify.

Jun 23 '22 08:06 joshmoore

Agree on only supporting strings, much easier for everyone.

Jun 29 '22 14:06 clbarnes

@clbarnes (or anyone else): and thoughts on completely disallowing versus stringifying arguments?

Jul 04 '22 13:07 joshmoore

Strong preference for documenting/ annotating that keys must be strings (and that values must be json-serialisable), and not stringifying non-string keys (which would be unpredictable for different types).

JavaScript gets pretty confusing with its string coercion, no need to open ourselves up to that. Python's got a whole lot better to develop in now there's more focus on sane types. Explicitly checking for key string-ness won't check for the JSON-validity the value (which might be a dict with non-string keys), so it's probably best to just leave it all up to the serialiser.

Jul 04 '22 13:07 clbarnes

@clbarnes, objection noted. I guess I'm most worried about the impact of the breaking change since at the moment a single update (z.attrs[1] = "foo") gets you:

/tmp $cat test.zarr/.zattrs
{
    "1": "foo"
}

Jul 11 '22 12:07 joshmoore

Maybe a couple of versions of noisy deprecation warnings, then? Even if it never actually gets patched out until v3.

Jul 11 '22 12:07 clbarnes

Thanks all. Rolling @malmans2's #1066 into 2.13 but leaving this open for the deprecation cycle.

Sep 08 '22 06:09 joshmoore

Strike that see #1125

Sep 08 '22 06:09 joshmoore

zarr-python zarr-python copied to clipboard

Error when using attribute keys with mixed types

Minimal, reproducible code sample, a copy-pastable example if possible

Problem description

Version and installation information

zarr-python
zarr-python copied to clipboard