Include `v3` in Dask's nightly CI when ready

[jrbourbeau]

Over in Dask's nightly CI build that tests against nightly versions of Dask dependencies, we started seeing Zarr-related test failures like

FAILED dask/array/tests/test_array_core.py::test_from_zarr_unique_name - TypeError: 'module' object is not callable
FAILED dask/array/tests/test_array_core.py::test_from_zarr_name - TypeError: 'module' object is not callable
FAILED dask/array/tests/test_array_core.py::test_zarr_roundtrip - AttributeError: module 'zarr' has no attribute 'create'
FAILED dask/array/tests/test_array_core.py::test_zarr_roundtrip_with_path_like - AttributeError: module 'zarr' has no attribute 'create'
FAILED dask/array/tests/test_array_core.py::test_to_zarr_accepts_empty_array_without_exception_raised - AttributeError: module 'zarr' has no attribute 'create'
FAILED dask/array/tests/test_array_core.py::test_zarr_return_stored[False] - AttributeError: module 'zarr' has no attribute 'create'
FAILED dask/array/tests/test_array_core.py::test_zarr_return_stored[True] - AttributeError: module 'zarr' has no attribute 'create'
FAILED dask/array/tests/test_array_core.py::test_zarr_inline_array[True] - TypeError: 'module' object is not callable
FAILED dask/array/tests/test_array_core.py::test_zarr_inline_array[False] - TypeError: 'module' object is not callable
FAILED dask/array/tests/test_array_core.py::test_zarr_existing_array - AttributeError: module 'zarr' has no attribute 'zeros_like'
FAILED dask/array/tests/test_array_core.py::test_read_zarr_chunks - AttributeError: module 'zarr' has no attribute 'create'
FAILED dask/array/tests/test_array_core.py::test_zarr_pass_mapper - ModuleNotFoundError: No module named 'zarr.storage'

due to missing top-level attributes like zarr.create, zarr.zeros_like, etc. (see this build for full tracebacks).

I'm able to reproduce these types of errors locally (w/o dask) with the following steps:

mamba create -n test python=3.12
mamba activate test
pip install git+https://github.com/zarr-developers/zarr-python
python -c "import zarr; print(zarr.create)"

which raises this AttributeError

Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'zarr' has no attribute 'create'

With the latest zarr=2.18.2 release, I don't get an error

python -c "import zarr; print(zarr.create)"
<function create at 0x105295c60>

One odd thing is that pip install git+https://github.com/zarr-developers/zarr-python seems to be installing zarr-2.16.2.dev95+gb1f4c50, which seems off given the latest release is 2.18.2.

It's unclear to be if something has changed in this repo, or if we should update our "install the nightly version of Zarr" command over in dask/dask.

[jrbourbeau]

Hmm looks like the default branch is v3 here and if we instead explicitly install from main

pip install git+https://github.com/zarr-developers/zarr-python.git@main

then things work as expected.

Could someone help me understand the difference between main and the (default) v3 branch?

[jhamman]

@jrbourbeau - thanks for opening the issue. I recommend taking a look at #1777 for a summary of the v3 effort. In the short term, I suggest pointing to main as you have done. In the medium term, once our alpha release is out, we would love to get Dask's test suite running against the v3 branch again.

Also, #1884 will be fixing the specific errors you encountered.

[jrbourbeau]

Thanks for the extra context @jhamman, that's helpful

I suggest pointing to main as you have done

Cool, doing that here https://github.com/dask/dask/pull/11151

once our alpha release is out, we would love to get Dask's test suite running against the v3 branch again

I'm happy to help out when that happens

[jhamman]

closed by https://github.com/dask/dask/pull/11388