zarr-python icon indicating copy to clipboard operation
zarr-python copied to clipboard
verified publisher icon zarr-developers

Tracking 3.0 Release Blockers

Open jhamman opened this issue 1 year ago • 1 comments

We are now in the last mile of the 3.0 refactor. This issue is tracking what we are considering release blockers: There is also a milestone for After 3.0.0 where we are putting tickets that can come later.

Docs

  • [ ] https://github.com/zarr-developers/zarr-python/issues/1765
  • [ ] https://github.com/zarr-developers/zarr-python/issues/1766
  • [ ] https://github.com/zarr-developers/zarr-python/issues/1767
  • [ ] https://github.com/zarr-developers/zarr-python/issues/1770
  • [ ] https://github.com/zarr-developers/zarr-python/issues/1798
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2245
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2246
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2247
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2248
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2251
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2252
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2253
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2289

Array

  • [ ] https://github.com/zarr-developers/zarr-python/issues/1750
  • [x] https://github.com/zarr-developers/zarr-python/issues/1977
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2029
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2267
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2347
  • [x] https://github.com/zarr-developers/zarr-python/issues/2404
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2409

Group

  • [ ] https://github.com/zarr-developers/zarr-python/issues/2135
  • [x] https://github.com/zarr-developers/zarr-python/issues/2402
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2406

Top level API

  • [ ] https://github.com/zarr-developers/zarr-python/issues/2403
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2407
  • [x] https://github.com/zarr-developers/zarr-python/issues/2408
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2411

Store API

  • [ ] https://github.com/zarr-developers/zarr-python/issues/1755
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2108
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2191
  • [x] https://github.com/zarr-developers/zarr-python/issues/2342
  • [ ] https://github.com/zarr-developers/zarr-python/issues/2359

Testing / CI

  • [x] https://github.com/zarr-developers/zarr-python/issues/1496
  • [ ] https://github.com/zarr-developers/zarr-python/issues/1648
  • [ ] https://github.com/zarr-developers/zarr-python/issues/1759
  • [ ] https://github.com/zarr-developers/zarr-python/issues/1762

This is non exhaustive but these are the major known issues. Feel free to post additional tickets and we can evaluate if they are really blockers.

jhamman avatar Oct 18 '24 20:10 jhamman

I would suggest:

  • #2377
  • #2380
  • #2419

DimitriPapadopoulos avatar Oct 20 '24 12:10 DimitriPapadopoulos

Hi, Thanks for all the work towards v3. I wanted to check if this is a blocker / existing issue etc or if I should create one...? This is using main branch 680142f6 :

import zarr
from zarr.storage import LocalStore

path = "root.zarr"
fmt = 2
store = LocalStore(path, mode="w")
root = zarr.group(store=store, zarr_format=fmt)

root_group = zarr.open(f"{path}", zarr_format=fmt)

Gives me:

$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 48, in <module>
    root_group = zarr.open(f"{path}", zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

will-moore avatar Nov 07 '24 16:11 will-moore

Hi, Thanks for all the work towards v3. I wanted to check if this is a blocker / existing issue etc or if I should create one...? This is using main branch 680142f :

import zarr
from zarr.storage import LocalStore

path = "root.zarr"
fmt = 2
store = LocalStore(path, mode="w")
root = zarr.group(store=store, zarr_format=fmt)

root_group = zarr.open(f"{path}", zarr_format=fmt)

Gives me:

$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 48, in <module>
    root_group = zarr.open(f"{path}", zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

It looks like you are not passing the store instance to zarr.open. Does root_group = zarr.open(store, path=path, zarr_format=fmt) work?

d-v-b avatar Nov 07 '24 16:11 d-v-b

If I pass in both the store and the path, it fails with FileNotFoundError: file://root.zarr/root.zarr, so it looks like I don't need both? It also seems I want to create a store with read mode. But this is still failing (but works with fmt = 3)...

read_store = LocalStore(path)
root_group = zarr.open(read_store, zarr_format=fmt)
FileNotFoundError: file://root.zarr 
$ python test_open_v2_group.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/ZARR/ome-zarr-py/test_open_v2_group.py", line 49, in <module>
    root_group = zarr.open(read_store, zarr_format=fmt)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/_compat.py", line 43, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 77, in open
    obj = sync(
          ^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 141, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 100, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 308, in open
    return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1077, in open_array
    return await AsyncArray.open(store_path, zarr_format=zarr_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 680, in open
    metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 135, in get_array_metadata
    raise FileNotFoundError(store_path)
FileNotFoundError: file://root.zarr

will-moore avatar Nov 07 '24 16:11 will-moore

ugh, this flow should definitely be super easy, sorry that it's not working. If I have some time tonight I will dig into this.

d-v-b avatar Nov 07 '24 16:11 d-v-b

@d-v-b with this change, that test passes for me and fixes about 100 failing tests at https://github.com/ome/ome-zarr-py/pull/404

diff --git a/src/zarr/api/asynchronous.py b/src/zarr/api/asynchronous.py
index 40f9b8d5..8dc69250 100644
--- a/src/zarr/api/asynchronous.py
+++ b/src/zarr/api/asynchronous.py
@@ -306,7 +306,7 @@ async def open(
 
     try:
         return await open_array(store=store_path, zarr_format=zarr_format, **kwargs)
-    except (KeyError, NodeTypeValidationError):
+    except (KeyError, NodeTypeValidationError, FileNotFoundError):
         # KeyError for a missing key
         # NodeTypeValidationError for failing to parse node metadata as an array when it's
         # actually a group

will-moore avatar Nov 11 '24 13:11 will-moore

Thanks for all your work on Zarr Python 3.0!

I know this is truly the last quarter mile (or km) before v3.0 and apologize for being so last minute, but since this is the last shot for potential user facing API changes I wanted to promote https://github.com/zarr-developers/zarr-python/issues/2437 as something to settle before the release. In particular, I think it's important to make an explicit decision on whether the tuple(int, int) representation will remain (start, length) vs (start, stop) and whether suffix requests should be denoted by negative integers vs. a typed dict. If people are open to following the same approach as Kyle proposes in https://github.com/zarr-developers/zarr-python/issues/2437#issuecomment-2452528308, I would be glad to try implementing this suggestion over the holidays to ensure there are no severe memory/time consequences. I'm +1 on Kyle's idea based on the number of people who have misunderstood the current semantics and the apparent negligible performance impact of more intuitive typing.

maxrjones avatar Dec 19 '24 19:12 maxrjones