tensorstore icon indicating copy to clipboard operation
tensorstore copied to clipboard

Are write operations with the zarr Driver guaranteed to be thread- and process-safe?

Open xantho09 opened this issue 4 months ago • 3 comments

Suppose I have an existing on-disk Zarr array. If I were to have two separate processes that:

  1. Open this Zarr array via tensorstore.open
  2. Write to separate regions that potentially share the same chunks within the Zarr array

Are these two write operations guaranteed to write correctly?

For example, suppose my.zarr has a chunk shape of (64,64,64).

  • Process 1 writes to (0,0,0):(64,64,32)
  • Process 2 writes to (0,0,32):(64,64,64)
# Process 1
path = "path/to/my.zarr"
arr = ts.open(
    {
        "driver": "zarr",
        "kvstore": {"driver": "file", "path": path},
    },
    open=True,
    read=True,
    write=True,
    create=False,
).result()

arr[(0,0,0):(64,64,32)] = 100
# Process 2
path = "path/to/my.zarr"
arr = ts.open(...) # Same as Process 1

arr[(0,0,32):(64,64,64)] = 200

The only mention I could find was in the homepage, under the list of highlights.

Supports safe, efficient access from multiple processes and machines via optimistic concurrency.

And some basic testing seems to suggest that this is indeed true.

However, is this guaranteed to be the case? Is there anything within the documentation that provides this guarantee?

P.S. Out of curiosity, how is the OCC actually implemented? Checking the last modified date of the Zarr chunk in which to write, or something along these lines? P.P.S. Great library, by the way

xantho09 avatar Sep 30 '24 10:09 xantho09