tensorstore
tensorstore copied to clipboard
Are write operations with the zarr Driver guaranteed to be thread- and process-safe?
Suppose I have an existing on-disk Zarr array. If I were to have two separate processes that:
- Open this Zarr array via
tensorstore.open
- Write to separate regions that potentially share the same chunks within the Zarr array
Are these two write operations guaranteed to write correctly?
For example, suppose my.zarr
has a chunk shape of (64,64,64).
- Process 1 writes to (0,0,0):(64,64,32)
- Process 2 writes to (0,0,32):(64,64,64)
# Process 1
path = "path/to/my.zarr"
arr = ts.open(
{
"driver": "zarr",
"kvstore": {"driver": "file", "path": path},
},
open=True,
read=True,
write=True,
create=False,
).result()
arr[(0,0,0):(64,64,32)] = 100
# Process 2
path = "path/to/my.zarr"
arr = ts.open(...) # Same as Process 1
arr[(0,0,32):(64,64,64)] = 200
The only mention I could find was in the homepage, under the list of highlights.
Supports safe, efficient access from multiple processes and machines via optimistic concurrency.
And some basic testing seems to suggest that this is indeed true.
However, is this guaranteed to be the case? Is there anything within the documentation that provides this guarantee?
P.S. Out of curiosity, how is the OCC actually implemented? Checking the last modified date of the Zarr chunk in which to write, or something along these lines? P.P.S. Great library, by the way