spatialdata icon indicating copy to clipboard operation
spatialdata copied to clipboard

Allow reloading of particular elements, and add `reload: bool` parameter to write functions

Open LucaMarconato opened this issue 1 year ago • 0 comments

With the new incremental IO PR we can write single elements or metadata into a Zarr store. However, reading is still a monolithic operation that reads all at once.

We should

  • [ ] refactor the code for reading into individual components to read single elements and metadata of single elements
  • [ ] add new APIs to reload the data, metadata and in particular transformations of a single element in a SpatialData object.
  • [ ] consider adding a parameter return_stored (as discussed here https://docs.dask.org/en/latest/generated/dask.array.store.html#dask.array.store), or maybe reload, to allow the user to call write_element() and write() and return an object that is already reloaded with Dask. write(reload=True) is equivalent to calling sdata = read_zarr(sdata.path)
  • [ ] write_element(reload=True) would introduce a new convenient functionality that has for instance been mentioned by @ArneDefauw here https://github.com/scverse/spatialdata/pull/501#issuecomment-2018315221, and could be used to simplify in particular the workaround 1 for overwriting elements (test_incremental_io_on_disk()).

LucaMarconato avatar Mar 27 '24 21:03 LucaMarconato