vizarr
vizarr copied to clipboard
hms-dbmi
Support OME-zarr labels layers
It would be great to support ome-zarr labels data. This is a zarr image where values of 0 are transparent (no label) and colour of each label may be specified by metadata: https://github.com/ome/omero-ms-zarr/blob/master/spec.md#image-label-metadata
I can imagine that loading the zarr data into new layers won't need much new code (happy to have a stab at this), but I don't know how to render particular pixel values with pre-defined colours? It's important that the labels layer is transparent so that the underlying image is visible, so I believe they would have to be part of the same Deck.gl layer?
See ome-zarr labels support in napari, added via ome-zarr-py plugin: https://github.com/ome/ome-zarr-py/pull/30
These labels layers may be part of the same ome-zarr that contains image layers (see spec) but it's possible that you might load image layers from one source and labels from another.
Having just been reading https://github.com/hms-dbmi/vizarr/issues/32, I can imagine that an API for showing/hiding or setting colors on masks would be very cool. Similar to what you're doing in vitessce http://vitessce.io/?dataset=linnarsson-2018
cc @joshmoore
Another example of showing / hiding / coloring labels: https://downloads.openmicroscopy.org/presentations/2020/community-call-2020-10-29/OMERO%20Plus,%20PathViewer%20and%20NGFF%202020-10-27.mp4
Ah, I wasn't aware that the labels data were dense arrays. What are the non-zero elements for the labels (value wise)? I think we should be able to reuse the loader modules (to fetch data tiles), but I think transparaent for 0 values will require a new layer (or shaders at minimum) perhaps @ilan-gold has some input.
Having just been reading #32, I can imagine that an API for showing/hiding or setting colors on masks would be very cool. Similar to what you're doing in vitessce http://vitessce.io/?dataset=linnarsson-2018
I think we use a deck.gl polygon layer for the segmentation in vitessce because the data are in a columnar format (not dense pixel arrays). Both @keller-mark and @ilan-gold have more experience with writing custom layers in deck.gl, and hopefully they have some ideas of how to do something similar for dense array masks
With that said, I like the idea of having different layers because then we can render UI elements based on different layerProps for each layer.
Yes we use selectable polygon layers from deck.gl which requires pre-processing the data to get polygons but likely will not require any sort of pyramid/multi-resolution handling.
We could certainly write shaders to handle this concept of value-per-color, raster/pixel-based-storage, segmentation mask, especially if this is the direction OMERO is headed - it would be very straightforward to do and could scale very well to lots of different labels if what I have in my head is right for the implementation. I agree @manzt that we should be able to reuse the loaders for it - we would change the layer (or really just the shaders) used by XRLayer to accommodate this new data and reuse the MultiscaleImageLayer with a few tweaks to the APIs.
We could certainly write shaders to handle this concept of value-per-color, raster/pixel-based-storage, segmentation mask, especially if this is the direction OMERO is headed
See Will's link above (https://github.com/ome/omero-ms-zarr/blob/master/spec.md#image-label-metadata). The OME-Zarr specification uses (dense) zarr arrays for masks.
@will-moore
Ilan and I talked about this today and I think we have an idea for implementation (likely a new layer with custom shaders). I would be useful to have an example dataset with these arrays to work with. I can't seem to find the an image with "image-label" metadata? Is this different from "masks"?
I've tried opening the examples in ome/ome-zarr-py#30 with the ome-zarr-py plugin and the masks don't seem to load the masks layers. Again, I'm not sure if mask / image-label are different.
$ napari https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001240.zarr # just opens image layers
Yeah, I had to ask around to find an image with labels. (we are planning a directory page to list all our ome-zarr images soon). This one has labels:
napari https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001247.zarr
The presence of labels is specified by the existence of https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/6001247.zarr/labels/.zattrs
Labels are multi-resolution arrays (same as pixel data) but tiles may be missing if they are all zeros to save space when you have sparsely-populated labels.