Visualzation of ROIs embedded in ome.zarr?

[BioinfoTongLI]

trafficstars

Hello all,

Thanks for this amazing work! Having interactive image browing within a notebook is really something that helps a lot! I am wondering whether you are planning on adding some other types of layers at all, such as shapes? I am aware that I can do that with Imjoy and vitessce. They are however much heavier that this amazing inline vizarr. thanks for you helps in advance!

Best, Tong

[will-moore]

The "support of ROIs in ome.zarr" should be discussed on the https://github.com/ome/ngff repo. But for the support of Shapes in vizarr, I started a PR at https://github.com/hms-dbmi/vizarr/pull/128

[BioinfoTongLI]

The "support of ROIs in ome.zarr" should be discussed on the https://github.com/ome/ngff repo. But for the support of Shapes in vizarr, I started a PR at #128

Sorry, the title is indeed misleading. Amended. Awesome! The PR looks great!

[will-moore]

This quickly gets more complex when you consider that you might want to provide Polygons (or other shapes) that should only appear on specific Z and T indices of a multi-dimensional image. I guess we need to move to providing Shape objects that have optional z and t indices, but OME-NGFF will soon move to arbitrary named axes (https://github.com/ome/ngff/pull/57) so that needs to be handled too! It may be that we need OME-NGFF to come up with a spec for Shapes first?

[will-moore]

Created https://github.com/ome/ngff/issues/73 for OME-NGFF support of Shapes.

[BioinfoTongLI]

This quickly gets more complex when you consider that you might want to provide Polygons (or other shapes) that should only appear on specific Z and T indices of a multi-dimensional image.

Indeed, the actual use case that I am aiming for is to not only show polygons, but also annotations (attached metadata as pd.DataFrame maybe?) while hovering. Although, having polygon overlays is definitely good start for us!

I guess we need to move to providing Shape objects that have optional z and t indices, but OME-NGFF will soon move to arbitrary named axes (ome/ngff#57) so that needs to be handled too!

That reminds me another detail, we've been playing with Vitessce recently as well. According to my shallow understanding, Vitessce only support the ome.zarr converted using bioformat2raw<=0.2.0. Is that the case for vizarr as well? Or vizarr is always using the latest ngff release? Asking this question because, the ROIs in ome.zarr should be already there if it was converted from ome.tiff with ROIs in omexml, right? The reason pointing to that PR is because you think it's better to be inline with the latest version of ngff, am I right?

It may be that we need OME-NGFF to come up with a spec for Shapes first?

Yes, agree - More than happy to help if I will be any useful :)

[will-moore]

I don't know about Vitessce I'm afraid, but we (OME) certainly aim to support the latest OME-NGFF versions in vizarr. Even if there are some new features of the spec that can't be supported in vizarr, at least it should open the image and support as much as possible. Since we don't support ROIs in OME-NGFF yet, I don't think that bioformats2raw can preserve them (unless it is simply preserving the XML within the file somehow, but I don't think so). cc @sbesson @joshmoore

I believe Vitessce should/did support OME-NGFF v0.1. See Demos at https://github.com/vitessce/vitessce/pull/816#issuecomment-750953410 and whole discussion there. I just tried latest vitessce.io with the demo ?url=... Demo1 works but in Demo2 the images look black, although the data is being loaded (looking at dev tools network tab).

[manzt]

Hi @BioinfoTongLI, thanks for opening the issue!

I don't know about Vitessce I'm afraid, but we (OME) certainly aim to support the latest OME-NGFF versions in vizarr.

Vitessce is more feature-rich compared to vizarr; however, that has meant that a combination of data-models are utilized which can be complicated to then support OME-NGFF completely. Vizarr is intended to be much more minimal, and target solely the latest OME-NGFF.

Even if there are some new features of the spec that can't be supported in vizarr, at least it should open the image and support as much as possible.

Agreed. The aim is to support as much of the latest OME-NGFF as possible. I'm somewhat purposely avoiding defining concepts that will be defined in OME-NGFF (e.g. ROIs) on their own in vizarr to avoid creating conflict of some vizarr/OME-NGFF data-model. This way an image that you can open in vizarr will also open in another tool that supports the latest OME-NGFF, and we won't run into the issue where an dataset (or parts of datasets) only opens in vizarr.

I just tried latest vitessce.io with the demo ?url=... Demo1 works but in Demo2 the images look black, although the data is being loaded (looking at dev tools network tab).

I'll need to take a closer look into this to see what is going on.