bridge to MoBIE

[martinschorb]

Hi,

we would be very keen in using micro-sam in conjunction with MoBIE.

Where would be good points for coding linkages for:

• opening OME-Zarr data from a MoBIE project in napari/micro-sam
• exporting segmentation/label data back to MoBIE (knowing the original source and adding it to the project accordingly)

Would you implement this in a separate napari plugin or right here?

[constantinpape]

we would be very keen in using micro-sam in conjunction with MoBIEThat's a great idea overall, but I think we don't need to make this micro-sam specific, but would rather need a plugin that provides a napari / MoBIE interface.

• opening OME-Zarr data from a MoBIE project in napari/micro-sam

This should already be possible with https://github.com/ome/napari-ome-zarr. But we could write something around this that makes it easier to navigate the MoBIE project structure.

• exporting segmentation/label data back to MoBIE (knowing the original source and adding it to the project accordingly)

Yep, we would need to add this functionality.

Would you implement this in a separate napari plugin or right here?

I think it would be best to have a separate MoBIE napari plugin that supports both of the points you described above. This would then be able to interact with any kind of segmentation plugin in napari, e.g. also stardist or mitonet/empanadas. This can live in https://github.com/mobie.

We are currently working on a proper plugin design for micro-sam: https://github.com/computational-cell-analytics/micro-sam/tree/dev. This is part of the revision and my focus right now. If you want you could get started with the napari mobie plugin already, for now I could give some hints, and once we are through with the revision (hopefully early April), I could also help out a bit more there.

[martinschorb]

Great! That's exactly what I have in mind as well. I will dig a bit into designing napari plugins (this would be my first), so if you can provide some rudimentary guidance from your experience going beyond the existing tutorials, that would be amazing.

[constantinpape]

The best starting point for the plugin is the cookiecutter repo: https://github.com/napari/cookiecutter-napari-plugin This will create the necessary strucutre and contains good examples already.

For the plugin, I think we should create three widgets (maybe there's more, but this is what I can think of right now)

• Widget to select the MoBIE Project (Project Root and Dataset)
• Widget to import raw data: Select an image from the dataset and add it as image layer (either via a field that let's the user specify the source name and/or a dropdown that lists all sources). This is relatively straightforward by returning layers in the widget, see https://napari.org/stable/guides/magicgui.html#return-annotations. You can get the list of image sources from the dataset.json
• Widget to export segmentation to MoBIE: select a segmentation layer and then use add_segmentation from mobie-python to add it to the project. (+ specify relevant metadata).

I would start implementing it for local data only, we can think of remote projects later.

[GenevieveBuckley]

I'm happy to help with this too @martinschorb. I've made a couple of small napari plugins, so possibly can be useful there for you too 😄

[martinschorb]

Hi again, The first widget that opens source(s) from a MoBIE view is ready. I still need to finalize some GUI tests. https://github.com/mobie/mobie-napari-bridge Also, there is quite some example code from the original cookiecutter recipe left, that I use to orient myself.

I would need some advice on the following points:

• do you have a test that deals with active QT dialogs, like QFileDialog. For me it always leads to some stall/infinite loop.
• is the metadata field in a napari layer where additional information should go (where the image came from, transformations that are needed to later write a processed layer back correctly into the MoBIE project)? https://github.com/mobie/mobie-napari-bridge/blob/d86ef33d5007bf0ee77a5dbcc8665f56b86e3ba9/src/mobie_napari_bridge/util.py#L27

[martinschorb]

@GenevieveBuckley I found your discussion here: https://github.com/pyapp-kit/magicgui/pull/23

is this still the recommended way?

[martinschorb]

My plugin now supports loading OME-Zarr sources from a MoBIE project.

Now, I like to use one resulting image layer and do some segmentation using micro-sam. However, when I load the annotator, I have to provide an image file. And in the selector OME-Zarr is not supported as it specifically expects a file.

Also, if I understand napari's label layers correctly, their size depends on the active layer. Therefore adding an imported OME-Zarr layer later will not work as the image dimensions don't fit.

Is there an elegant way to get micro-sam working in these cases? How does it treat multi-scale image layers?

[constantinpape]

Hi @martinschorb , we are still finalizing the latest release, which will improve the napari integration significantly. It will only be a few more days (PR is already there, see #516 but we still need to test and document it). I will get back to you once this is done :)

[constantinpape]

Hi @martinschorb, I want to look into some things for this next week. I think the biggest questions for now is how we handle the pyramid layer and how we connect the segmentation layer back to MoBIE. Do you have a small example for the MoBIE plugin somewhere that can be easily used? That would be super helpful to start working on this. Thanks!

[martinschorb]

Hi @constantinpape ,

you can use the "Select MoBIE repository" button to open any OME-Zarr from S3, even without a local MoBIE project folder. This results in a multiscale napari image layer (using the napari-ome-zarr plugin).

I tend to use https://github.com/mobie/clem-example-project -> hela -> EM-overview for testing.

[constantinpape]

Hi @martinschorb , the release took a bit longer, but is now done!

So I started looking into this and implemented the first approach for dealing with multiscale image layers in #590. There is still some things to do, but it should not be too much effort. It is already working for computing embeddings on the highest resolution, see details in the PR.

[GenevieveBuckley]

@GenevieveBuckley I found your discussion here: pyapp-kit/magicgui#23

is this still the recommended way?

I think so? I'm not aware of any substantial changes since that time, but I also have not been keeping track of updates, etc.

[constantinpape]

@martinschorb #590 is working for 2d multiscale data now! I still need to test it for 3d data, but will probably only have time for that later next week. But you can already check it out and see if this connects to the MoBIE plugin as expected.