Extend affinder to 3D

[marbre2]

Hello @jni I recently installed Napari in the context of the NEUBIAS Pasteur course (30/05-02/06/2023). We got a very nice introduction and tutorial on creating our own Napari plugins by Robert Haase and Kevin Yamauchi. In my case, I am really interested in using Napari for a project about 3D microscopy image registration. I discovered your plugin "affinder" and like it a lot. However, as far as I understand, so far it is only available for 2D, while I would definitely be interested in a 3D version. Thus, I have the following questions:

• Do you know if there is another equivalent plugin that works in 3D?
• Is there any plan to extend affinder to the 3D? By you or other contributors?
• Do you know if someone is by chance already working on it? In case of NO to all three questions above, I might consider to work on it with another colleague of mine. Do you have any guidelines and recommandations ?

Thank you very much by advance for your reply.

Best regards,

[jni]

Hello @marbre2!

Thanks for opening this issue!

So, in theory, it should actually already work in 3D: all of the alignment and callback code makes no assumptions about the dimensionality of the underlying images. The three issues are that:

• adding correspondence points in 3D is tricky / doesn't work at all; we could potentially do it with a similar method to the 3D filament annotator: https://twitter.com/anna_medyukhina/status/1578114418805800976 ie, click once to create a "ray", rotate the view, click again to select a point within the ray. I'd be super excited to help with that.
• napari doesn't handle non-orthogonal transformations in 3D when displaying 2D slices. That is, if you e.g. rotate a volume 45º along the xz plane, when you look at xy napari will show orthogonal planes to z. Therefore, if you are using the 2D view (which is the only view where you can currently add points), the images won't be aligned correctly, which is kinda sorta the whole point with affinder. 😂
• The other whole point with affinder 😂 is to set the view to center on the predicted location of your next click, once you have an initial alignment. I think that probably doesn't work in 3D view; in 2D view, which as we said above is flawed, it could be made to work, but we should also use dims.set_point in addition to setting the viewbox.

So, in summary, those are the challenges, but the alignment stuff should basically work; contributions would be super super welcome and I would be happy to help, give you a tour of the code if anything's unclear (😬) (omg just saw that the main function doesn't even have a docstring yet 😂 😭), etc!

PS: You might also be interested in @thanushipeiris's in-progress PR to allow mixed-dimension registration, #58.

[marbre2]

Hello @jni, First I'd like to thank you for your kind and detailed reply. So far I just wanted to give you a feedback on the fact that I managed to use affinder with my 3D images in the 3D view mode (MIP). I first added landmark "points" in the dorsal plane, then a few others in transverse and median plane.

This gave a first rough alignment. I will check on the details of your answer to see how I could contribute to this plugin.

Best regards, Marine