Preserve axis labels within napari layers

[gselzer]

Splitting this issue out from #252

PyImageJ converts ImageJ ecosystem image data structures into xarray.DataArray objects, which have named axis. There is no Layer metadata for named axes in napari, as far as I can tell, although I've seen the desire for the feature in issues like napari/napari#2917. The closest thing I know of is the Viewer.axis_labels property on the Viewer, but we'd have to massage the data to use that to map our image data.

For now, I store the axis within Layer.properties, however a better solution would store them somewhere they can be used in napari.

[ctrueden]

I think the idea was to indeed use Viewer.dims.axis_labels. I forgot it was on Viewer rather than Layer though; that's unfortunate.

Nonetheless, I was thinking we should simply be rude, and update the Viewer every time we add a new Image layer. But it may be only limitedly doable:

we'd have to massage the data to use that to map our image data.

Yeah. But only if there are multiple image layers in the viewer. I did an experiment where I opened four new image layers (without napari-imagej), as follows:

1. dub.ome.tif - (33, 85, 512, 768) - (pln, t, y, x)
2. clown.jpg - (200, 320, 3) - (y, x, ch)
3. lymp.jpg - (130, 130) - (y, x)
4. t1head - (129, 256, 256) - (pln, y, x)

Note that in this scenario, the dimensional axis labels above are only known to us as humans, not to the computer.

I then did:

viewer.dims.axis_labels = ('pln', 't', 'y', 'x')

and napari handled the layers (1) through (3) above properly. When scrubbing the pln and t sliders, the clown and lymp images stay constant (i.e. they exist on all pln and t indices). But the t1head layer (4) is incorrect, because its dimension index 0 is pln, not t. So then I tried:

viewer.layers[3].data = numpy.expand_dims(viewer.layers[3].data, 1)

which changed layer 4 to:

4. t1head - (129, 1, 256, 256) - (pln, t, y, x)

So that it matched dub. With this change, the pln slider now scrubs through the t1head planes as desired, but the data only exists at t index 0: if you scrub the t slider away from 0, the t1head disappears. This makes sense, because the dimension is now explicit and only of length 1. But it makes behavior inconsistent w.r.t. padding vs not-padding.

(Side note: I probably should have swapped dub to (t, pln, y, x), which would align properly with scikit-image's standard dimension order. But the issue I highlight above would then still be problematic, just for 2D time-series instead: (t, y, x) → (t, pln, y, x).)

There are also larger concerns here with how to align the dimensional indices of each axis. That's what axis calibration is meant for: anchoring your data in a global coordinate space. For specifying a layer's location in the global coordinate space, it appears that napari Layers support affine transforms (constructor arguments scale, translate, rotate, shear, or the more general affine). But presumably that only applies to XYZ dimensions, not nD for dimensions beyond 3D like time—it's not clear to me whether napari has an API for orienting such dimensions in the global coordinate space (though I didn't check the source to see whether the transform parameters generalize beyond 3D; maybe they do).

What about numpy? Does it have any coordinate transformation support? I couldn't find anything in a quick search, but it looks like scikit-image has some geometrical transformation support. I don't know whether it actually copies sample values into a new array, or wraps the data as a view, though...

For now, I store the axis within Layer.properties

Until napari has some semblance of support for labeled dimensional axes, and/or spatial coordinate systems, I fear this is the best you can do. I recommend closing this issue as "not planned" until such time.

[gselzer]

What about numpy? Does it have any coordinate transformation support? I couldn't find anything in a quick search, but it looks like scikit-image has some geometrical transformation support. I don't know whether it actually copies sample values into a new array, or wraps the data as a view, though...

Sure, there are things like numpy.transpose, which returns a view whenever possible.

Until napari has some semblance of support for labeled dimensional axes, and/or spatial coordinate systems, I fear this is the best you can do.

Yeah, maybe we should get the opinions of napari folks. @brisvag @andy-sweet has there been any more work on preserving axis metadata within napari as of late? Are we missing something?

[brisvag]

Not work as far as I know, but a lot of desire :P Maybe @andy-sweet did have a small plugin prototype somewhere?

Anyways, it would be great if you could use this as a jumping point to start working this into napari core, and I'd be happy to help out getting started on it :) @jni would likely be intersted as well!

[andy-sweet]

Yeah, no core work that I'm aware of either, though @melonora has also been thinking about this too.

I made https://github.com/andy-sweet/napari-metadata which effectively uses Layer.metadata to store layer specific metadata like axis/dimension labels (likely more appropriate than properties), then rudely updates Viewer.dims.axis_labels mostly as @ctrueden describes above, which may be good enough for simple usage. The other suggestions/workarounds from @ctrueden (e.g. adding singleton dimensions) are the best I can recommend now too.

Layer axis labels (e.g. https://github.com/napari/napari/issues/906) and dimension correspondence (e.g. https://github.com/napari/napari/issues/3848) are long standing issues in napari. If you want to lead an effort to improve them, I'd also be very happy to help review stuff.

[ctrueden]

If you want to lead an effort to improve them, I'd also be very happy to help review stuff.

I would love to work on improving napari core's global coordinate system logic and dimensional metadata support. Unfortunately, the LOCI team will probably not have time to work on it until at least Q3 2024, due to upcoming grant deadlines pulling us back onto the Java side of the ImageJ2 codebase. :disappointed: Now that napari-imagej is released, our next major priority is to complete an initial release of ImageJ Ops2 (currently incubating here), which I expect will take several more months. However, the next priority after that will be better integration of Ops with the Python world, at which point it could be in the cards for us to work on enhancing napari core along these lines. We'll see how things unfold!

[melonora]

I will have to wrap some other projects up first, but yes also regarding OME-NGFF there is a big desire to get it to work in Napari. @andy-sweet and also I did try some things, but at the moment at least from my side I had to prioritize. I would still like to do it (before Q3 2024 haha). napari-spatialdata would also benefit from it so I would take it up as part of the development work for it.

[jni]

Yeah I will just chime in here to say sorry 😅, it's not you it's us. It's just a big lift in napari. But we are currently doing a roadmapping exercise with the napari core and I personally put the transformation model at the top of my list (this includes axis broadcasting, in my mind at least).

@ctrueden this behaviour in particular:

With this change, the pln slider now scrubs through the t1head planes as desired, but the data only exists at t index 0: if you scrub the t slider away from 0, the t1head disappears. This makes sense, because the dimension is now explicit and only of length 1. But it makes behavior inconsistent w.r.t. padding vs not-padding.

has been bugging me for a very very long time. 😤 It was painful to read it.

This is the most thorough record of that issue in napari: https://github.com/napari/napari/issues/3882, though it's worth linking also to the earlier issue: https://github.com/napari/napari/issues/3848#issuecomment-997659490.

I don't know when this will be fixed but it's a priority, and you should subscribe to those issues if you want to be informed of when that happens. 😊

[ctrueden]

@jni Yeah, regarding multiple images in the same viewer, I essentially did the same thing when designing ImageJ2: let's make the Display class have a List<DatasetView>, which each DatasetView wrapping a Dataset (ImageJ2's nD image class) plus visualization settings! And then after initially implementing it, as soon as the length of that list of views became greater than 1, the design challenges began to emerge. :sweat_smile: It was never solved in ImageJ2—we pretty much just assume that every Display has only one Dataset inside—so you have my sympathies and best wishes in achieving a sane design for napari. I personally think a nice and flexible way to go is to use world coordinates + invertible transforms per image layer.