GUI element to set layer transforms (scale/ translate)

[sofroniewn]

🚀 Feature

Layer transforms - in particular scale and translate should be settable via GUI elements

Motivation

GUI users want to be able to set layer scale and translate too.

Pitch

There should be some dialog, combobox/ textbox? that allows the properties of the layer transforms to be set. these include scale and translate, but could also include rotation, shear, or a full affine

Alternatives

These are only settable via the API, which limits GUI users

Additional context

Curious which layer transform properties need to be exposed? Is it just scale/ translate - or also rotation, shear, affine etc. What do @jni @andy-sweet @JoOkuma and others think?

Maybe this is something @isabela-pf can begin some design work for and then @ppwadhwa could implement when we have a design

[sofroniewn]

from @tlambert03 in zulip

my first thought of a place for it is:

1. a new item in the layerlist context menu. right-click and it brings up a widget to modify the affine info
2. an as-of-yet-uncreated Layer menu, that would provide similar access to all of the stuff in the context menu. (let's start with the context menu maybe, will be easy to add there later)

[JoOkuma]

I'm in favor of exposing only scale, translation, and maybe rotation. From my experience shear and affine are more complex and are usually computed programmatically.

[sofroniewn]

Yeah, I'll flag I think the most challenging aspect of this design will be how to handle the "nD" data case. Do we want one combobox per axis? Can we use a simple QForm? Does the ideal solution for dim < 5 look different then ideal dim >= 5 (h/t @tlambert03). Just some questions that aren't clear to me

[isabela-pf]

Thanks for the @! I'm totally unfamiliar with this part of napari, so I've been reading the docs to get a better sense of layers and these transformations. I first have some questions to make sure I'm working in the right direction.

1. In the pitch, @sofroniewn mentioned that

There should be some dialog, combobox/ textbox?

Is there a reason this needs to be outside of the layer controls (maybe the "nD" case you mentioned above where it could make the layer controls very long)? My early thoughts are that a combobox in the layer controls might be a good fit.

2. Do these parameters exist with some kind of null value before a user decides to transform it, or does the parameter get added to a layer once a user chooses so? I think this might determine whether this UI is persistent or if it only appears once it belongs to the layer?
3. Scale and rotate seem to have operations per dimension of the image. Does translation operate the same? (If so, it isn't mentioned in the layers API reference.)
4. Translated layers are translated relative to other layers (as mentioned in the image layer tutorial). Is there a unit that's important to these relative distances?

---

P.S. if anyone wants to show off their super cool layer transformations or something I'd be happy to schedule time to watch how people use this in the wild.

[andy-sweet]

2. Do these parameters exist with some kind of null value before a user decides to transform it, or does the parameter get added to a layer once a user chooses so?

They get default identity-like values if they are not specified. E.g. image = Image(np.ones((64, 128)) will create a 2D image layer where image.translate == [0, 0] and image.scale == [1, 1].

3. Scale and rotate seem to have operations per dimension of the image. Does translation operate the same?

Yes, there's a translation value per dimension.

4. Translated layers are translated relative to other layers (as mentioned in the image layer tutorial). Is there a unit that's important to these relative distances?

No units (yet). The translations are actually relative to a fixed origin that all layers share - each layer's translation values kind of specifies an origin for that layer. But you can set those values to describe relative translations between layers.

@isabela-pf : I also highly recommend running napari examples/interaction_box_image.py , which should give a familiar graphical interaction scheme (e.g. powerpoint) for translating, scaling, and rotating a layer (in this case an image layer).

[sofroniewn]

Is there a reason this needs to be outside of the layer controls (maybe the "nD" case you mentioned above where it could make the layer controls very long)? My early thoughts are that a combobox in the layer controls might be a good fit.

Yeah my fear here is that the layer controls will just get too long and crowded. This is also functionality that will work with all of our layer types. We might need a new "transform widget"

[sofroniewn]

@MBPhys has done some cool work on cropping/ adjusting scales/ translates here https://www.napari-hub.org/plugins/napari-nd-cropper we should maybe think about folding this into core or make people aware of this. See the widget on the right. It scales to nD too I think - @MBPhys let me know what you think

[isabela-pf]

In this comment, I have two things I’m proposing UI for:

1. the new UI needed for translate and scale specifically
2. an updated UI for layer controls (to better accommodate an increasing number of layer controls appearing in the viewer)

1. translate and scale UI

Full disclosure, this is the part I’m least confident in. I’ve read what I think are the attributes of each on the API reference. I also referenced the screenshot of @MBPhys’ plugin posted above for translate.

I expect I’ve missed (or misunderstood) something along the way, though, so please let me know what else I should be accounting for.

translate

As far as I can tell, this has two properties (is there a more accurate word for this?) to account for.

First, a translation value per image dimension (three shown here) as a drop down combo box. This means users may manually type in a value, or select from standard values. I felt like this allowed for the same amount of control as the sliders pictured above, but would also show a range of valid values without needing to move a slider to both extremes.

Second is type. Type is a standard drop down. I do not know what options go in here at all; I chose a drop down because the options sounded discrete.

scale

The only value I found for scale was the scale factor. I matched this pattern to an existing UI I found in the default labels layer controls. Like its inspiration, numbers can be typed in manually as well as manipulated to whole numbers with the plus and minus buttons.

2. layer controls more broadly

Quick notes that apply to all layer controls mockups:

• All options maintain the current behavior of operating on the layer selected in the layer list.
• Any options with icons have quick placeholder icons. If we end up wanting an approach to use icons, I’ll actually put time into those then.

Tabs

Perhaps the most familiar option, this adds a list of layer controls as icons to a sidebar. I do expect we’d need some kind of overflow section for this long term. All layer controls available for that layer could appear by default.

https://user-images.githubusercontent.com/50221806/152449354-c941e5cb-51a0-427f-8c0e-8d4445d63cae.mov

Drop down

Instead of making space on the UI for the many layer controls I expect will eventually become available in the viewer, you could switch controls via a drop down. This is similar to the Adobe workspace switcher we discussed at one of the community meetings (example shown from InDesign):

This might scale better in terms of UI real estate since drop downs often have a scroll, or even a search to accommodate long lists.

https://user-images.githubusercontent.com/50221806/152449371-578dcdaf-43ec-4240-85a9-2ff91b24396f.mov

“Add layer control”

This version is a list of layer controls that can be expanded/collapsed. The key change is that not all layer controls are visible in the list by default. Users add layer controls they want to see for that layer by searching and selecting in the top input box.

https://user-images.githubusercontent.com/50221806/152449464-8ccc26f8-4680-41d7-baa0-7b56ff3e345b.mov

This behavior was inspired by Adobe AfterEffects, where all components on screen have a set of (very many) default properties that do not appear on screen until the user chooses to either expand that component or edit that property any way. I think this can reduce the cognitive load of having so many options on a screen by default by prioritizing what the user is actually working with.

All layer controls would still be discoverable because they appear in the input box list.

To avoid users having to add a control one-by-one, we may want to have an add all layer controls option (or something similar).

[isabela-pf]

I got feedback on the above proposal at one of the community meetings. Here's a summary:

Decisions made

• For all layer control options with a numerical value, make them spinboxes
• Use the tabs approach for layer controls!
• Group related layer controls in tabs (instead of having tabs per control)(more on this in the future, as these groups are not yet documented)

Next steps

• Determine which controls/groupings need to be available for each of the seven layer types
• Come up with UX for reordering tabs and popping out the tabs as their own windows
• Provide mockups for more detailed interactions (like the more tab to account for overflow)(self-assigned)
• Possibly have a schema for what types of values correspond to what types of UI a la QForm? (this isn't something I personally have the knowledge to do, but would impact these designs)

[MBPhys]

@sofroniewn

Thanks. This would be cool in core:) Just for clarification There a two plugins:

1. nd-cropper with four modes seeing the attached video

https://user-images.githubusercontent.com/83079938/152886129-6f1e942b-4cca-4ed4-82d3-a6464ffa6ad5.mp4

This works in nd for all modes except the interaction box. Perhaps, we can update this into the nd-space in the future. The nd-cropper works in data-space, but take into account the world representation.

2. Partial-Aligner This plugin is created in order to affine transform images and parts of images just in 2D and 3D. This enables affine registration of images and deformable registration via selecting just parts of images (look at the Readme and the attached videos) For 2D, we have also a Drag&Drop solution seeing attached video Furthermore there are two modes, too: 1. With Slider, 2. Absolute values (Spinbox) and we record the values for each layer. The user can switch between the two modes very intuitively. We added this, because we noticed and got feedback that a slider itself is too difficult to control for only very small translations/scales/... for instance

https://user-images.githubusercontent.com/83079938/152887745-edbe4db6-3aab-4108-b875-0257fe4e1c3b.mp4

https://user-images.githubusercontent.com/83079938/152887750-3dedbc65-1607-43a1-bf9e-cd892bea6902.mp4

All this happens in the world space. In order to convert the world to the data space, I wrote the World2Data plugin.

I think napari makes it very good to push unique the data representation with the metadata to a world space (where image registrations lives, too) and separate the underlying data into a data space. Nonetheless my expression was in the past that many peoples ( in particular GUI users), who are not so familiar with this concept, are very confused unfortunately... Therefore I think, it is a very good idea @sofroniewn to make people aware of this and show the people the advantages of this. Do I understand this here correctly @sofroniewn?

Altogether this means just to notice that the Partial-Aligner do not fit to the nd-space, since shear/rotations will be very complex in nd and was not necessary for our and other real-life applications.

However I think that there are no problems to just extend the translation/scale to nd. In my opinion, there should be two modes for the users (SpinBox and Sliders) selecting by own preferences and their related tasks

@isabela-pf I love your solution and layout :smiley: Very beautiful. To add a mode with a slider should not be very complicated, or? What do you think about? Is this an opportunity?

[isabela-pf]

To add a mode with a slider should not be very complicated, or?

I think a slider works fine. I think you'd know better than I what the most useful interaction is, so I'll defer to your advice. Thanks for asking @MBPhys!

What I showed was me thinking that sliders seem like they don't allow for quite the degree of fine-tuning that some of the other options have, but based on how many are currently in napari viewer and your plugin, I think it's the best option here.

[ppwadhwa]

Hi! From what I see above, it looks like we want tabs in place on the layer control panel, so I was going to start getting those in place with the translate and scale options added. If anyone has any objections to that, please let me know! thanks. :)

[isabela-pf]

@ppwadhwa that sounds correct to me! I have some time freed up from bundled app issues, so I will put it on my list to make some more relevant mockups incorporating the latest feedback. I think this will make will make it clear how we want scale and transform to be represented specifically.

[dalthviz]

Hi there, I gave this a check and started exploring adding a slider/spinbox per value in the scale attribute over the layer controls. The controls could look something like the following:

• Using QDoubleSpinBox:

• Using QLabeledDoubleSlider:

Where the label before the spinbox/slider widget marks the dimension the slider/spinbox affects.

Also, my guess is that a similar approach can be used to create controls for the translate property?

Edit - Implementing a popup widget for the scale following the contrast limits control functionality could look something like this:

Edit 2: A branch with the described controls (using sliders) - https://github.com/napari/napari/compare/main...dalthviz:napari:scale_controls

[dalthviz]

Hi there, thinking again about this, maybe adding a button for the Transform mode could help too? I think the mode is already accesible via keybindings so maybe having it as an actual button could be useful? For example, with an image layer which transform mode is activated pressing 2, you could have something like:

[Czaki]

Using API you could easily reset view. Interface should also allow reset transforms.

[psobolewskiPhD]

I like the idea of a way to activate transform mode from the GUI--I bet a lot of people don't know it exists. Not sure every layer needs its own button vs. a viewer button at the bottom? 🤔 Not sure there's room really. But maybe in the layer controls is better, since most of the other layers already have the double-arrow for Pan/Zoom...

[dalthviz]

Using API you could easily reset view. Interface should also allow reset transforms.

Maybe having then the button but also a control? So, using as an example the scale attribute from the affine transform:

Not sure every layer needs its own button vs. a viewer button at the bottom? 🤔

Since the button activates a mode and also the given shortcut is the last number in the sequence from the modes a layer offers, probably makes sense to put it over the layer controls around the buttons for changing modes. For example, over a Shapes layer (where the transform mode is activated with 7 via keyboard) you could see something like:

Also, seems like the transform mode uses the layer affine attribute (physical2world transform) while the individual layer attributes like scale, translate, etc use the data2physical transform. I don't really know the difference between using one or the other 😅 but probably something to check

[psobolewskiPhD]

Yeah, thinking further it's per layer so the layer controls makes the most sense.

I can't say I love the scale sliders in the layer controls -- feel like something that should be in its own widget and more thought as to how to expose? I think the first step would be a button to activate the transform mode. And then some way to reset the transforms -- maybe Alt-click the butto -- with a confirmation dialog? 😬

[dalthviz]

I can't say I love the scale sliders in the layer controls -- feel like something that should be in its own widget and more thought as to how to expose?

Something to further think about for sure :+1:

I think the first step would be a button to activate the transform mode.

Makes sense to me :+1:

And then some way to reset the transforms -- maybe Alt-click the butto -- with a confirmation dialog? 😬

I thought about a similar idea, although more like a right-click or something that shows the pop up with the reset option. So something like:

Will try to explore the alt-click idea too :+1:

[andy-sweet]

Also, seems like the transform mode uses the layer affine attribute (physical2world transform) while the individual layer attributes like scale, translate, etc use the data2physical transform. I don't really know the difference between using one or the other 😅 but probably something to check

The intent was to separate out the transforms associated with the pixel/sample space (data2physical, similar to ITK's spacing) and then a transform applied on top of that (physical2world) which might align on image with another.

In the long term, I think napari could benefit from a more flexible approach allowing any chain of transforms (including 0 or 1 transforms), but that asks more complicated design questions too.

[jni]

yeah, one thing I will say is that these shouldn't be "always on" in the layer controls, because e.g. you could have a 5D image so that's a lot of vertical space when you account for scale, translate, and rotate.

This could be something that uses @andy-sweet's metadata plugin and is included in builtins, perhaps — that is, it opens in a new side panel rather than be part of the standard controls. Either that or it's a collapsible section, I guess. Either way, it's going to be tricky to make the layout look good and be flexible for any number of dimensions...

[dalthviz]

Just in case, some further exploration of the transform mode button idea using an alt-click/right-click to show a way to reset the transform. Checked with a popup and a dialog showing both a similar set of reset buttons (for scale, rotate and translate properties as well as a full reset):

• Popup

• Dialog

[psobolewskiPhD]

I like the contextual style menu a lot better than the floating dialog. I think that looks pretty good. I think if it's to make a menu, it should be a right-click. This is just the most intuitive thing for a contextual menu. The other option is a simple option-click the button to reset everything -- boom!

Either way I think the first pass should be to expose the layer transform mode in the layer tools (analogous to the keybinds being in each layer) and then we can think about enabling more granular widgets that let you use a slider or such to control the rotation, vs. just rotating stuff.

I do think this is pretty high value stuff for people working with multiple modalities of the same "stuff".

[dalthviz]

The other option is a simple option-click the button to reset everything -- boom!

Oh I think I now understand the dialog idea you had! So showing after and alt-click something like:

right?

Either way I think the first pass should be to expose the layer transform mode in the layer tools (analogous to the keybinds being in each layer) and then we can think about enabling more granular widgets that let you use a slider or such to control the rotation, vs. just rotating stuff.

I do think this is pretty high value stuff for people working with multiple modalities of the same "stuff".

Will try opening a PR adding the transform mode button then :+1: Not totally sure if there are any other ideas for the reset behavior besides the alt-click with dialog confirmation or righ-click popup with reset buttons or option-click full reset (and probably worthy to note that it's possible to have all of those alternatives at the same time too) so maybe doing some more exploration over that could be worthy

[jni]

Just to clarify, the pan-zoom icon is not the proposed final icon for transform? I find it too overloaded. Having said that I'm not finding a great one. swap? rotate? expand? rotate inside an expand?

[psobolewskiPhD]

Yeah, that's basically it @dalthviz as a simpler alternative. And 💯 with @jni that there needs to be a good icon for transform mode. Here's a couple I saw: https://fontawesome.com/icons/group-arrows-rotate?f=classic&s=solid (and related) the pivot table ones (lol) kinda look like resize and rotate: https://fontawesome.com/icons/table-pivot?f=classic&s=duotone

[dalthviz]

Regarding the icon, I put the current one shown in the previews as a place holder using the available icons and following the names for icons available (so putting for the pan-zoom button the pan_zoom name which make available the pan name), but I think too that a better icon should be defined. Maybe @isabela-pf could suggest icon alternatives and in general give some feedback/new ideas about what has been discussed

[isabela-pf]

Hi there! The most common icon I have seen used for transform features is similar to the one we currently use for adding a rectangle on a shape layer (rectangle with vertices marked). Keeping something that references the handles in the transform controls would be my recommendation, but I think it will be a balance to differentiate it from the existing rectangle icon.

My first proposal is branching from the previously mentioned expand icon, adding the handles back in for transforms, and turning it on its side to reference the rotation element.

And here it is in @dalthviz’s image from above (thank you for that!) to see it in context.

Let me know if this seems like the right idea or not. I’ll export it as a proper SVG when we make a decision.

[psobolewskiPhD]

I like that, it's simple. Being a diamond evokes rotation to me.