Improve interaction handles

[lassoan]

Summary

The new interaction handles are awesome. Here are a few tweaks that would make them even better:

• [x] "Interaction in 3D view" subject hierarchy context menu item was added because the interaction was only possible in 3D, but now that we have it in 2D views as well, the menu item should be just "Interaction" (and turn on/off overall visibility) (https://github.com/Slicer/Slicer/pull/7586)
• [x] In Transforms module / Display / Interaction: add "Visibility in 3D view" option (https://github.com/Slicer/Slicer/pull/7586)
• [ ] In Transforms module if "Enable translation" is off then I cannot move (cannot even see) the center of rotation anymore. If translation is disabled but rotation is enabled, then left-click-drag should move the center of rotation (with or without pressing Alt)
• [x] Left-click on the rotation center should jump slice views (same as when left-clicking on a markup control point) to make it easier to position the centerpoint in 3D (#7595)
• [x] If "Enable translation" checkbox is unchecked then disable corresponding x, y, z, in-plane checkboxes to make it more clear that those checkboxes depend on the main "enable" checkbox. Same for rotation and scaling. (https://github.com/Slicer/Slicer/pull/7586)
• [x] Center of rotation position should be displayed in Transforms module / Information section. (#7599)
• [x] In slice views, I find in-plane rotation very easy to understand, but rotation around parent coordinate system axes is hard to grasp. Therefore, in slice views, default only in-plane rotation handle should be displayed. (https://github.com/Slicer/Slicer/pull/7586)
• [ ] Cannot drag through a center of rotation handle through the another transform's in-plane rotation circle (the handle stops moving moving as soon as the mouse pointer reaches the circle-shaped in-plane rotation handle):

• [ ] Upload a short tutorial video to the PerkLab youtube channel
• [ ] Add some information to the Slicer documentation
• [ ] Write a new topic in Support category on Slicer Forum (add "feature" tag)
• [ ] Plane "center of rotation" cannot be dragged using alt-click-and-drag
• [ ] Make it easier to move a center (markups and transforms) to the current mouse position (current slice) - maybe right-click menu?
  • [ ] Also for 3D/Slice views to move the center of rotation to the current mouse position.
  • [ ] Include some sort of visualization to differentiate when a center of rotation is not on the current Slice.
• [x] Make it possible quit an interaction by hitting escape (save the initial state) (#7600)
• [ ] Make it easier to re-center the plane origin on the points (right-click menu)
• [x] Displacement field transform handles are visible but cannot be interacted with (maybe when loading from scene) (https://github.com/Slicer/Slicer/pull/7586)
• [x] Add option to reset center of rotation to different points (center of transformed node(s), local/world origin, etc.) (#7599)
• [x] When interaction handles are disabled, changing layouts can cause them to re-appear (https://github.com/Slicer/Slicer/issues/7570#issuecomment-1948249304) (https://github.com/Slicer/Slicer/pull/7590)
• [ ] When handles are hidden they can occlude other handles. The other handles become transparent where they would overlap with the rotation handles.
• [x] The interaction handle size slider (interactionHandleScaleSlider) in Markups does not any effect now (it should call SetInteractionScalePercent) (#7599)
• [x] No slider for controlling widget size in the Transforms module (#7599)
• [ ] Make the handle size adjustable for transforms
• [x] Allow interaction handles to be rendered in VR (https://github.com/Slicer/Slicer/pull/7630)
• [ ] Allow users to grab interaction handles in VR

Environment

• Slicer version: Slicer-5.7.0 2024-02-02
• Operating system: Windows

[Sunderlandkyl]

In slice views, I find in-plane rotation very easy to understand, but rotation around parent coordinate system axes is hard to grasp. Therefore, in slice views, default only in-plane rotation handle should be displayed.

Does this apply to translation as well? Out of plane translation is also often difficult to understand in slice views.

[lassoan]

Good point, yes, it applies to the translation handles as well. I find the white dot in the middle much easier to use.

All these do not mean that translation and rotation around parent coordinate system axes is not needed, it is actually very important (e.g., for device placement, to translate and rotate around device axes), it is just a different use case.

[Sunderlandkyl]

We currently use Red/Green/Blue for the axes colors since that is what is used in other 3D software, but @pieper brought up an interesting suggestion during the project week.

What if we used Yellow/Green/Red for the default axis colors to match the Coronal/Sagittal/Axial views?

[Punzo]

@Sunderlandkyl I have a linear transform applied to a volume, and if I activate the interaction handles, the center of transformation does not correspond to the center of the volume (I noticed that the center can be changed also interactively, alt + left click). Would it make sense to calculate the default center on the boundingbox of all the nodes for which the transform is applied?

[Punzo]

Compiling on my ubuntu: Ubuntu 23.10 gcc (Ubuntu 13.2.0-4ubuntu3) 13.2.0 cmake version 3.28.3 Qt 5.15.2 Debug mode.

I get a lot of OpenGL errors from the handles. OpenGLerrors.odt

I don't get the errors in this other setup: Ubuntu 20.04.6 LTS gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 cmake version 3.16.3 Qt 5.15.2 Release mode.

have you experienced any similar issue?

[pieper]

calculate the default center on the boundingbox of all the nodes for which the transform is applied

Since transforms can be applied to multiple nodes this may not be well defined. Instead, maybe there could be a context menu that lists a set of possible options for centers of rotation, such as the centers of any nodes to which the transform is applied (or not), the center of all transforms, the world space origin, control points of any markups, centers of markups, etc. Potentially the center could interactively snap to these candidates as it is dragged, like a lot of drawing tools do these days.

[Sunderlandkyl]

Compiling on my ubuntu: Ubuntu 23.10 gcc (Ubuntu 13.2.0-4ubuntu3) 13.2.0 cmake version 3.28.3 Qt 5.15.2 Debug mode.

I get a lot of OpenGL errors from the handles. OpenGLerrors.odt

I don't get the errors in this other setup: Ubuntu 20.04.6 LTS gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 cmake version 3.16.3 Qt 5.15.2 Release mode.

have you experienced any similar issue?

No, I haven't seen this issue before. Do you think it's related to debug vs release on Linux?

@Sunderlandkyl I have a linear transform applied to a volume, and if I activate the interaction handles, the center of transformation does not correspond to the center of the volume (I noticed that the center can be changed also interactively, alt + left click). Would it make sense to calculate the default center on the boundingbox of all the nodes for which the transform is applied?

This is not done automatically for a pre-existing transform nodes, however if you right-click on an untransformed node in the SH and enabling interaction the new transform should have its center of rotation adjusted to match the center of the transformed node's bounds.

Since transforms can be applied to multiple nodes this may not be well defined. Instead, maybe there could be a context menu that lists a set of possible options for centers of rotation, such as the centers of any nodes to which the transform is applied (or not), the center of all transforms, the world space origin, control points of any markups, centers of markups, etc.

This should be relatively easy to implement. I will add it to the list up above.

[Punzo]

No, I haven't seen this issue before. Do you think it's related to debug vs release on Linux?

I hadn't too much time to investigate. I will try to have soon both a debug and release buils on ubuntu 23.10 and I will let you know.

This is not done automatically for a pre-existing transform nodes, however if you right-click on an untransformed node in the SH and enabling interaction the new transform should have its center of rotation adjusted to match the center of the transformed node's bounds.

not sure I follow. I don't see the option in the right click menu for the volume. I am on slicer5.7.0-2024-02-13

This should be relatively easy to implement. I will add it to the list up above.

nice!

[Sunderlandkyl]

not sure I follow. I don't see the option in the right click menu for the volume. I am on slicer5.7.0-2024-02-13

Sorry, right clicking in the visibility column. Should be "Interaction in 3D view", but it will be re-named "Interaction" in https://github.com/Slicer/Slicer/pull/7586.

[Punzo]

I hadn't too much time to investigate. I will try to have soon both a debug and release buils on ubuntu 23.10 and I will let you know.

I confirm the OpenGL errors show up only in debug mode. tested:

Ubuntu 23.10 gcc (Ubuntu 13.2.0-4ubuntu3) 13.2.0 cmake version 3.28.3 Qt 5.15.2 both Release and Debug mode.

[lassoan]

I saw a lot of OpenGL errors on Windows, too, in debug mode (VTK does not do OpenGL checks in Release mode). Here are the messages: https://pastebin.com/Ud8DbkaZ

It may be related to the VTK fixes that had to be done for shadow support, which changes how the Z and color buffers are cleared.

[Punzo]

I have also experienced that sometimes the rendering of the interaction handles is bugged on the 2D views (probably because of the OpenGL errors). For example the rendering of the handles is visible in the 2D views even if interactions are disabled and viceversa (rendering not showing if interactions are on). The behaviour is very random and very difficult to reproduce.

This at least on my linux build in debug. I will test the binaries from 3D slicer website and I will let you know

[Punzo]

I have also experienced that sometimes the rendering of the interaction is bugged on the 2D views (probably because of the OpenGL errors). For example the rendering of the handles is visible in the 2D views even if interactions are disabled and viceversa (rendering not showing if interactions are on). The behaviour is very random and very difficult to reproduce.

This at least on my linux build in debug, but sometimes also in release. I will test the binaries from 3D slicer website and I will let you know

in release mode from the slicer website does not seems to happen.

I could only reproduce this (not a big issue, just wanted to report it):

1. you load a volume
2. you activate the interaction 3D from the data module (visibility column)
3. activate also the interaction in the 2D view from the transform module
4. deactivate all the interactions
5. switch to layout with new slice views (3x3 for example)
6. you will see the handles rendered (but not interactive) in the new slices.

[Punzo]

in release mode from the slicer website does not seems to happen.

but they do happen both in release and debug mode on this build

Ubuntu 23.10 gcc (Ubuntu 13.2.0-4ubuntu3) 13.2.0 cmake version 3.28.3 Qt 5.15.2 commit: https://github.com/Slicer/Slicer/commit/1b247c26402825047cacf3235e90a0ee81520ee3

[Sunderlandkyl]

I could only reproduce this (not a big issue, just wanted to report it):

• you load a volume
• you activate the interaction 3D from the data module (visibility column)
• activate also the interaction in the 2D view from the transform module
• deactivate all the interactions
• switch to layout with new slice views (3x3 for example)
• you will see the handles rendered (but not interactive) in the new slices.

I've refactored the displayable manager. I think this issue should be fixed by: https://github.com/Slicer/Slicer/pull/7590.

[cpinter]

@Sunderlandkyl @lassoan I managed to try this in my application that relies on such interactions. I'm putting my comments here and then we can move the actionables above to the list when there is a consensus.

• [ ] The interaction handle size slider (interactionHandleScaleSlider) in Markups does not any effect now (it should call SetInteractionScalePercent)
• [ ] There is no such slider in the Transforms module
• [ ] Is it possible to change the center of interaction?
• [ ] I saw that it was possible to change from "3D style" rotation to "2D style" (only a white circle on the slice). I haven't found the function, however. Is it available?

Thank you!

[Sunderlandkyl]

The interaction handle size slider (interactionHandleScaleSlider) in Markups does not any effect now (it should call SetInteractionScalePercent)

Good catch, looks like the current slider is linked to glyph size instead.

There is no such slider in the Transforms module

True, I think it makes sense to add one to Transforms as well.

Is it possible to change the center of interaction?

• Alt + Left-click on a translation handle will move the center of interaction.
• Through code it can be set with vtkMRMLTransformNode::CenterOfTransformation.
• There is a plan to add the center of transformation to the Transforms module (currently listed in the issue above), where it will be editable.

I saw that it was possible to change from "3D style" rotation to "2D style" (only a white circle on the slice). I haven't found the function, however. Is it available?

Since last Thursday (https://github.com/Slicer/Slicer/commit/b2be7c3d3dad73e2501780d4c355f900bc8e809d) interactions can be controlled separately on 3D and 2D views:

Default behavior is that only 2D interactions are shown in 2D views:

In C++/Python The visibility of individual components are controlled through: vtkMRMLTransformDisplayNode::(Rotation/Scale/Translation)HandleComponentVisibility(3D/Slice).

[cpinter]

Hi Kyle, thanks a lot for the exhaustive answer! I'll update again check out the new slice rotation option then. I guess we can add the first two items to the long list on top. Those are quite small issues anyway. Thank you very much for working on this, this is an awesome feature!