Object Manipulator breaks when Left XR Controller and Left Hand tracking are enabled

[Babilinski]

Describe the bug

I noticed that the Object Manipulator Breaks when when using two controllers with the same handedness values. For example: if I use a Left XR Controller to pick up an object, then highlight the object with my left hand ray, the object will rotate to match the rotation of my hand even though the hand ray did not initialize the grab behavior.

To reproduce

This bug can be reproduced using the Simulator and at runtime on the Magic Leap 1 and 2 since the Magic Leap controller can be used in either hand while hand tracking is active.

To reproduce using the Input Simulation Service Provider:

First open the Hand Interaction example scene and configure the default Mixed Reality Toolkit profile to support two Input Simulation services (Hand and Motion Controller), then interact with an object using both a hand and the motion controller :

1. Open the MRTK Hand Interaction Example scene
2. Select the MixedRealityToolkit in the hierarchy
3. Select Copy & Customize to edit the configuration profile, then Clone
4. Once a copy of the Configuration profile has been made, navigate to it's Input setting
5. Clone the Input System settings
6. Select Add Data Provider
7. Navigate to the New data provider 16 and set it's type to Microsoft.MixedReality.Toolkit.Input > InputSimulationService
8. Clone the Input Simulation Profile
9. Once cloned, set the Default Controller Simulation Mode to Motion Controller
10. Enter Play Mode
11. Use W,A,S,D and your mouse to navigate to the Globe Mode in the Manipulation Interaction section
12. Press Shift to enable both Hand and Motion Controller
13. Carefully select the glob with the ray extending from the simulated hand first, then move the object so it intersects with the ray from the motion controller.
14. Notice that the object rotates as soon as the controller ray intersects it

Video Provided below

Expected behavior

The pose of the object should not be effected by a ray the is not actively interacting with the object. The same behavior that occurs when manipulating an object with your left hand and then highlighting the object with your right hand.

Screenshots

Notice how the object rotates when it intersects with the other ray coming from the left controller/hand after already being grabbed by one of the inputs

https://user-images.githubusercontent.com/10122344/204035488-b8e22b34-da7e-4c3a-8f77-ea822199e392.mp4

Your setup (please complete the following information)

• Unity Version [e.g. 2022.2.0b7]
• MRTK Version [e.g. v2.8.2]

Target platform (please complete the following information)

• Simulator
• Magic Leap

Additional context

I tried to set the XR Controller Handedness to Any but this broke the controller's interactions and I couldn't figure out how to create an additional controller definition for a Generic Unity Controller with this handedness option. Setting the XR Controller Handedness to Both did not change the result.

[Babilinski]

After looking into this issue further I found that the Object Manipulator queries the source rotation with the following logic

        private bool TryGetGripRotation(IMixedRealityPointer pointer, out Quaternion rotation)
        {
            rotation = Quaternion.identity;
            switch (pointer.Controller?.ControllerHandedness)
            {
                case Handedness.Left:
                    rotation = leftHandRotation;
                    break;
                case Handedness.Right:
                    rotation = rightHandRotation;
                    break;
                default:
                    return false;
            }
            return true;
        }
....
       public void OnSourcePoseChanged(SourcePoseEventData<MixedRealityPose> eventData)
        {
            switch (eventData.Controller?.ControllerHandedness)
            {
                case Handedness.Left:
                    leftHandRotation = eventData.SourceData.Rotation;
                    break;
                case Handedness.Right:
                    rightHandRotation = eventData.SourceData.Rotation;
                    break;
                default:
                    break;
            }
        }

Adding additional properties to store the leftControllerRotation and RightControllerRotation and assigning their values based on the Source Type seems to resolve the issue:

       private Quaternion leftControllerRotation;
       private Quaternion rightControllerRotation;
....

        private bool TryGetGripRotation(IMixedRealityPointer pointer, out Quaternion rotation)
        {
            rotation = Quaternion.identity;
            if (pointer.InputSourceParent.SourceType == InputSourceType.Controller)
            {
                switch (pointer.Controller?.ControllerHandedness)
                {
                    case Handedness.Left:
                        rotation = leftControllerRotation;
                        break;
                    case Handedness.Right:
                        rotation = rightControllerRotation;
                        break;
                    default:
                        return false;
                }
            }
            else
            {
                switch (pointer.Controller?.ControllerHandedness)
                {
                    case Handedness.Left:
                        rotation = leftHandRotation;
                        break;
                    case Handedness.Right:
                        rotation = rightHandRotation;
                        break;
                    default:
                        return false;
                }
            }
            return true;
        }
        
....

       public void OnSourcePoseChanged(SourcePoseEventData<MixedRealityPose> eventData)
        {
            if (eventData.InputSource.SourceType == InputSourceType.Controller)
            {
                switch (eventData.Controller?.ControllerHandedness)
                {
                    case Handedness.Left:
                        leftControllerRotation = eventData.SourceData.Rotation;
                        break;
                    case Handedness.Right:
                        rightControllerRotation = eventData.SourceData.Rotation;
                        break;
                    default:
                        break;
                }
            }
            else
            {
                switch (eventData.Controller?.ControllerHandedness)
                {
                    case Handedness.Left:
                        leftHandRotation = eventData.SourceData.Rotation;
                        break;
                    case Handedness.Right:
                        rightHandRotation = eventData.SourceData.Rotation;
                        break;
                    default:
                        break;
                }
            }
        }

[Zee2]

@keveleigh this is a pretty common issue with MRTK2, right? Multiple controllers/pointers with the same handedness causing havoc?

[Babilinski]

The only issue I've seen is with this Object Manipulator script. I've found that setting the controller to handedness any broke a lot of our interactions

[IssueSyncBot]

Hi @Babilinski,

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visithttps://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!