Articulated Hand Mesh stuck when hand is occluded

[jloehr]

Describe the bug

There is a Articulated Hand Mesh tracking regression when switching from Windows XR plugin to OpenXR.

In situations where the hand is still tracked, but the hand mesh/joints are not, on OpenXR the hand mesh is stuck at the last tracked position instead of moving with the still tracked wrist.

To reproduce

Steps to reproduce the behavior:

1. Run a scene with the hand mesh visualizer enabled on the device
2. Stretch you arm forward away from you, like you're pointing towards something in the distance.
3. Tilt your hand slightly lower, so it is occluded by your wrist.
4. Move your arm around while keeping the hand occluded.

Expected behavior

Hand Mesh will move with your wrist position.

Screenshots

Issue when using OpenXR (ignore the hand mesh coloring, this build is using linear color space)

https://user-images.githubusercontent.com/4631548/180179997-de3abf3f-1299-4b9e-a4cd-fc4aef863687.mp4

Expected behavopr using Windows XR plugin

https://user-images.githubusercontent.com/4631548/180180059-959527fa-091e-473a-9b9f-17294585ebaf.mp4

Your setup (please complete the following information)

• Unity 2020.3.20f1
• MRTK 2.8.2
• OpenXR

Target platform (please complete the following information)

• HoloLens
• HoloLens 2
• WMR immersive
• (MR) OpenXR 1.4.2

Additional context

I tried to investigate this and come up with a fix. Here are my findings:

The difference between WindowsMixedRealityXRSDKArticulatedHand (Windows XR Plugin) and MicrosoftArticulatedHand (OpenXR) is that the former is calling the handMeshProvider?.UpdateHandMesh outside of the if (inputDevice.TryGetFeatureValue(CommonUsages.handData, out Hand hand)) check, while OpenXR does so inside.

I tried to move the handMeshProvider?.UpdateHandMesh out of the condiditional (with an additional guard that it is only running once per frame), and it did fix the issue. Unfortunately it did came with a huge performance impact.

[Zee2]

I am getting the "desired" behavior on MRTK3 on OpenXR. @keveleigh , this is probably something to do with the whole tracked/valid/etc headache.

[ChrisTJie]

I have another question. Is it normal for the hand model of the editor to appear when the hand is detected?

[IssueSyncBot]

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!