Exposing skeleton finger data.

[2-3-5-41]

trafficstars

The data in question:

• Finger curl
• Raw skeleton bone positions and rotations

I'd like to be able to reference finger curl, or raw data from the SteamVR skeletal system to be applied to an IK Avatar.

[fire]

@vilhalmer is this something we can do with the new openvr?

[vilhalmer]

Yeah, there's a TODO about grabbing skeleton data. I've not looked into what is needed to do it, but I will eventually want access to this data as well. Someone is welcome to pick that up, I don't think it will conflict with any other works.

[fire]

Do you have any pointers where?

[vilhalmer]

I haven't checked to see what state the existing skeleton code is in, but this seems to be where it needs to be fed in: https://github.com/GodotVR/godot_openvr/blob/b1aea64ac07732d6f73910d871b4d3af66965850/src/open_vr/openvr_data.cpp#L427

[fire]

There's two primary data sources for openvr skeletons according to Google Gemini search (Could be wrong so need to verify).

https://github.com/ValveSoftware/openvr/wiki/SteamVR-Skeletal-Input

1. Raw Bone Transforms:

   • The function vr::VRInput()->GetSkeletalBoneData() is used to retrieve an array of vr::VRBoneTransform_t structures, which contain the position and orientation for each bone.
   • The VRBoneTransform_t structure is defined in the OpenVR headers (like openvr_driver.h) and typically contains a position vector (e.g., HmdVector4_t) and an orientation quaternion (e.g., HmdQuaternionf_t).
   • This data can be requested in different coordinate spaces (EVRSkeletalTransformSpace) and motion ranges (EVRSkeletalMotionRange).

2. Summary Data (Finger Curls and Splays):

   • The function vr::VRInput()->GetSkeletalSummaryData() retrieves a VRSkeletalSummaryData_t structure.
   • This structure contains arrays of float values (typically 0.0 to 1.0) representing the curl of each finger (flFingerCurl) and the splay between adjacent fingers (flFingerSplay).

[vilhalmer]

Wouldn't trust Gemini's advice as far as I can throw their data center, given that the wiki it's sourcing from is barely ever even up to date. :)

[fire]

1. https://github.com/ValveSoftware/openvr/blob/ae46a8dd0172580648c8922658a100439115d3eb/headers/openvr.h#L5211
2. https://github.com/ValveSoftware/openvr/blob/ae46a8dd0172580648c8922658a100439115d3eb/headers/openvr.h#L4944
3. https://github.com/ValveSoftware/openvr/blob/ae46a8dd0172580648c8922658a100439115d3eb/headers/openvr.h#L4944

All the data structures and methods mentioned seem to be correct.

[vilhalmer]

Are you planning to pick this up? If so I'll stay out of that area of the code as much as I can.

[fire]

I can't promise anything, but if anyone provides a pull request we can review.

[fire]

My plan is to do the raw bone transforms mapping them to the Godot Humanoid via https://docs.godotengine.org/en/latest/tutorials/xr/openxr_hand_tracking.html.

Will be postponing Finger Curls and Splays until someone wants it.

[fire]

@Malcolmnixon did I miss any approaches?

[Malcolmnixon]

@Malcolmnixon did I miss any approaches?

I don't believe so. My https://github.com/Malcolmnixon/GodotXRHandPoseDetector project subscribes to the https://docs.godotengine.org/en/stable/classes/class_xrhandtracker.html#class-xrhandtracker events, as it provides easy access to the joint data. From there it calculates the spread and curl information.

[fire]

@Malcolmnixon Does the entire skeleton need to be remapped to Godot Engine Humanoid or can I leave it raw?

[Malcolmnixon]

@Malcolmnixon Does the entire skeleton need to be remapped to Godot Engine Humanoid or can I leave it raw?

It doesn't need a skeleton. The XRHandTracker data (tracking data from the XR subsystem) contains joint data you can read directly, however if you really want a skeleton, you can use an XRHandModifier3D to drive a Godot-Humanoid skeleton from the tracking data, then harvest what you want from the skeleton instead - although that would be slower.

[fire]

I am implementing the openvr skeleton source. Huh?

[Malcolmnixon]

Ah sorry, trying to read everything over the phone. So yes your strategy of converting the vr::VRBoneTransform_t values into Godot Humanoid and feeding the result into an XRHandTracker sounds correct - the data would then be usable by XRHandModifier3D, the old OpenXRHand, and any scripts like my HandPoseDetector plugin.

[vilhalmer]

I rearranged how poses are collected in #177, the skeleton actions can probably follow the pattern of always living under /actions/godot.