elite-vr-cockpit
elite-vr-cockpit copied to clipboard
dantman
[FEATURE] Valve Index Camera Hand Tracking + Joystick
See this post: https://www.reddit.com/r/ValveIndex/comments/itte5o/apparently_the_index_already_supports_finger/
So apparently HTC wrote some camera hand tracking code that also works with the Valve Index cameras.
Feature: Use physical joystick to fly and hand tracking to interact with holo buttons and throttle.
I'm aware of it, it's #97. I heard its tracking wasn't that great. Though that was mostly reports about the Vive.
However HTC puts the software behind an extremely restrictive license agreement which I just noped out of when I read it. You have to go to their site and agree to the license before you can even download the SDK. "Distributing" the SDK outside of this (i.e. including it in the codebase so EVRC just works when you checkout the project and open the Unity Editor) is prohibited. Even inclusion of the sample code is pretty restricted. Apparently they don't even offer SDK source code and "attempt[ing] to access source code of the SDK" or "work around any technical limitations in the SDK" are prohibited. And of course there is no way to automatically install the SDK (like Unity's package manager, which I use to install some of the proprietary libraries I can't just bundle with the codebase).
Basically if I ever included the SDK I would have to specifically filter it out of the codebase. And anyone wanting to contribute to EVRC would have to go to the HTC website, agree to their terms, and manually install the SDK themselves just to run EVRC in the editor to contribute to it.
Maybe I might be able to find a way to dynamically include it so that EVRC can run without hand tracking when the SDK is not installed. However that is not guaranteed. So on the priority list trying to play with HTC's SDK is dead last (Skeletal Input for Kuckles and any device willing to just make use of the open built-in API; then LeapMotion; then the Vive Hand Tracking SDK).
I'm surprised Steam VR doesn't have their own version of this. I guess Valve hasn't seen value in camera hand tracking. They don't even seem interested in their own cameras on their Valve Index. Maybe someone will write an open one for OpenXR someday.
I checked out the demos in the HTC SDK, they are pretty bad.
The hand tracking is very laggy, like as if it's super smoothed. The hand lags behind your real hand, enough that it'll make for a perceptible delay while trying to press a button.
Also the two most important gestures "pointing forward with just your index finger" (which I presume most people will use for pressing buttons) and "grabbing something between your thumb and index finger with your palm facing you" (which I presume most people will use for moving buttons) are unstable. i.e. the hand flickers in and out of view indicating that the SDK cannot reliably track the hand in these poses.
The accuracy of the finger pose is also really bad when you put your thumb and index finger together. The tips don't end up where they should and move around while your fingers are pressed together not moving. So in order to use this as a pinch gesture I would have to make the detection very loose.
Here's a recording of my testing.
- Holding open hands still
- Trying to press something with my index fingers really fast
- Holding my hands in a pinch pose
https://youtu.be/IknAk3Jg9W0
That's interesting, thanks! Just curious, which headset have you used for testing?
And now there is the Quest 2 Tho ... I know it's behind a Facebook wall of WTFuckery, b ut it is still a 300$ headset that SOLD HARD for christmas ... and the hand tracking is VERY decent !
And now there is the Quest 2 Tho ... I know it's behind a Facebook wall of WTFuckery, b ut it is still a 300$ headset that SOLD HARD for christmas ... and the hand tracking is VERY decent !
Oculus Quest's hand tracking is locked exclusively to the QUEST SDK. Even if I went to all the trouble to buy a Quest (a scary idea as a software developer with apps and pages on Facebook, given the reports of some people having their perfectly fine Facebook accounts banned soon after connecting it to a Quest). Even if I went to the trouble of reimplementing the entire input and overlay handling layers to support the Oculus SDK. Even if I assumed that it was possible to develop overlays for Oculus devices using the Oculus SDK (it's not something they officially support, there are some hints in the API that "maybe" it can be done, but no guarantees). It's ultimately still not possible, because the Oculus PC SDK doesn't have access to hand tracking info from the Quest to my knowledge. Which of course means it's impossible for EVRC to use because Elite Dangerous is not available on the standalone Quest, only via the Oculus PC SDK.