networked-aframe
networked-aframe copied to clipboard
Add networked hand tracking component
Hi, thanks for the networked component. We're adding it to the XR# CommunityToolkit. So multi-user experience would quickly be available for C# developers.
It would be really nice to have hand-tracking shared between users.
Here is a simple component that adds networked
attribute for every hand joint. Only modelStyle: dots
is supported.
Please check example project: https://glitch.com/edit/#!/naf-hand-tracking
Related issue #394
@avdynut do you have a live Glitch to test?
Using the dots model isn't ideal and yes networking it all this way is too costly. Yet, I'd argue at this point something inefficient yet functional with known limits, e.g 2 participants only, is better than nothing, as is currently the case.
I'd suggest to accept the PR with minimal change and clear documentation and keep the improved version (hand model and Float32Array) for a subsequent PR.
How about the following considerations to make it easier/lighter to network so much data:
- Quantize the data
- Minimize which joints you network
- Only send update if movement exceeds a threshold
- Differential/delta encoding
Then perhaps once per second or two, send a full "keyframe" without the above optimizations to catch for any glitches from 2, 3 or 4.
Additionally, consider an adaptive approach in which the extent of optimization depends on the amount of hands being tracked. So a 2 person event will track every joint with full accuracy etc, while a 50 person event will be closer to just tracking spheres...