networked-aframe icon indicating copy to clipboard operation
networked-aframe copied to clipboard

Add networked hand tracking component

Open avdynut opened this issue 1 year ago • 2 comments

Hi, thanks for the networked component. We're adding it to the XR# CommunityToolkit. So multi-user experience would quickly be available for C# developers.

It would be really nice to have hand-tracking shared between users. Here is a simple component that adds networked attribute for every hand joint. Only modelStyle: dots is supported.

Please check example project: https://glitch.com/edit/#!/naf-hand-tracking

Related issue #394

avdynut avatar Dec 14 '23 14:12 avdynut

@avdynut do you have a live Glitch to test?

Using the dots model isn't ideal and yes networking it all this way is too costly. Yet, I'd argue at this point something inefficient yet functional with known limits, e.g 2 participants only, is better than nothing, as is currently the case.

I'd suggest to accept the PR with minimal change and clear documentation and keep the improved version (hand model and Float32Array) for a subsequent PR.

Utopiah avatar Dec 17 '23 19:12 Utopiah

How about the following considerations to make it easier/lighter to network so much data:

  1. Quantize the data
  2. Minimize which joints you network
  3. Only send update if movement exceeds a threshold
  4. Differential/delta encoding

Then perhaps once per second or two, send a full "keyframe" without the above optimizations to catch for any glitches from 2, 3 or 4.

Additionally, consider an adaptive approach in which the extent of optimization depends on the amount of hands being tracked. So a 2 person event will track every joint with full accuracy etc, while a 50 person event will be closer to just tracking spheres...

MusicStudioNYC avatar Mar 10 '24 10:03 MusicStudioNYC