[Proposal - WebXR controllers] Add WebXR controllers interaction

[jogarnier]

This issue is a feature proposal. Feel free to upvote (with :+1: ), comment and provide your use-cases if you're interested by this feature.

Context

(Following the TODOs of https://github.com/iTowns/itowns/pull/2092)

• add VR controllers
• add VR visual tool for debugging

Expected behavior

• Add WebXR interaction with a GIS scene / 3D object / layer.
• The proposal is to cover the public API of WebXR (see details below) :
  • [x] Move via the right joystick
  • [x] Rotate via the left joystick
  • [x] teleportation via the right button
  • [x] Height adjustement via the left button

Proposal

Add a demo implementation of WebXR controllers. The perimeter of this example is exclusive to VR and its bounds are yet to define (see problems part).

Defining Itowns controllers API

Registering and using WebXR controllers can be factored to let Itown create and bind technically the instances. There are 2 types of controllers :

• Hands detection (using ThreeJs XRHandModelFactory)
• controllers (using ThreeJs XRControllerModelFactory)

Tests and PoC has been done on the latter one. Controller binding is not guaranteed to be consistant from a model to another. In this context, it has been mainly tested on a Pico4.

Add more WebXR parameters at instance creation

• The webXR scene is defined using the behavior of local (Refering to https://developer.mozilla.org/en-US/docs/Web/API/XRReferenceSpace#reference_space_types). In this setup we can move freely into the scene as we can move physically at a scale 1:1. -- TODO test bounded-floor could also fit well for small local scene, like around a building.

• Added a callback to the webXR render loop for user rendering needs.

Listening controllers modification

The exhaustive list of events that can be listened from controller :

• https://www.w3.org/TR/webxr/#event-types This covers one trigger button.

For this demo purpose, I added a few more that are fired by itowns WebXR logic and listened by the user example :

• itowns-xr-axes-changed (controller)
• itowns-xr-axes-stop (controller)
• itowns-xr-button-pressed (controller, buttonIndex, button)
• itowns-xr-button-released (controller, buttonIndex, button)

TODO test binding event squeeze

Interactive behaviors are not provided by default, the user has to write its owns.

The example separate two layers of binding :

• The event emitter extension (stored in Itowns WebXR.js for now) is aimed to only listen internal controllers state modification (and adds a few more) to provide a more user-friendly listeners. Code documentation may be tight to the controller type, (see problems #3)
• An Itowns Controller.js file which listens those events to provide a set of interaction behaviors. This part has to be discussed if we want Itowns to provide default behaviors. See incoming screenshot for the current mapping proposed.

Potential Problems

1. Controller position tracking precision, due to the usage of getOffsetReferenceSpace() method to teleport the user in the xr context.
   • positions are highly rounded which is the cause of a staircase effect when moving the controllers
2. XR terrain intersection via buffer reading gives wrong distance result using c3DEngine.depthBufferRGBAValueToOrthoZ() method.
   • I assume it is due to the camera array provided for this computation
3. Controllers binding differs from a model to another, there is no clear way to identify by code which model is used yet. So should we define basic interactions only for the common API provided ? (https://www.w3.org/TR/webxr/#event-types)

Potential Solutions

1. Fact : Precision issue does not occurs in a classical scene (near the origin). -- One track followed was to take advantage of the parent group object containing the VR helmet cameras + controllers. This parent would be holding the world positioning while the XRReferenceSpace would be left to the 3D origin coordinates. One struggle met is that Itowns doesn't let tricks itself so easily. -- Another track was to separate the VR camera array positions from the Itowns camera and synchronize both positions of the parent group previously set up. The goal was to keep Itowns camera positions unchanged while taking the benefits of a rendering near 3D origin. Again, it didn't fit that easily. -- Move everything but the cameras + VR parent group, aka applying camera inverse WorldMatrix to the root scene and reset cameras matrices. Again and again, Itowns is not fit for that purpose.
2. In investigation
3. Find a way to identify which device is in use ?

Identified use-cases

known issues

• Scaling down the scene at start is not suitable with controllers instanciation.
• Controller binding mistake when only one of the two controllers are registered.
• intersection result using a raycaster.intersectObjects([objects]) is mild working :
  • [x] ok over a Mesh,
  • [ ] ko for a pointCloud => coordinates obtained by point intersection came from bounding box tests
  • [ ] ko for the terrain elevation => geometry intersection only intersect with level0Node which is the geoide. => pixel reading is also giving false positive / wrong distance value as said in previous section.

Documentation

Can be tested with Chrome extensions such as :

• https://github.com/MozillaReality/WebXR-emulator-extension
• https://chrome.google.com/webstore/detail/immersive-web-emulator/cgffilbpcibhmcfbgggfhfolhkfbhmik