WebXRDepthSensing can result in invalid projectionMatrix (`NaN` values)

[mrxz]

Description

When using WebXR depth sensing, Three.js adjusts the far and near properties of the camera to match the depthFar and depthNear coming from the depth-sensing module. In the case of the Meta Quest 3, the far value becomes Infinity. As a result, the left, right, XR and user cameras all end up with NaN values in their projection matrix as Matrix4.makePerspective does not support an infinite far plane.

Rendering during the immersive session still looks fine as the projection matrices provided by WebXR are used when rendering the left and right views (WebXRManager.js#L752-L753). NaN values also show up in the Frustum, but somehow don't cause noticeable issues (perhaps nothing gets frustum culled?).

When exiting the immersive session, the scene is rendered with the user-camera, which now has NaN values in its projection matrix, resulting in the screen showing up empty (see screenshots).

The current depth sensing implementation assumes that Three.js supports a far value of Infinity. A proper solution would either be to support it (#11755) or handle occlusion differently (#28877). A crude workaround could be to fix/restore the projection matrix of the user-camera upon exiting the session (something I now do in some of my experiences).

Reproduction steps

1. Open the webxr_xr_dragging example on a Quest 3
2. Click on 'Start XR'
3. Click on 'Stop XR' to exit
4. Notice the canvas being blank

Code

document.body.appendChild( XRButton.createButton( renderer, {
	'optionalFeatures': [ 'depth-sensing' ],
	'depthSensing': { 'usagePreference': [ 'gpu-optimized' ], 'dataFormatPreference': [] }
} ) );

Live example

• webxr_xr_dragging

Screenshots

Before entering XR	After exiting XR

Version

r167

Device

Headset

Browser

Chrome

OS

Android

[hybridherbst]

The values can also be 0; we have this check in our code:

and we also reset the near/far plane after exiting AR because of this issue.

I believe this issue might be related (Audio is indeed one of the things that breaks with broken camera mats): https://github.com/mrdoob/three.js/pull/27588

[Mugen87]

A crude workaround could be to fix/restore the projection matrix of the user-camera upon exiting the session (something I now do in some of my experiences).

How about we add this and the additioanl checks of @hybridherbst as a workaround until we have a different solution?

[mrxz]

I gave it a shot in https://github.com/mrdoob/three.js/pull/29120. Instead of restoring the near/far values, I opted for not overwriting the user camera's near/far. Reason being that these values already act as inputs (when not using depth-sensing) so only capturing their values on session start would miss updates made during the session.

To properly avoid any NaN values, the setProjectionFromUnion now has a code-path to handle an infinite far plane. This uses the non-generalized approach (see https://computergraphics.stackexchange.com/questions/1736/vr-and-frustum-culling/4765). This does mean that the near plane isn't aligned, resulting in a slightly larger frustum (which is no problem for culling purposes)

[Mugen87]

Considering this issue as fixed via #29120.