Canvas children not rendering when starting an immersive XR session before rendering the Canvas

[lourd]

• react-three-fiber version: 5.1.3
• threejs version: r121
• react: 17.0.0
• hardware: Oculus Quest 1 & 2

I'm working on a project that has an immersive mode and a flat mode. Rather than using the VRButton from Three.js like the examples do, I'm handling that interaction with the navigator.xr.requestSession API myself in order to own the control flow between the two modes.

The bug or confusing behavior I'm running into is that if the canvas hasn't been created before the XR session starts, the app will appear to hang. In the example below, after pressing the Start VR button from the Oculus browser, you'll enter the immersive view, but then just see a loading spinner that never goes away. There's no information logged or errors thrown; nothing happens.

If I exit the immersive session by using the Oculus menu, the app correctly switches back to the flat entry point, showing the Enter VR button.

Full code with the buggy behavior

import { OrbitControls, Sphere } from 'drei';
import React, { Suspense, useEffect, useState } from 'react';
import { Canvas, useThree } from 'react-three-fiber';
import { XRSession } from 'three';

const vrSupportedResource = wrapPromise(
  new Promise<boolean>((resolve) => {
    if (!navigator.xr) {
      resolve(false);
      return;
    }
    navigator.xr.isSessionSupported('immersive-vr').then(resolve);
  })
);

export function App() {
  const vrSupported = vrSupportedResource.read();
  const [xrSession, setSession] = useState<XRSession>();

  const startVR = async () => {
    const session = await navigator.xr.requestSession('immersive-vr', {
      optionalFeatures: ['local-floor', 'bounded-floor', 'hand-tracking'],
    });
    setSession(session);
    // If the session is ended from elsewhere, like the device menu
    session.addEventListener(
      'end',
      () => {
        setSession(null);
      },
      { once: true }
    );
  };

  if (vrSupported) {
    // in the headset, not gone immersive yet
    if (!xrSession) {
      return (
        <button onClick={startVR} style={{ position: 'absolute', top: '50%', left: '50%', transform: 'translate(-50%, -50%)', fontSize: 36, zIndex: 1 }}>
          Start VR
        </button>
      );
    }

    // in the headset, entered immersive
    return (
      <Canvas vr>
        <ambientLight />
        <ImmersiveApp session={xrSession} />
      </Canvas>
    );
  }

  // else, we're on desktop
  return (
    <Canvas>
      <Stats />
      <ambientLight />
      <OrbitControls />
      <FlatApp />
    </Canvas>
  );
}

function ImmersiveApp(props: { session: XRSession }) {
  // NEVER GETTING TO HERE
  console.log('in immersive app');
  const { gl } = useThree();

  useEffect(() => {
    gl.xr.setSession(props.session);
  }, [gl.xr, props.session]);

  return (
    <>
      <Sphere>
        <meshBasicMaterial attach="material" color="green" />
      </Sphere>
    </>
  );
}

function FlatApp() {
  return (
      <Sphere>
        <meshBasicMaterial attach="material" color="hotpink" />
      </Sphere>
  );
}

type Status = 'pending' | 'success' | 'error';

function wrapPromise<T>(promise: Promise<T>) {
  let status: Status = 'pending';
  let response: T;

  const wrapper = promise.then(
    (res) => {
      status = 'success';
      response = res;
    },
    (err) => {
      status = 'error';
      response = err;
    }
  );

  const read = () => {
    switch (status) {
      case 'pending':
        throw wrapper;
      case 'error':
        throw response;
      default:
        return response;
    }
  };

  return { read };
}

Debugging the issue, it seems that the ImmersiveApp component is never being rendered. I'm not sure why; my guess is some sort of internal timing issue with the canvas' readiness state?

I can workaround the issue by rendering the canvas before the XR session has started, like this:

Workaround code

  if (vrSupported) {
    let button;
    // in the headset, not gone immersive yet
    if (!xrSession) {
      button = (
        <button onClick={startVR} style={{ position: 'absolute', top: '50%', left: '50%', transform: 'translate(-50%, -50%)', fontSize: 36, zIndex: 1 }}>
          Start VR
        </button>
      );
    }

    // in the headset, entered immersive
    return (
      <>
        {button}
        <Canvas vr>
          {xrSession && (
            <>
              <ambientLight />
              <ImmersiveApp session={xrSession} />
            </>
          )}
        </Canvas>
      </>
    );
  }

but this isn't ideal.

Thanks in advance for the help 🙏 and kudos on a fantastic library 👏

[lourd]

On some further investigation, it looks like the likely cause is the ResizeContainer. The renderer callback from here is never getting called:

https://github.com/pmndrs/react-three-fiber/blob/8b07c7ab13777ea9218a08baef6e4c20b86e5059/src/targets/web.tsx#L12-L28

because the size dimensions coming back from useMeasure are 0, which means that the ResizeContainer's ready is never true.

https://github.com/pmndrs/react-three-fiber/blob/8b07c7ab13777ea9218a08baef6e4c20b86e5059/src/targets/shared/web/ResizeContainer.tsx#L78-L88

So resolving this would come down to decoupling the Canvas and ResizeContainer components, exposing some sort of "raw canvas" component that doesn't include the resizing behavior that's not needed for some applications, such as immersive ones.

[lourd]

One more piece of information -- regardless of the specified dimensions of the canvas, the size measured on the first pass has a height and width of 0, and there's never a second render with the real measurements. Inspecting the elements through the browser dev tools show that the canvas is created and has the expected size, but however react-use-measure works isn't picking it up.

[drcmda]

is this a fix that belongs to use-measure?

[gsimone]

So resolving this would come down to decoupling the Canvas and ResizeContainer components, exposing some sort of "raw canvas" component that doesn't include the resizing behavior that's not needed for some applications, such as immersive ones.

This will be the case in v6, please let us know if you have additional ideas about it, there's a generic issue tracking v6 stucc here #750

[lourd]

This is still happening with the latest versions of @react-three/fiber and Quest Browser... I think it might be a good idea to re-open this issue, as I've just unwittingly fallen into this pitfall again 🤦

This is my code simplified:

function App() {
  const [state, setState] = ...

  if (!state.xrSession) {
    <button onClick={requestXrSession}>Get started</button>
  }
  return (
    <Canvas>
      <ImmersiveApp xrSession={state.xrSession} />
    </Canvas>
  )
}

function ImmersiveApp(props: { xrSession: XRSession }) {
  const { gl, scene } = useThree()

  useEffect(() => {
      gl.xr.setSession(props.xrSession)
    return () => {
      gl.xr.setSession(null)
    }
  }, [gl.xr, props.xrSession])

  return null
}

One interesting note is that the behavior is different in Safari on Vision Pro than Quest Browser; Safari does not have the issue.

I followed that clue and I think that the issue is that when starting an immersive immediately Quest Browser isn't firing the callback given to the ResizeObserver constructor here in react-use-measure:

https://github.com/pmndrs/react-use-measure/blob/8639e5a93d60930159dc83743780ce4787fe90bb/src/web/index.ts#L137

This could be worked around by changing useMeasure to set the element's initial size in a useLayoutEffect or in the ref callback.

Regardless, I think as far as this library is concerned, the fact that a browser's quirk with measuring a div's size is somehow stopping an XR session from functioning at all seems like an issue. It should be possible to not use the resizing machinery when it's not needed. Even if there wasn't an issue, I don't want to waste work creating ResizeObservers while starting an immersive session in my application.

[lourd]

Ah I see, just read through the createRoot docs and figured out how to use that. Does this look right for making a custom barebones Canvas component? Aside from the extend(THREE) part that would be needed.

import { createRoot } from "@react-three/fiber"
import { useEffect, useState } from "react"

export function Canvas(props: { children: React.ReactNode }) {
  const [canvas, setCanvas] = useState<HTMLCanvasElement | null>(null)

  useEffect(() => {
    if (!canvas) return
    const root = createRoot(canvas)
    root.render(props.children)
    return () => {
      root.unmount()
    }
  }, [canvas, props.children])

  return <canvas ref={setCanvas} />
}