3D tiles are not updated during VR session

[afdmulder]

Question

I start a VR session with a specific errorTarget value. As a result, I initially see only a few 3D tiles which is fine. When I move my avatar towards an adjacent tile, I expect that tile to be loaded and displayed. However, that does not happen. The adjacent tiles are only shown after I interrupt or pause the VR session.

Question: Is this behavior expected or am I missing something?

Supplemental Data

Steps to reproduce behaviour:

1. Start VR session
2. Navigate to the an adjacent tile 3 .Wait for a moment: the adjacent tile is not shown
3. interrupt the VR session
4. the tile(s) will show up

Example video: https://3dcitytour.nl/support/3dcitytour_support.mp4

Example environment: https://3dcitytour.nl/support

Library Version

0.4.17

Three.js Version

0.160.1

[gkjohnson]

Hi @afdmulder - without simple demo code it's hard to say what may exactly be going on but I recommend taking a look at the vr example which demonstrates a assigning a custom scheduling callback for tile loads so they can be handled in the relevant request animation from loop. WebXR uses a separate rAF loop from the one on window (which is used for scheduling) and the tiles renderer otherwise has no way of knowing which render loop to use without assigning a custom callback.

[afdmulder]

Hi Garrett, thanks for your reply and tips. In fact I based my code on the VR example. In mains.js/reinstantiateTiles() I add the tilesets and I apply schedulingCallback. In the mains.js/animate() I call handleTasks like you do in the VR example.

However, it looks like tasks are not added during the VR session. As soon as I pause the VR session, the numbers of tasks increases and the tasks are processed.

My test repository and branch https://github.com/afdmulder/3DCityTour/tree/feature/vr_rendering

[afdmulder]

I've noticed that the 3D tile update process does work if I leave the web browser open (after pausing my session) as a kind of standby screen. I can still navigate in the VR world then. But as soon as I resume the VR session the update proces holds

See screenshot.

Device Oculus Meta Quest 2

[gkjohnson]

My test repository and branch https://github.com/afdmulder/3DCityTour/tree/feature/vr_rendering

Unfortunately I can't review large scale projects or repos for problems like this - it's not sustainable with the amount of questions I get. I generally have to ask for small reproductions of the problem ideally confined to a single file or two.

[afdmulder]

Hi, that's makes fully sense. I've been thinking about it... In your vr example I replaced the url by mine 'https://www.3drotterdam.nl/datasource-data/d0c755ef-ba1b-43fc-bc44-24fc7bb152ce/tileset.json'.

Although the 3D tiles are flipped and float in the air, their behavior is the same. If I point to a building in VR, I teleport there. Initially, however, you see simplified geometry. As soon as you pause/interrupt the VR session, the detail is loaded.

The small video below is based on your VR.html with the tileset I mentioned above. https://3dcitytour.nl/support/3dcitytour_support2.mp4 Regards

[afdmulder]

I realized that I had made a small modification to B3DMLoader.js in order to be able to read the tileset at 'https://www.3drotterdam.nl/datasource-data/d0c755ef-ba1b-43fc-bc44-24fc7bb152ce/tileset.json'.
I had to add a loop inside the loader.parse function.

loader.parse(gltfBuffer, workingPath, model => {
				const { batchTable, featureTable } = b3dm;
				const adjustmentTransform = this.adjustmentTransform;

				// can have multiple scenes
				for (const scene of model.scenes) {

					const rtcCenter = featureTable.getData('RTC_CENTER');
					if (rtcCenter) {
						scene.position.x += rtcCenter[0];
						scene.position.y += rtcCenter[1];
						scene.position.z += rtcCenter[2];
					}

					// Transformatie toepassen
					scene.updateMatrix();
					scene.matrix.multiply(adjustmentTransform);
					scene.matrix.decompose(scene.position, scene.quaternion, scene.scale);

					model.batchTable = batchTable;
					model.featureTable = featureTable;
					model.scene = scene; // overschrijf default scene met de juiste

				}

				resolve(model);

			}, reject);

[gkjohnson]

Hi @afdmulder - it would be best to provide a self-contained example that sets the terrain up and positions the camera in a position where this can be reproduced without navigation. Ideally this can be tested and reproduced in the WebXR emulator chrome extension. Unfortunately I have limited time so spending time setting these things up to test including getting a VR headset set up is more than I can invest at the moment.

[afdmulder]

Hi Garrett,

Thanks for your response. I will work on an example.

I have done research myself in the meanwhile and I found out that:

• I do not have problems with the webxr extension in Edge/Firefox/Chrome
• I have another webbrowser in my Quest 2 (Wolvic) and I did not have this particular problem. All LOD's were loaded during webxr session dynamically but furthermore this webbrowser was very unstable for my use case.

Initially I thought loading was the problem. I'm not sure about that. Custom scheduling callback seems to work well. I connected my Quest with Chrome Inspect and I learned from the log files that the highest possible (16) LOD was not loaded and rendered during the VR session.

In webxr VR mode I see the maximum LOD level is 15 (../../15/33591/6930.b3dm) As soon I pause the session the tile loading continues and I see level 16 passing by (../../16/67172/13860.b3dm)

So I thought Frustom might not determined correctly. But I does work in Wolvic... I tried to lower down FPS (30) to make free resources to load and render the highest detail LOD.

Unfortunaltely no luck sofar but I will continue my investigations. As soon as I have a good working example I will let you know.

But if you have tips or tricks where I should have a look at they are more than welcome.

André

[gkjohnson]

I have another webbrowser in my Quest 2 (Wolvic) and I did not have this particular problem.

If this is the case then it seems likely it is a quirk with the original browser. I would test with other browsers including Chrome or Firefox XR to compare (assuming the problematic browser is the built in Meta one).

I would also check the following:

• Ensure the priority queue callback is firing as expected.
• Ensure the LRUCache is not "full", which will prevent new tiles from loading (check tilesRenderer.lruCache.isFull())

[afdmulder]

Hi Garrett,

Device: Quest 2 OS version: v81.1034 (4-11-2025)

I created a minimal working VR example (based on your VR example). Steps to reproduce behaviour.

• clone repository https://github.com/afdmulder/minimalvr.git

• npm install

• npm run dev (run locally)

• npm run build (create distribution)

• visit my website in integrated webbrowser or build your own

• note that the buildings in the center are not rendered at the highest possible LOD

• enter VR mode. You will be directly positioned in front of the buildings. No navigation is needed. It doesn't matter how close you are to the building; the details won't load

• Pause the webxr VR session. The tiles will be loaded immediately.

• if desired teleporting is enabled by aiming at the white plane / floor You will notice that the tiles won't be loaded dynamically.

Other notes:

• I cloned your sources in the mininmalvr project as I made a small modification in B3DMLoader.js At line 63 I implemented a loop as this seems to be needed to load this tileset.

• In a webxr simulation environment (Edge.Firefox/Chrome) as well as Wolvic webbrowser in Quest 2 it works as aspected

• I do realize (recently) that the integrated browser or Quest OS might be the problem and for that reason is not solvable in tilesrenderer.js But your knowledge or the knowledge of community members might help me.

With best regards, André

[phoenixbf]

I can confirm the exact same issue on a Quest 2 updated to latest version THREE r181

Meta browser fails somehow to load new tiles (not previously visited) during immersive VR session, while Wolvic for instance works as expected. I initially thought about some issue with camera, but the issue can be replicated also on the Dingo Gap VR example.

When pausing the session, the tiles loading and refinement resume, and everything works again as expected depending on current view and location. When resuming the session, and navigating around the tileset, the issue appears again.

I can investigate a bit more what is causing this, it seems related to Meta browser

[gkjohnson]

Meta seems to have made setting up debugging on the Quest 2 particularly difficult some time since I've last used it. I generally am not doing VR dev and unfortunately won't have the cycles to set this up soon. I'd appreciate some help debugging this and I can provide some guidance.

WebXR provides a separate "requestAnimationFrame" function for VR from the window version. Some browsers may run both simultaneously while others (like Meta's) may only fine the relevant rAF functions. My best guess at what might be happening is that the project has added instances of requestAnimationFrame in a few places that the renderer requires evaluation for (you can search for requestAnimationFrame in the project). I would recommend trying the following:

• Replace all instances of requestAnimationFrame with setTimeout( callback, 16 ) to see if that fixes it.
• If that does fix it then change progressively change setTimeout calls back to rAF to narrow down which one is causing the problem.

If this addresses the issue then we know what the issue is and can figure out the right way to go on from there. Unfortunately afaik there's not completely transparent way to handle per-frame callbacks in VR without the user providing some custom callback but may we can simplify that process.

[phoenixbf]

Thanks! I can confim that replacing requestAnimationFrame with setTimeout (and clearTimeout) solves the issue on Meta Quest browser, altough introducing ofc a bit of flickering when a tile is mounted during the immersive session. I'll try to narrow down the problem

[afdmulder]

Hi, FYI, I also got it working in both the minimalvr example and in my own project. I'm already very happy with this progress. This is an important feature for large areas.

[gkjohnson]

As an idea for how to improve the ergonomics of swapping between the window and xr session rAF functions, we can include a "setXRSession" function on the TilesRendererBase class to provide it with access to the necessary callbacks (cancel and schedule rAF). Then we can use these rAF functions throughout the class:

xrSesstion.addEventListener( 'start', () => tiles.setXRSession( xrSession ) );
xrSesstion.addEventListener( 'end',   () => tiles.setXRSession( null ) );

[afdmulder]

Hi, A structural solution is certainly welcome. However, I don't know if this is a bug or a feature in Meta Quest Software. I also don't know if other brands are experiencing issues (or will experience them in the near future). We could wait and see, but if Meta Quest Software isn't updated, it's likely that more users will experience this problem.

[gkjohnson]

From the MDN docs onXRSession.requestAnimationFrame:

Note: Despite the obvious similarities between these methods and the global requestAnimationFrame() function provided by the Window interface, you must not treat these as interchangeable. There is no guarantee that the latter will work at all while an immersive XR session is underway.

So this is not a bug and will likely not be changed in the Meta browser so we should fix it here.

[phoenixbf]

So, I tried successfully this approach:

requestAnimationFrame = (cb)=>{
    if (!tiles._xrSession) return window.requestAnimationFrame(cb);
    return tiles._xrSession.requestAnimationFrame(cb);
};

cancelAnimationFrame = (handle)=>{
    if (!tiles._xrSession) window.cancelAnimationFrame(handle);
    else tiles._xrSession.cancelAnimationFrame(handle);
};

with tiles._xrSession being the internal XR session set on a tileset when a session is started (see above snippet suggestion from @gkjohnson )

Problem is now these routines are called in multiple places (e.g.: throttle, QueryManager, PriorityQueue, LRUCache), so I dont know what's the best approach to refer the revisited req/cancel AF in these.

[gabschettino]

Glad to see it wasn't just me with the same issue.

When I did put up an basic app to render Google Photorealistic 3D Tiles a few months ago with my Quest 3, I had the same issue, you can see me pressing the Meta/Oculus button for it to "refresh" the tiles

[gkjohnson]

Problem is now these routines are called in multiple places (e.g.: throttle, QueryManager, PriorityQueue, LRUCache), so I dont know what's the best approach to refer the revisited req/cancel AF in these.

I think it would be appropriate to have something like a "RAFManager" (name TBD 😅) that can be shared among multiple classes and will manage this which callback approach to use:

class RAFManager {
  // uses rAF or cAF on the xr session if present, otherwise uses the global versions
  requestAnimationFrame( cb );
  cancelAnimationFrame( cb );

  // sets an xr session to use the animation frame functions for, null to remove
  setXRSession( xrSession ); 
}

The class will have do the following:

• Defer callback queuing to the xr session if set. Otherwise call rAF on the window.
• If the XRSession is set (or removed) while events are already queued we can cancel all events and run the events immediately (or requeue them with the newly set rAF functions)

Then it can be used (and shared) by the TilesRenderer, lru cache, and priority queue. There will likely be details to work out but I think this is something we'll have to start on and discover what works and what doesn't as we go. Lets focus on getting the core TilesRenderer use case working first. Then we can look at other plugins and whatnot to see else needs to be modified. Maybe someday the browser will have some kind of mechanism to manage and schedule callbacks in the browser event queue with a bit more fidelity but it seems like this might be our best option until then.

Feel free to submit a PR with an initial rough implementation so we can see how it looks and discuss further if you'd like.

[phoenixbf]

made an initial PR with a simple AFManager class with routines, from here we should then instantiate it in throttle, QueryManager, PriorityQueue and LRUCache at least. Also I think the VR example should be modifiied to show how to set the XR session when it's started