WebGL2 Occlusion Queries feature

[simon-paris]

Example: https://raw.githack.com/simon-paris/three.js/occlusion-queries-with-build/examples/webgl_occlusionqueries.html

Documentation: https://raw.githack.com/simon-paris/three.js/occlusion-queries-with-build/docs/api/en/objects/OcclusionQueryMesh.html

This change integrates WebGL2 queries into the renderer, allowing occlusion querying like this:

var queryMesh = new THREE.OcclusionQueryMesh(
	new THREE.BoxBufferGeometry( 1, 1, 1 ),
	new THREE.MeshBasicMaterial( { depthWrite: false, colorWrite: false } )
);
queryMesh.frustumCulled = false;
queryMesh.renderOrder = Infinity;
var highestFrame = 0;
queryMesh.occlusionQueryCallback = function ( result, frame, camera ) {

	if ( camera === myMainCamera && frame > highestFrame ) {
		highestFrame = frame;
		doStuff( result ); // queryMesh was visible if result === true
	}

};
scene.add( queryMesh );

Hope you like it

[Mugen87]

Have you considered to choose a different API style similar to BabylonJS? Objects have a property isOccluded which indicates whether the mesh is occluded or not. If you assign a specific value to occlusiontype, an occlusion query is configured for the respective object. You can now evaluate isOccluded e.g. in your render loop and react accordingly. Check out this approach right here:

https://www.babylonjs-playground.com/#QDAZ80#5

[Mugen87]

I personally tend to prefer such an API since it feels easier to use. I'm curious how others think about this 😇

[simon-paris]

Can I instead output the result by pushing it to an array?

If I output the result as a single boolean, you wouldn't get a result for each frame. A guarantee to get one output for each frame rendered would be useful for some people. A single boolean also wouldn't work with multi-camera setups.

For the serialization problem, I could always deserialize it into a disabled state? Then programs that aren't aware of the occlusion query don't have to worry about leaks or disposal.

[Mugen87]

A guarantee to get one output for each frame rendered would be useful for some people. A single boolean also wouldn't work with multi-camera setups.

I like to keep things simple. I would prefer an API that is easy to use and covers 90% of use cases. Better than a more complex one which also regards edge cases. A single boolean value for 3d objects should be sufficient for most scenarios even for VR applications.

But like I said before, this is just my opinion. Let's see how others think about the API style.

[Ben-Mack]

Any update on this? This would be a great feature for Three.js.

[mrdoob]

Do you mind resolving the rollup.config.js conflicts?

[simon-paris]

Sorry! I've been busy at work.

I've resolved the conflict.

[vegarringdal]

Is this something that will be added ? Sound like a good feature to have for some cases where you have a lot of meshes of different types.

Is the queries expensive to use ? can worker

Trying to load very large project converted from many IFC files to glb files.

Atm it looks like I need to find a way to remove what user isnt able to see away from the memory. And have a very low quality model I use when users starting to get close.

But again I have a LOT to learn... Im very new to 3d 😁😂

[Mugen87]

TBH, I would prefer a different API style explain here: https://github.com/mrdoob/three.js/pull/15450#issuecomment-448953864

[vegarringdal]

@Mugen87

I know to little to say whats best. I just tried to use babylonjs occlusion, but didnt give me performance like I expected. Actually threejs is doing a lot better. But I guess some of this might have to do more about my skills 😂 there is so much I dont know... so I might have used it wrong and generated to much queries.

[Mugen87]

but didnt give me performance like I expected.

Such statements are always problematic if you don't formulate the actual testing conditions.

Besides, the value of a feature is also about ease of use. I personally prefer the simple interface of BabylonJS.

[simon-paris]

Hey, sorry for not updating this for so long. I no longer work in 3d graphics, but I might get time later this week to clean this up.

I agree the API needs to be simpler, but I think there's a better way than writing a boolean propety to the object.

Just wanted to explain why it works the way it does:

• I decided to use a callback mainly because as far as I could see there were no other places in the codebase where the renderer writes a property back to an object. Additionally, the timing is important. The callback fires at a somewhat predictable time in the rendering process, allowing action to be taken immediently. We always want to check the query result as late as possible to maximise the chance of a success, and we want to be able to act as soon as possible.

• I made it per-camera mainly because the renderBufferDirect method doesn't know if the camera it's rendering is the main camera. I really didn't want to pass that information down into the occlusion code because I'd have to make changes to some public method signatures. Plus it would give wrong results for multipass, shadows, or VR. Shadow-camera support is useful because knowing if your object is visible in shadows is just as important as knowing when it's visible on the main camera.

• WebVR/ArrayCamera is particularly tricky because multiple queries have to be made at the same time. Outputting a single boolean would be difficult since we would need two queries and we don't know which camera is which when we're inside the occlusion code. The same applies to multi-materials and multiple renderers. But in my opinion it'd be better to explicitly not support these things than to produce a wrong result.

• In my opinion, deserialization shouldn't enable the query by default. Occlusion queries are wasteful if there's nothing acting on the results. Whatever code is interested in the results should simply enable the query.

• The pollAllOcclusionQueries method is useful for users who render their scene only when something changes. In my application, I didn't render the scene unless the camera moved, and I would never get my query results, because it's waiting for the next frame.

So I'd like to propose simplifying the api in the following ways:

1. Keep the callback, but only pass (result, camera)
2. Require cameraFilter to be set to [mainCamera] otherwise do nothing. Don't default to "all cameras".
3. Don't support webVR
4. Remove pollQueries but keep pollAllOcclusionQueries

Example:

var queryMesh = new THREE.OcclusionQueryMesh(
	new THREE.BoxBufferGeometry( 1, 1, 1 ),
	new THREE.MeshBasicMaterial( { depthWrite: false, colorWrite: false } )
);
queryMesh.frustumCulled = false;
queryMesh.renderOrder = Infinity;
queryMesh.cameraFilter = [ myMainCamera ];
queryMesh.occlusionQueryCallback = function ( result, camera ) {

        doStuff( result ); // queryMesh was visible from camera if result === true

};
scene.add( queryMesh );

As a side note, the other occlusion query PR #20554 has some issues. I'll leave a comment over there.

@vegarringdal When I was working with 3d graphics I was also rendering IFC files. The performance I got from occlusion queries wasn't great. Are you doing instancing? That works pretty well. But in general rendering IFCs is a bad time because the geometry in them is usally crap, there's only so much you can do.

[vegarringdal]

but didnt give me performance like I expected. Such statements are always problematic if you don't formulate the actual testing conditions. Besides, the value of a feature is also about ease of use. I personally prefer the simple interface of BabylonJS.

@Mugen87 Yes, main problem was that I was hoping for magic and I have really no clue what Im doing 😄 . Its not babylonJS fault.

---

@simon-paris

When I was working with 3d graphics I was also rendering IFC files.

Did you also use IFC -> glTF ?

Are you doing instancing? That works pretty well.

No, not sure about the best way to do it/how it would work on the data we get in the IFC's Was also thinking about using a stripped down model as default, so when user is moving I only show this for better fps. Then use LOD for close parts with a little delay. Not sure how Im going to deal with memory usage either, cant load all geometry and keep it. Maybe use the filesystem access api so I get faster access to glbs.

The performance I got from occlusion queries wasn't great.

Ok, I was kinda hoping for some magic here since its a huge oil rig Im trying to render. And a lot will be hidden behind walls or other large parts. I was hoping the check could be done in a worker, so new status would have a little delay, but as default it would keep state it had until check said otherwise. So if hidden, it stays hidden. Great if you get time to clean up the PR btw. 👍

But in general rendering IFCs is a bad time because the geometry in them is usally crap, there's only so much you can do.

Its been going a lot better then I though it would, when I started trying 2 weeks ago. Would have been nice if IFC was better supported. Only other option is RVM, and not very well supported either. not that I have found.

If you have any other tricks/tools you used to make ifc geometry better then please tell me. But instancing/merged mesh(since I have so many) and web worker will hopefully help.

[simon-paris]

@vegarringdal

Did you also use IFC -> glTF ?

I used IFC -> Custom format. But pretty similar where it matters.

I was hoping the check could be done in a worker, so new status would have a little delay, but as default it would keep state it had until check said otherwise

I wouldn't recommend rendering occlusion tests in a worker because you'll be sharing the same GPU as the main thread. You should use them to test bounding boxes around your most expensive meshes. Don't try to occlusion test every object in your scene.

Was also thinking about using a stripped down model as default, so when user is moving I only show this for better fps. Then use LOD for close parts with a little delay.

Do you have good quality LODs? That was an issue for me, it needed automatic LOD generation but the geometry was so bad we just couldn't. I relied on instancing and merging to render 60k objects in 2k draw calls at (mostly) 60fps. The issue with my approach was that there's nothing I could really do from there. Instancing makes it hard to sort objects front to back, to frustum cull, to load only part of the model and so on. Certain camera positions performed badly and there was nothing we could do.

Here's what I'd do in retrospect. I'm assuming your model has many thousands of small objects, most of which are opaque, untextured, and low-poly, like boxes for walls, cylinders for pipes, etc.

1. For opaque, low-poly, untextured objects:

   1. Take all the objects that are large by volume, but small by polygon count (walls, floors, structural). Merge them all into one big model with per-vertex color/roughness/whatever so that you can render it with one material, Then chop it up into cubes. Always render those cubes, they can act as the low LOD.
   2. Repeat for object that are small by volume and small by polygon count (pipes, light fixtures). Merge them and chop them into cubes in the same way. Only render these cubes when the camera is near. You can unload these to save memory.

2. For everything else:

   1. If the object is very repetitive, render it with instancing (when the model is large enough or repeated enough times that the benifit of deduplicating the mesh outweighs the cost of the instanced draw)
   2. If the object if very high poly, don't use instancing. These are good candidates for occlusion culling. Draw a bounding box around them, if it passes the occlusion check, render it, else don't.

It's also very helpful to use smaller datatypes for your vertices. You can do 3x1 byte normals or flat normals. You can do 2 bytes for color. You might even be able to do 16-bit floats for position.

FYI it's possible to hack together occlusion culling without this branch:

// at the start of your frame
previousQueryObjects.forEach((queryObject) => {
    if (gl.getQueryParameter( queryObject, gl.QUERY_RESULT_AVAILABLE )) {
       lastQueryResult = gl.getQueryParameter( queryObject, gl.QUERY_RESULT ) > 0;
    }
});
// hide/show the object based on lastQueryResult

// at the end of your frame
const query = gl.createQuery();
previousQueryObjects.push(query)
gl.beginQuery( gl.ANY_SAMPLES_PASSED, query );
renderer.renderBufferDirect(...); // render bounding box of expensive object
gl.endQuery( gl.ANY_SAMPLES_PASSED );

[vegarringdal]

@simon-paris

Thanks for taking the time to answer. 👍 Will be trying out different solutions. The feedback you just gave will be very helpful. Very thanks for the occlusion culling sample/hack. I can maybe have some large bounding boxes for some of the areas, and smaller ones inside. This can get expensive fast i guess, any max I should think of ? Did you try out with offscreenCanvas api btw? If you can come on any other ideas then please tell me. Or tools you used.

[gkjohnson]

Why is the callback needed? I would expect an option for whether to enable occlusion culling on Mesh and if it's enabled and that mesh wasn't seen by the current camera during the previous query then it shouldn't be rendered. Maybe an additional option to render the bounding box rather than the original geometry as a query like BabylonJS. To me that seems much simpler and in line with existing three.js patterns.

[kyjanond]

Hi, it would be very nice if this made it to the next release. If you need any help let me know please @simon-paris

[jcguarinpenaranda]

This would be awesome to have

[HeadClot]

Just thought I would share my two cents on this. I am building a fairly complex project from a geometry point of view. I need to get 90 or more Frames per second for VR support. It would be great if Threejs supported this out of the box.

[benzsuankularb]

Seems like this is getting more attractive since we're getting more and more WebGL2 available on the market (iOS 15).

[LeviPesin]

For some reason in the example "Sphere visible/occluded by the shadow map" works correctly, while "Sphere visible/occluded on the screen" does not - it almost always says "sphere visible", including cases when it is not.

[ndesseaux]

Any news on this? This would be awesome to have @simon-paris

[JonasBlazinskas]

+1 this would help lots in AEC industry

[luozhonghai]

want this feature

[nikolas-karinja]

I can't express enough how much me and my colleagues would love to have this feature.

[evandarcy]

I would love to see this feature make it into a release! Are there any thoughts on what needs to happen to make that possible?

[nowaythisworks]

This feature would really help mature the optimization options Three.js has :) Great work!

[GoJermangoGo]

Hi! I come from the future where #26335 has been merged. It'd be great to have occlusion queries for WebGL as well since WebGPU isn't widely supported yet. I haven't looked at how either PR is implemented yet, but I'm sure this PR could use some polishing to get it up to the same standard as the WebGPU version. We're willing to allocate some development time to this project if anyone's interested. @mrdoob @Mugen87

[LeviPesin]

"WebGPURenderer" is a confusing name for the universal renderer that works for both WebGPU and WebGL2. You can try replacing WebGLRenderer in your project with it.