godotengine
Split rendering of shadowmaps into a static and dynamic pass
This PR changes the shadow map rendering logic so static objects (objects marked as not moving) are rendered to a static shadow map. This static shadow map is only updated if the light changes or if the light is assigned a new region in the shadow map.
This PR is now rebased on top of #77085 as I'm building on some of the debug changes in that PR
This functionality can be enabled through a new project settings:
Each frame the static shadow map is copied and we render dynamic objects (objects marked as potentially moving) into this shadow map. This is this shadow map that is used to render shadows.
This removes a lot of overhead in rendering shadow maps when there are many static objects.
Geometry can be marked as static or dynamic to determine whether they need to be rendered to shadow maps every frame (if applicable). There is an automatic mode that, with a little overhead, figures out which objects move after the static shadow map is rendered and marks those objects as dynamic automatically. After a few frames dynamic objects are identified and the efficiency is applied.
The property name is currently named
Render shadow.
See the discussion in the comments about possible changes with an approach for using an "inherit from parent" approach though the automatic mode seems to work well enough.
Note if animations are linked to a mesh, we automatically recognize it as dynamic.
Our debug display will overlap the static and dynamic results.
- Static shadows are shown in black and white (black is close)
- Dynamic shadows are shown in black and blue (blue is close)
Currently only the Vulkan renderer supports this feature.
Todos:
-
y-flip the debug image, everything is upside down atm
-
Make this work for directional shadows (this may become a separate PR or part of #76291)
-
Production edit: This closes https://github.com/godotengine/godot-proposals/issues/4635.
This resolves https://github.com/godotengine/godot-proposals/issues/4635, and which is one part of https://github.com/godotengine/godot-proposals/issues/6948
And this issue partially i think. https://github.com/godotengine/godot/issues/73411
Just food for thought, if marking objects as dynamic / static, could it be an idea to future proof this and have a generic flag rather than labelling it specifically for shadows?
I ~~know~~ believe :smile: historically @reduz was against static / dynamic flags, but once you add them, they open up the door to a bunch more optimizations, and we probably don't want to end up in situation with multiple static / dynamic flags.
(NOTE: This is badly remembered second-hand info from @Calinou from a few years ago so could be "lost in translation" :smile: . This could have been referring to a particular case, and besides, Godot has evolved a lot over the years. If I remember right the idea was to keep Godot as easy to use as possible and with as few options as possible to keep things simple. This is understandable - as time goes on there are inevitably more and more settings - paralysis of choice is a real problem.)
(An alternative which I think @Calinou has suggested in the past is to automatically detect static objects, but that could be a bit cumbersome in some cases.)
It might also be an idea to have the static / dynamic flag as inherited / static / dynamic, with default inherited, that way users could easily mark a branch as static without marking every mesh.
Just food for thought, if marking objects as dynamic / static, could it be an idea to future proof this and have a generic flag rather than labelling it specifically for shadows?
I think this new flag makes sense to be called a generic rendering flag, but there's definitely some demand for keeping the GI and general rendeering flags separate (cc @embyrdev). How do other engines handle this in comparison?
@lawnjelly indeed @clayjohn was discussing with me the merits of introducing a flag for broader use. I kept it straight forward for now just to get the basics working and tested but we can see where we want to take this.
I'm interested to know what @reduz suggestion was to not have the flag. We can infer what objects are dynamic to some extend, especially if they are children of physics objects, but there are a lot of situations where we won't know if an object will move until code moves it. The overhead of keeping track of which objects are rendered to a light and which have changed from static to dynamic would be intense not to mention we don't capture situations where an object is just moved one time or as part of a level change.
How do other engines handle this in comparison?
Unity has a single per-object static checkbox, that actually sets a bunch of flags for different systems. You can also set it per-system.
When changing these values, the user is asked whether to assign it to the current object only, or to its children as well.
Personally, I think it's a nice workflow. For basic uses, it's a simple "toggle this for free optimization" check, while also offering advanced options.
Right now I'm going in the opposite direction, everything is static by default and you need to mark dynamic objects, seeing 90% of your scene will be static, this way the amount of work is lessened. The problem is that it also means things are "broken" until you mark the correct objects as dynamic.
I do believe the answer lies with good IDE functions, configuration warnings if your a child of a physics body and dynamic is not selected, indeed the ability to make a whole tree as dynamic/static.
Maybe this should work like our pause function, where the default mode is "inherited from parent" and we go down the tree. If we come all the way down to the root level it's static, if we encounter a physics body it's dynamic, if we encounter a parent that has static of dynamic set, we use that?
Maybe this should work like our pause function, where the default mode is "inherited from parent" and we go down the tree. If we come all the way down to the root level it's static, if we encounter a physics body it's dynamic, if we encounter a parent that has static of dynamic set, we use that?
I was just about to suggest this exact thing. With this, it would be pretty easy to create scene configurations that "just work". For example making basic Statics / Dynamics nodes near the root. You would then have to enable the dynamic mode only on a single node and the children would then automatically inherit that (unless they specify otherwise).
What happen when using lightmaps and what about harirchy? For example in a complex scene you have to go to all meshes and set them static even if a parent node is set to static? Could this be done not in a mesh but in a 3DNode and children inherit from it?
I do believe the answer lies with good IDE functions, configuration warnings if your a child of a physics body and dynamic is not selected, indeed the ability to make a whole tree as dynamic/static.
Not sure, i think it should be automatic, to avoid the user having to click one by one the dynamic mesh. Also what would happen with 4.0 projects opened in 4.1, will they be static automatically?
Also what would happen with 4.0 projects opened in 4.1, will they be static automatically?
Yes, but this feature isn't slated to be merged for 4.1 due to feature freeze.
everything is static by default and you need to mark dynamic objects
I agree with of your comment, except for this part. Making 'static' the default will lead to confusion and unexpected behaviors, specially for newcomers.
If I place a 3d object on scene and move it by code, the behaviour should be correct by default. as I see it, performance is "optional", correctness is not.
If I place a 3d object on scene and move it by code, the behaviour should be correct by default. as I see it, performance is "optional", correctness is not.
The problem though is that you're introducing a hell of a lot of workload, the vast majority of objects in a scene will be static. Not setting an object to dynamic that should be dynamic has an immediate visual effect that the user can correct. Not setting an object to static that should be static goes completely unnoticed, it becomes like finding a needle in a haystack with no queue's to the user that there is performance to be gained.
I think the idea to inherit the behaviour has a lot of merit especially when some of this is automatic (i.e. a mesh instance as a child of a rigid body becomes dynamic automatically) but I was originally planning on only doing this for geometry nodes.
We could make it a property of Node3D I guess but that may lead to confusion when there is a property that doesn't do anything on most nodes other than pass information up the tree.
What happen when using lightmaps and what about harirchy? For example in a complex scene you have to go to all meshes and set them static even if a parent node is set to static? Could this be done not in a mesh but in a 3DNode and children inherit from it?
Any input on this?
What happen when using lightmaps and what about harirchy? For example in a complex scene you have to go to all meshes and set them static even if a parent node is set to static? Could this be done not in a mesh but in a 3DNode and children inherit from it?
Any input on this?
Right now nothing much to say because we're separating the settings for GI/lightmaps and normal lights.
To the best of my knowledge, when you mark a light as having to be baked, that light no longer gets evaluated as the light information is now coming from the lightmap or from VoxelGI or SDFGI. So that light in effect is only static at that point in time. Though I might be overlooking something here, I'm not sure if dynamic objects are still handled through shadowmaps, I haven't dived into this much.
If that is the case, only objects marked as dynamic for GI will be handled by the normal shadowmap system which will then divide that remainder in what needs to be rendered into the static shadow map (hopefully nothing) and what goes into the dynamic shadow map. There is no additional overhead if nothing is rendered to the static shadowmap, the logic is able to skip over that nicely.
That is a bigger discussion however and comes down to whether it makes sense to have a separate setting for GI and for shadowmaps, or if we should have a single setting that just markes a node as static or dynamic.
The problem though is that you're introducing a hell of a lot of workload, the vast majority of objects in a scene will be static. Not setting an object to dynamic that should be dynamic has an immediate visual effect that the user can correct. Not setting an object to static that should be static goes completely unnoticed, it becomes like finding a needle in a haystack with no queue's to the user that there is performance to be gained.
The default could always be different for things created inside Godot and imported scenes. I doubt the performance really matters in level blockouts and I'd rather have my crappy tween driven cube doors and other placeholders work without any extra effort. Would also mean you'd only have to worry about the value during import process which I think would be the natural time and place to set the value anyway (good spot to do any fancy auto-detection too).
Though I might be overlooking something here, I'm not sure if dynamic objects are still handled through shadowmaps, I haven't dived into this much.
When using LightmapGI, both objects with static and dynamic GI cast shadows (and therefore incur shadow draw calls). In a scenario with fully baked lights, this allows dynamic objects to receive accurate shadows from the world.
This however has more performance overhead compared to skipping light rendering entirely for static lightmapped lights. Eventually, we'll need to add a 4th "Fully Static" GI mode that skips light rendering entirely (except when baking the lightmap) and bakes the direct light in the dynamic object capture system (lightmap probes). This could also be supported for VoxelGI and SDFGI, but it's not as important there due to the light data's density generally being too low. The current way of rendering static lights in LightmapGI should remain available as an option though, since it allows for accurate shadow casting onto dynamic objects (which can be valuable in many cases but with better performance than baking indirect light only).
Ok omni and spot lights now render their shadow maps properly with the dynamic and static split. Cubemap omni lights still have their two step approach but I think I found a good compromise here in the approach. I've also updated displaying the shadow atlas to show what part of the map reflects dynamic objects by drawing those in blue and inverting their depth.
Still trying to find an issue with the dynamic cubemap being copied in the wrong order..
Ok, finally found why it was messing up cubemap shadows. Turns out that the originally logic will overwrite the projection matrix of the light once the cubemap is converted to paraboloid. Now that we're potentially doing two passes, the second pass (updating dynamic shadows) would use the wrong projection matrix.
So I've now made it so that it only does this when it's finished updating the dynamic shadows.
In doing so I also figured out it was always updating static maps when objects moved or it was detected they moved into/out of a lights cull frustum. I've changed this so it only does this when we're dealing with static objects.
Now that leads to an interesting issue. If you forget to mark an object as dynamic, it will trigger a full update of the shadowmap and thus cost MORE performance instead of optimising without any visual indication you've done something wrong. I suggest we either:
- warn the user that an object marked as static was moved (but only in runtime, we want this behaviour in editor)
- not mark the light as dirty (unless in editor), showing that the shadow of the object doesn't update when moved
Note The above system does present an interesting alternative worth discussing. We could remove the system of tagging dynamic objects all together and make this automatic. It would mean that there is some overhead when we detect an object to be dynamic but a few frames in and everything will be tagged.
With this change I think we have all the core stuff working. Just to recap on what is left to do:
- Improve on the dynamic/static setting on geometry, possibly my suggestion of an "inherited from parent" setting. Also need to decide on final name and placement of this setting and default behaviour.
- Make a setting to turn this feature on/off.
- Possibly y-flip our debug image
- Implement directional light version
@lawnjelly By the way, I wonder if the approach in this PR is compatible with https://github.com/godotengine/godot/pull/33340 which was attempted a while ago.
Note if animations are linked to a mesh, we automatically recognize it as dynamic.
Is this based on AnimationPlayer or skeleton/blend shape presence? Does it apply if the animation isn't currently playing?
@lawnjelly By the way, I wonder if the approach in this PR is compatible with https://github.com/godotengine/godot/pull/33340 which was attempted a while ago.
The snag I faced when writing the PR was that the spotlights and omnis had a very controversial existing "optimization" : the objects were paired with the light, and the shadow map was only re-rendered if either the objects or the light moved. The idea I guess is that if objects in view of the light weren't moving, no need to re-render the shadow map.
My problem with this is that although this situation might occur in demos / benchmarks, in real games, things move, and you tend to be re-rendering shadow maps more than they can be reused. And you were losing out by this technique, because a spotlight might catch 1000 objects in its' frustum, but only 10 of which were also in the camera frustum. So you could easily end up rendering 10-100x as many object shadow maps than were actually necessary.
So while this lazy updating was an optimization in some situations like static scenes (e.g. benchmarking), in real games, there was a danger of it being a "de-optimization".
If, as in this PR, you decide you are always going to re-render dynamic objects if they are in view of the light, you can always cull them tightly to the camera frustum (as in #33340) , so yes, this technique is directly applicable, especially as dynamic objects can be more expensive to render if they are e.g. skinned.
Is this based on AnimationPlayer or skeleton/blend shape presence? Does it apply if the animation isn't currently playing?
Not sure, it was existing code
In doing so I also figured out it was always updating static maps when objects moved or it was detected they moved into/out of a lights cull frustum. I've changed this so it only does this when we're dealing with static objects.
As for this, discussed a possible approach with Clay earlier today that I'm working on atm where I'm going to trial this.
In the render server we'll now have 3 states for each instance: auto, static and `dynamic. The default being auto.
Then in our culling implementation:
- If an object is paired to light, we mark light dirty causing static maps to re-render, but we leave the mode alone
- If an object is unpaired from light and is auto, mark it as dynamic and mark light dirty
- If an object that is paired to light, is auto and moves, mark it as dynamic and mark effected lights as dirty
This means that in a few frames all objects that should be dynamic will automatically be marked as dynamic. In the editor this will often result in static objects becoming dynamic until a scene reloads, but I don't see that as a problem but it will need to be in the documentation.
It also means that if you mark an object as dynamic from the start, rendering happens correct from the start. It also means that if you mark an object as static, we keep treating it as static even if you move it. This can be desired in certain cases where an object may move once.
The status is only changed within the rendering server, on the node we keep the value as set by the user.
For auto mode:
- When an object is first added, if you treat it as static (until moved again) then fast lifetime objects (e.g. bullet hell) will continually cause static shadow map redraws, negating the benefits.
- If you treat it as dynamic until it has stayed still for e.g. a couple of seconds, then this could work (you could also do a "bulk update" of the static shadows), but might be a bit more complex to get working, as you'd need some kind of timer list on moving the auto objects from dynamic to static. (This could also potentially work with physics objects that are sleeping until moved by another object.. so could go dynamic -> static -> dynamic multiple times)
If the light didn't move, anything that enters it's list is dynamic.. One only needs to update the "Static List" when the light is Spawned and when it moves, and when a user marked static object is inserted, otherwise treat as dynamic.
Likewise it'd make sense to make sure that you know the relative transforms of Light and Static objects so that you can determine if they're moving in sync, thus not needing to be updated, however this is likely a future optimization
@lawnjelly the problem is that I'm trying to fit this in with the current way things are working without rewriting the entire system. No automatic logic is going to be perfect, at some point it will be up to the user to finetune things by manually marking things as static or dynamic in edge cases where the automatic logic gets it wrong (like your bullet hell example).
@mrjustaguy kind of the same answer, the list is re-build every frame as part of the culling logic. I'm trying to stay within the current confines of the rendering engine without a complete redesign. In essence what I'm attempting has the same result but relying on the current dependency approach where changes to position are broadcast
Ok, the automatic logic seems to work decently enough, the biggest problem I've identified atm is that when you turn around, our current logic for identifying which lights are assigned to the shadow maps pretty aggressively reorders them as lights enter/exit the frustum. This will need to be improved as everytime this happens the static maps are updated and we loose any benefit from this approach.
@Calinou we also need to think about how to benchmark this, we definately need to see if the change does not have negative effects when this approach is turned off, and we need to think of some way that tests performance both in best and worse case scenarios (i.e. player moving forward, vs player turning). I wonder if we can come up with a more efficient test project than I'm using right now as it's hard to tell how much the shadowmap rendering impacts performance when physics and normal rendering get in the way.
I'm also still looking into a GPU validations error but it may be one that I've already solved in another PR.
Marking this PR ready for review as I think all the core logic is in place.