Improve frustum culling of skinned meshes through per-joint bounds

[greeble-dev]

Objective

Mostly fix #4971 by adding a new option for updating skinned mesh Aabb components from joint transforms.

https://github.com/user-attachments/assets/c25b31fa-142d-462b-9a1d-012ea928f839

This fixes cases where vertex positions are only modified through skinning. It doesn't fix other cases like morph targets and vertex shaders.

The PR kind of upstreams bevy_mod_skinned_aabb, but with some changes to make it simpler and more reliable.

Dependencies

• #21732 (or something similar) is desirable to make the new option work with RenderAssetUsages::RENDER_WORLD-only meshes.
  • This PR is authored as if 21732 has landed. But if that doesn't happen then I can adjust this PR to note the limitation.
• #21845 adds an option related to skinned mesh bounds.
  • If that PR lands first then I'll incorporate their option into this PR.

Background

If a main world entity has a Mesh3d component then it's automatically assigned an Aabb component. This is done by bevy_camera or bevy_gltf. The Aabb is used by bevy_camera for frustum culling. It can also be used by bevy_picking as an optimization, and by third party crates.

But there's a problem - the Aabb can be wrong if something changes the mesh's vertex positions after the Aabb is calculated. The most common culprits are skinning or morph targets. Vertex positions can also be changed by mutating the Mesh asset (#4294), or by vertex shaders.

The most common solution has been to disable frustum culling via the NoFrustumCulling component. This is simple, and might even be the most efficient approach for apps where meshes tend to stay on-screen. But it's annoying to implement, bad for apps where meshes are often off-screen, and it only fixes frustum culling - it doesn't help other systems that use the Aabb.

Solution

This PR adds a reliable and reasonably efficient method of updating the Aabb of a skinned mesh from its animated joint transforms. See the "How does it work" section for more detail.

The glTF loader can add skinned bounds automatically if a new GltfSkinnedMeshBoundsPolicy option is enabled in GltfPlugin or GltfLoaderSettings:

app.add_plugins(DefaultPlugins.set(GltfPlugin {
    skinned_mesh_bounds_policy: GltfSkinnedMeshBoundsPolicy::Dynamic,
    ..default()
}))

The new option is enabled by default. I think this is the right choice for several reasons:

• Bugs caused by skinned mesh culling have been a regular pain for both new and experienced users. Now the most common case Just Works(tm).
• The CPU cost is modest (see later section), and sophisticated users can opt-out.
• GPU limited apps might see a performance increase if the user was previously disabling culling.

For non-glTF cases, the user must ask Mesh to generate the skinned bounds, and then add the DynamicSkinnedMeshBounds marker component to their mesh entity.

mesh.generate_skinned_mesh_bounds()?;
let mesh_asset = mesh_assets.add(mesh);
entity.insert((Mesh3d(mesh_asset), DynamicSkinnedMeshBounds));

See the custom_skinned_mesh example for real code.

There's some downsides to the new approach:

• It's broken by RenderAssetUsages::RENDER_WORLD because the data is inside Mesh. I'm relying on #21732 to fix this.
• It only accounts for skinning - morph targets, vertex shaders and other things that modify vertex positions are not accounted for.

Bonus Features

GltfSkinnedMeshBoundsPolicy::NoFrustumCulling

This is a convenience for users who prefer the NoFrustumCulling workaround, but want to avoid the hassle of adding it after a glTF scene has been spawned.

app.add_plugins(DefaultPlugins.set(GltfPlugin {
    skinned_mesh_bounds_policy: GltfSkinnedMeshBoundsPolicy::NoFrustumCulling,
    ..default()
}))

PR #21845 is also adding an option related to skinned mesh bounds. I'm fine if that PR lands first - I'll update this PR to include the option.

Gizmos

bevy_gizmos::SkinnedMeshBoundsGizmoPlugin can draw the per-joint AABBs.

fn toggle_skinned_mesh_bounds(mut config: ResMut<GizmoConfigStore>) {
    config.config_mut::<SkinnedMeshBoundsGizmoConfigGroup>().1.draw_all ^= true;
}

The name is debatable. It's not technically drawing the bounds of the skinned mesh - it's drawing the per-joint bounds that contribute to the bounds of the skinned mesh.

Testing

cargo run --example test_skinned_mesh_bounds

# Press `B` to show mesh bounds, 'J' to show joint bounds.
cargo run --example scene_viewer --features "free_camera" -- "assets/models/animated/Fox.glb"
cargo run --example scene_viewer --features "free_camera" -- "assets/models/SimpleSkin/SimpleSkin.gltf"

# More complicated mesh downloaded from https://github.com/KhronosGroup/glTF-Sample-Assets/tree/main/Models/RecursiveSkeletons
cargo run --example scene_viewer --features "free_camera" -- "RecursiveSkeletons.glb"

cargo run --example custom_skinned_mesh

I also hacked custom_skinned_mesh to simulate awkward cases like rotated and off-screen entities.

How Does It Work?

Click to expand

Summary

Mesh::generated_skinned_mesh_bounds calculates an AABB for each joint in the mesh - the AABB encloses all the vertices skinned to that joint. Then every frame, bevy_camera::update_skinned_mesh_bounds uses the current joint transforms to calculate an Aabb that encloses all the joint AABBs.

This approach is reliable, in that the final Aabb will always enclose the skinned vertices. But it can be larger than necessary. In practice it's tight enough to be useful, and rarely more than 50% bigger.

This approach works even with non-rigid transforms and soft skinning. If there's any doubt then I can add more detail.

Awkward Bits

The solution is not as simple and efficient as it could be.

Problem 1: Joint transforms are world-space, Aabb is entity-space.

• Ideally we'd use the world-space joint transforms to calculate a world-space Aabb, but that's not possible.
• The obvious solution is to transform the joints to entity-space, so the Aabb is directly calculated in entity-space.
  • But that means an extra matrix multiply per joint.
• This PR calculates the Aabb in world-space and then transforms it to entity-space.
  • That avoids a matrix multiply per-joint, but can increase the size of the Aabb.

Problem 2: Joint AABBs are in a surprising(?) space.

• When creating joint AABBs from a mesh, the intuitive solution would be to calculate them in joint-space.
  • Then the update just has to transform them by the world-space joint transform.
• But to calculate them in joint-space we need both the bind pose vertex positions and the bind pose joint transforms.
  • These two parts are in separate assets - Mesh and SkinnedMeshInverseBindposes - and those assets can be mixed and matched.
  • So we'd need to calculate a SkinnedMeshBoundsAsset for each combination of Mesh and SkinnedMeshInverseBindposes.
  • (bevy_mod_skinned_aabb uses this approach - it's slow and fragile.)
• This PR calculates joint AABBs in mesh-space (or more strictly speaking: bind pose space).
  • That can be done with just the Mesh asset.
• One downside is that the update needs an extra matrix multiply so we can go from mesh-space to world-space.
  • However, this might become a performance advantage if frustum culling changes - see the "Future Options" section.
• Another minor downside is that mesh-space AABBs (red in the screenshot below) tend to be bigger than joint-space AABBs (green), since joints with one long axis might be at an awkward angle in mesh-space.

Future Options

For frustum culling there's a cheeky way to optimize and simplify skinned bounds - put frustum culling in the renderer and calculate a world-space AABB during extract_skins. The joint transform will be already loaded and in the right space, so we can avoid an entity lookup and matrix multiply. I estimate this would make skinned bounds 3x faster.

Another option is to change main world frustum culling to use a world-space AABB. So there would be a new GlobalAabb component that gets updated each frame from Aabb and the entity transform (which is basically the same as transform propagation and the relationship between Transform and GlobalTransform). This has some advantages and disadvantages but I won't get into them here - I think putting frustum culling into the renderer is a better option.

(Note that putting frustum culling into the renderer doesn't mean removing the current main world visibility system - it just means the main world system would be separate opt-in system)

Performance

Click to expand

Initialization

Creating the skinned bounds asset for Fox.glb (576 verts, 22 skinned joints) takes 0.03ms. Loading the whole glTF takes 8.7ms, so this is a <1% increase.

Per-Frame

The many_foxes example has 1000 skinned meshes, each with 22 skinned joints. Updating the skinned bounds takes 0.086ms. This is a throughput of roughly 250,000 joints per millisecond, using two threads.

The whole animation update takes 3.67ms (where "animation update" = advancing players + graph evaluation + transform propagation). So we can kinda sorta claim that this PR increases the cost of skinned animation by roughly 3%. But that's very hand-wavey and situation dependent.

This was tested on an AMD Ryzen 7900 but with TaskPoolOptions::with_num_threads(6) to simulate a lower spec CPU. Comparing against a few other threading options:

• Non-parallel: 0.141ms.
• 6 threads (2 compute threads): 0.086ms.
• 24 threads (15 compute threads): 0.051ms.

So the parallel iterator is better but quickly hits diminishing returns as the number of threads increases.

Future Options

The "How Does It Work" section mentions moving skinned mesh bounds into the renderer's skin extraction. Based on some microbenchmarks, I estimate this would reduce non-parallel many_foxes from 0.141ms to 0.049ms, so roughly 3x faster. Requiring AVX2 (to enable broadcast loads) or pre-splatting (to fake broadcast loads for SSE) would knock off another 25%. And fancier SIMD approaches could do better again.

There's also approaches that trade reliability for performance. For character rigs, an effective optimization is to fold face and finger joints into a single bound on the head and hand joints. This can reduce the number of joints required by 50-80%.

FAQ

Click to expand

Why can't it be automatically added to any mesh? Then the glTF importer and custom mesh generators wouldn't need special logic.

bevy_mod_skinned_aabb took the automatic approach, and I don't think the outcome was good. It needs some surprisingly fiddly and fragile logic to decide when an entity has the right combination of assets in the right loaded state. And it can never work with RenderAssetUsages::RENDER_WORLD.

So this PR takes a more modest and manual approach. I think there's plenty of scope to generalise and automate as the asset pipeline matures. If the glTF importer becomes a purer glTF -> BSN transform, then adding skinned bounds could be a general scene/asset transform that's shared with other importers and custom mesh generators.

Why is the data in Mesh? Shouldn't it go in SkinnedMesh or SkinnedMeshInverseBindposes?

That might seem intuitive, but it wouldn't work in practice - the data is derived from Mesh alone. SkinnedMesh doesn't work because it's per mesh instance, so the data would be duplicated. SkinnedMeshInverseBindposes doesn't work because it can be shared between multiple meshes.

The names are a bit misleading - Mesh does contain some skinning data, while SkinnedMesh and SkinnedMeshInverseBindposes are more like joint bindings one step removed from the vertex data.

Why not put the bounds on the joint entities?

This is surprisingly tricky in practice because multiple meshes can be bound to the same joint entity. So there would need to be logic that tracks the bindings and updates the bounds as meshes are added and removed.

Why is the DynamicSkinnedMeshBounds component required?

It's an optimisation for users who want to opt out. It might also be useful for future expansion, like adding options to approximate the bounds with an AABB attached to a single joint.

Why are the update system and DynamicSkinnedMeshBounds component in bevy_camera? Shouldn't they be in bevy_mesh?

bevy_camera is the owner and main user of Aabb, and already has some mesh related logic (calculate_bounds automatically adds an Aabb to mesh entities). So putting it in bevy_camera is consistent with the current structure. I'd agree that it's a little awkward though and could change in future.

What Do Other Engines Do?

Click to expand

• Unreal: Automatically uses collision shapes attached to joints, which is similar to this PR in practice but fragile and inefficient. Also supports various fixed bounds options.
• Unity: Fixed bounds attached to the root bone. Automatically calculated from animation poses or specified manually (documentation).
• Godot: Appears to use roughly the same method as this PR, although I didn't 100% confirm. See MeshStorage::mesh_get_aabb and RendererSceneCull::_update_instance_aabb.
• O3DE: Fixed bounds attached to root bone, plus option to approximate the AABB from joint origins and a fudge factor.

An approach that's been proposed several times for Bevy is copying Unity's "fixed AABB from animation poses". I think this is more complicated and less reliable than many people expect. More complicated because linking animations to meshes can often be difficult. Less reliable because it doesn't account for ragdolls and procedural animation. But it could still be viable for for simple cases like a single self-contained glTF with basic animation.

[janhohenheim]

I very much like the approach presented here and think it's a very reasonable default.

The policy enum should be expanded to include @pcwalton's approach: don't generate any AABB, but keep frustum culling, since all AABBs will be artist-authored and inserted manually

[pcwalton]

Oh nice, I just filed #21845 for the rest pose approach.

[greeble-dev]

The policy enum should be expanded to include @pcwalton's approach: don't generate any AABB, but keep frustum culling, since all AABBs will be artist-authored and inserted manually

I'd like to consider some different approaches:

enum GltfSkinnedMeshBoundsPolicy {
    ...
    /// Skinned meshes are assigned this `Aabb` component.
    Fixed { aabb: Aabb },
    /// Skinned meshes are assigned a `SkinnedMeshBounds` component, which will
    /// be used by the `bevy_camera` plugin to update the `Aabb` component.
    ///
    /// The `Aabb` component will be updated to enclose the given AABB relative to the named joint.
    /// If the joint name is `None`, the AABB is relative to the root joint.
    FixedJointRelative { aabb: Aabb3d, joint_name: Option<String> },
}

So Fixed is similar to Unreal's fixed options. FixedJointRelative is basically Unity's approach but more flexible (can be any joint, not just the root). Unfortunately this needs some changes to SkinnedMeshBounds to support bounds being in joint-space.

I'm not sure if I'll try adding these options in this PR as it's already a bit chonky.

[robtfm]

awesome work, really happy to see a solution to such a long-standing issue.

• Dynamic should 100% be the default. I would lean towards replacing NoFrustumCulling with CullingStrategy: Static | Dynamic | None (and possibly other variants)
• joint space / world space / mesh space: would it make sense to consider bone bounding spheres? this would make the transformations very cheap. this could be an extra variant for CullingStrategy above then. considering the aabbs are crossed between spaces anyway, the benefit of using aabbs over spheres is probably minimal?
• separate asset: i did read the FAQ but i still think this should be stored in the mesh
  • I have a pr (#21732) that would remove the RenderAssetUsages::RENDER_WORLD problem (we would keep joint bounds data cpu-side even after the vertex data is extracted - that doesn't need any skin data to compute, just the positions and joint weights from the vertex data)
  • that leaves the issue of having to look up the mesh to perform updates. that could maybe be improved by adding bounding sphere / aabb data directly to the joint entities when they are spawned? i haven't thought through the space issues here but it seems clean / logical
  • we could then auto-apply for non-gltf meshes as well

[greeble-dev]

joint space / world space / mesh space: would it make sense to consider bone bounding spheres? this would make the transformations very cheap. this could be an extra variant for CullingStrategy above then. considering the aabbs are crossed between spaces anyway, the benefit of using aabbs over spheres is probably minimal?

I briefly looked at this, but concluded that joint transforms being affine kinda spoils things - bounding spheres would end up only slightly cheaper. But I didn't work it through in detail or benchmark.

There could also be an option where scale is ignored and the bounding sphere is always at zero in joint-space. That would be significantly cheaper but requires a sophisticated user to understand the risks.

separate asset: i did read the FAQ but i still think this should be stored in the mesh. I have a pr that would remove the RenderAssetUsages::RENDER_WORLD problem

Sounds good to me. I think I'd end up adding a HasDynamicSkinnedMeshBounds marker component so the update only touches meshes with dynamic bounds.

that leaves the issue of having to look up the mesh to perform updates. that could maybe be improved by adding bounding sphere / aabb data directly to the joint entities when they are spawned? i haven't thought through the space issues here but it seems clean / logical

The catch there is that multiple meshes can bind to one joint, so whatever's setting up the joints has to be careful to track the exact set of meshes that are bound to each one and merge the AABBs. This is fairly straightforward if done once in the glTF importer and the user leaves it untouched. But rapidly gets messy if the user starts changing stuff - e.g. swapping clothes/accessories on a character.

[greeble-dev]

So my current state is:

• There's been enough feedback that I feel confident finishing up the PR.
  • This would include dropping the separate asset and moving the data back into Mesh (unless I find a blocker).
  • So I'll wait until #21732 lands.
• I'll try waiting until #21845 lands, but might go ahead if that stalls for some reason.

[robtfm]

if you're eager or 21732 stalls you could go ahead with the mesh embedding knowing it will only work for MAIN_WORLD assets initially. i can amend that to store the joint bounds as well if this lands first.

[greeble-dev]

PR is now ready for review.

The bounds data has been moved to Mesh, so it will break if the mesh only has RenderAssetUsages::RENDER_WORLD. This will be fixed by #21732. If that PR doesn't land within the same release cycle then the docs and release notes will need updating.

I also added an example based test - see the video at the top of the PR. I'm not entirely sure this is justified as it's a manual visual test, and not well suited to screenshot comparison. bevy_mod_skinned_aabb has a more comprehensive automated test that generates random meshes and checks them against the skinned vertices. But it's +1000 lines of code, so maybe a bit much. I could make a follow up PR if anyone feels more automated testing should be explored. I can also remove the current test from this PR if it's a bit much to review.