Animated meshes disapear if their original position is not within the field of view

[nicopap]

Bevy version

cdb62af4bf52 (main as of 2022-06-09)

What you did

I've a giraffe where the eyes are a different mesh from the rest of the body, and when the neck moves enough, it's possible that the eyes after application of animations are so far away from their initial position that the engine thinks it's outside of frustum, while they should be within view, and doesn't render them.

Video demonstrating the bug:

https://user-images.githubusercontent.com/26321040/172806192-d2bcb442-0a93-4e0d-a841-2c19edb34506.mp4

As you can observe, the blue eyes disappear if the original eye position are not within the field of view.

The rest of the mesh (skin) also disappear when the camera view doesn't include the original mesh's position.

Additional information

This is because entity mesh AABBs are not computed based on the animation. The animation shader runs on the GPUs, while AABB filtering runs on CPU before anything is sent to the GPU. There is no way for bevy to filter meshes based on the animation position.

This seems to be something barely possible if not impossible to fix within bevy. Suggestions include:

• Pre-processing step for generating AABB before loading models, accounting for joint limits or pre-backed animations.
• Cunningly associate the invert-bind-pose to the vertices it's supposed to transform, create AABBs for each bones, and use that AABB with the transform of the bone for culling of animated meshes. (note that it's not a full solution and it's fairly expensive). This is a rough idea and I'm not sure if it can even work.

Workaround

There are two options if you hit this issue:

• Add the NoFrustrumCulling component to the entities which mesh disappear because of animation.
• Change the Aabb manually so that it can cover the full range of possible visible position for the model. (note that this workaround might break when #4944 lands)

[aevyrie]

A simple starting point might be to use a conservative bounding sphere of the longest bone chain to generate an AABB: https://gamedev.stackexchange.com/questions/43986/calculate-an-aabb-for-bone-animated-model

[Shatur]

This also causes raycast to work incorrectly for animated meshes (because actual mesh could be in a different location). @aevyrie is it something that should be addressed in raycast implementation or could be fixed on Bevy side?

[nicopap]

The problem with raycasting is double:

• raycasting uses AABB to narrow down which mesh to look at, if the AABB is bogus, well it won't pick up the right meshes when narrowing down
• raycast uses the bevy Mesh, which is the CPU side representation of the model, the CPU-side representation has no awareness of the position of vertices when animated, so it will always be wrong

If you content yourself with CPU-side raycasting, you'll have to accept either ridiculous performance loss or inaccurate (too broad) collision detection.

But there is a way! This is where GPU picking is useful. GPU picking constructs a buffer with individual ID per entity, you can then read that buffer for the pixel under the cursor, and you have the picked entity!

Issue is I don't think someone has yet made a GPU picking implementation in bevy.

[torsteingrindvik]

As a note, I had a GPU picking impl that worked with skinned meshes as well a while ago: https://github.com/bevyengine/bevy/pull/6991

it is superseded by @IceSentry 's PR here: https://github.com/bevyengine/bevy/pull/8784

I see the most recent comment there by @mtsr is to consider having GPU picking as a pre-pass. I don't know how/when/where culling happens so I'm not sure how relevant it is for this issue.

[coreh]

Workaround snippet, for convenience:

use bevy::{
    prelude::*,
    render::{mesh::skinning::SkinnedMesh, view::NoFrustumCulling},
};

/// System that automatically disables frustum culling for
/// all skinned meshes, as soon as they are added to the world.
fn disable_culling_for_skinned_meshes(
    mut commands: Commands,
    skinned: Query<Entity, Added<SkinnedMesh>>,
) {
    for entity in &skinned {
        commands.entity(entity).insert(NoFrustumCulling);
    }
}

Then just add it as a system:

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_systems(Update, disable_culling_for_skinned_meshes);
}

[Sirmadeira]

This issue doesn't occur solely with animations. It occurs with any type of entity that has a skeleton animation. In my case scenario, I have a player following camera. And if i stop looking at the mesh boom, no rendering. Or shift the orbit in y boom half rendering.