Add support for skinning >4 bones per vertex with a bone weight texture

[cstegel]

Overview

Related issue: #26137

High-fidelity skeletal animations often need a large number of bone weights per vertex. The current vertex buffer skinning approach limits meshes to <= 4 bone influences per vertex.

This PR adds an alternative skinning approach to the existing bone index/weight vertex buffers which uses a vertex shader texture to support an arbitrary amount of bone influences per vertex.

For example, a mesh with the following bone weights/indices in vertex buffers...

        WEIGHTS_0        JOINTS_0
        ---------------  --------
[index] 0   1   2   3    0 1 2 3 
    v0: 0   1.0 0   0    0 1 2 3
    v1: 0.1 0.7 0.1 0.1  0 1 2 3

becomes this array/texture of (index, weight) pairs:

        v0                v1
        --------          -----------------------------------
[index] |   0  |     1    |  2        3        4        5   |    6   
  data: (1, 1.0) (-1, -1) (1, 0.7) (0, 0.1) (2, 0.1) (3, 0.1) (-1, -1)
                   ⬆                        ⬆   ⬆ 
              sentinel value         bone index  weight 

and this vertex buffer:

    weights texture start index
    ---------------------------
v0: 0
v1: 2

I have worked on this change as part of my job at Google since my org has projects which would like Three.js to have this feature.

What changed:

1. If a model has more than 1 bone weight buffer (>4 weights per vertex) then a bone weight texture is created and the shader behavior is changed to use it instead of vertex buffer skinning.
2. Vertex weight/index buffers are sorted by weight -- across all buffers -- before normalization in the buffers that will be used.
   • This allows skin texture creation to only include non-zero weights.
   • This "fixes" some artifacts for vertex buffer skinning when models have >4 weights and the higher weights are not in the first 4 weights.
   • This also means that loading skinned meshes takes longer because the per-vertex weights are sorted in addition to being normalized.

What did not change:

1. The vertex buffer skinning approach is still the default for models that have at most 4 bone weights per vertex.

What needs decisions:

1. Default behavior and how to control which skinning method to use
   • Currently defaults to the old buffer method if a model has <= 4 weights otherwise the texture method is used.
   • Loader/mesh options have controls for always or never using the texture approach.
2. Do we still want to keep the old method?
   • The code would be simpler without it and I haven't seen a big performance impact.
   • Main tradeoff seems to be model load time (creating the skinning texture from the vertex buffers)
   • I'm unsure if other parts of three.js or clients require/assume the presence of the skinning buffer

Examples

1. webgl_animation_skinning_many_bone_influences.html - Shows the difference between < 4 and >= 4 bone skinning for a model that was created with 16 bone skinning.
   • Notice:
     • The artifacts on the upper lip (left image, only 4 weights)
     • The difference in how much the nose gets pulled up/down with the mouth movement.

5. webgl_animation_skinning_performance.html - loads many skinned meshes playing animations with a toggle between the old and new behavior to see if there are performance differences.

Performance

All performance numbers are from a 2019 MacBook Pro running Chrome 114.0.5735.106.

Framerate

webgl_animation_skinning_performance.html renders at the same 23 fps for both the vertex buffer and vertex texture skinning methods. The Chrome profiling image below shows that GPU code is executing for about 25% of the frame (11ms / 45ms) and vertex skinning is only a portion of that so this benchmark scene doesn't do the best job of stressing the vertex skinning:

That being said, the soldier model still seems like a realistic asset that someone would use which is why I used it in the benchmarking scene. If there are other animated models with higher vertex counts that would increase GPU vertex shader runtime differences, I am happy to try them.

Note: Texture skinning could have beneficial performance for models with <= 4 weights per vertex because it strips out weights that are zero. For the Soldier.glb model, this removed 38% of the weights.

Model Loading

method	Soldier.glb (7434 vertices, 4 weights) runtime	HeadWithMax16Joints.glb (2474 vertices, 16 weights) runtime
SkinnedMesh.normalizeSkinWeights (dev)	5 ms	N/A
SkinnedMesh.normalizeSkinWeights (this PR)	12 ms	21 ms
SkinnedMesh.createBoneIndexWeightsTexture	7 ms	8 ms
GLTFLoader.load (buffer skinning)	190-380 ms	N/A
GLTFLoader.load (texture skinning)	190-380 ms	240-290 ms

The new implementation of SkinnedMesh.normalizeSkinWeights() is slower because it sorts weights in addition to normalizing them. It could be made faster by sorting each vertex's buffer data in-place instead of copying to separate arrays and then copying them back.

The total load time and variance of the load time was so large that the additional processing in normalizeSkinWeights and createBoneIndexWeightsTexture did not have a noticeable effect.

[github-actions[bot]]

📦 Bundle size

Full ESM build, minified and gzipped.

Filesize dev	Filesize PR	Diff
672.8 kB (166.7 kB)	676.5 kB (167.7 kB)	+3.73 kB

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

Filesize dev	Filesize PR	Diff
452.5 kB (109.3 kB)	454.7 kB (109.9 kB)	+2.23 kB

[WestLangley]

Friendly tip. MeshPhysicalMaterial requires an environment map. I used RoomEnvironment here, but it is your choice. Scene lights are not required. I set material color to white, metalness 0, roughness 0. You also need to set tone mapping and exposure.

[cstegel]

Friendly tip. MeshPhysicalMaterial requires an environment map. I used RoomEnvironment here, but it is your choice. Scene lights are not required. I set material color to white, metalness 0, roughness 0. You also need to set tone mapping and exposure.

Thanks @WestLangley! I've added an environment map and tone mapping now, but I kept the same color and material properties because the smooth material appears too bright in the "performance" scene even with tone mapping. This makes the skinning difference hard to see. Reinhard tone mapping fixed it from being too bright but it doesn't look good (the colors are dull) so I stuck with ACESFilmicToneMapping and the default exposure.

[cstegel]

I do not plan on making more changes until comments are received so I am changing this from a draft PR to a normal PR.

Unit tests, documentation, and full loader support (FBX, Object, Collada, MMD, GLTF) have been added. I did not try loading a >4 bone weight model in those formats because all I have is a .glb, but the existing examples continued to load fine.

[mrdoob]

Sorry for the delay! Looking at this PR now 👀

[cstegel]

@mrdoob Did you get a chance to look at this? I'm happy to make revisions and would love to get this merged.

[sunag]

About WebGPURenderer, I'm thinking in work on an update for the new architecture to support these implementations in Nodes without having to change NodeMaterial codes, I think updates like this could be just adding a new class to the renderer. e.g:

const renderer = new WebGPURenderer();
renderer.skinningClass = SkinningExtendedNode; // SkinningNode as default

// Add other properties for MorphNode, InstanceNode, etc...

What do you think about this? /cc @mrdoob @Mugen87 @LeviPesin

[Mugen87]

I'm not confident yet to say how important more than four bones weights per vertex is as a feature. Hence, I'm not feeling super well with so many core changes at the moment.

Implementing something like this as an addon than can be opted-in sounds preferable to me. At least it's worth to explore this option.

[LeviPesin]

renderer.skinningClass = SkinningExtendedNode

Maybe not assigning class, but rather a node, so introducing something like skinningNode? It would be great to have some API for managing skinning, lighting models, etc...

[TheCodeTherapy]

@mrdoob Did you get a chance to look at this?

[GwilymIO]

🙏 would be great to see this in! 🥰

[cstegel]

Happy belated birthday to this PR 🎂

[Thebluedaredevil]

cstegel--- y'all gotta get this into the glTF format---- destroy that pesky fbx format---- hahahahh--- seriously though, your work on this aspect of glTF is just pure AWESOMENESS--- I'd love to be able to use my 6-20 boneaffect limit skinweights

[cstegel]

cstegel--- y'all gotta get this into the glTF format---- destroy that pesky fbx format---- hahahahh--- seriously though, your work on this aspect of glTF is just pure AWESOMENESS--- I'd love to be able to use my 6-20 boneaffect limit skinweights

Thanks for the appreciation, @Thebluedaredevil :)

The glTF format actually already supports this, but three.js doesn't correctly load those kinds of glTF files. The glTF spec does not have a limit on skinning weight / skinning index buffers. It allows buffers to be created which support multiples of 4 bone influences. Many glTF loaders / renderers -- like three.js -- only look at the first buffer and ignore the rest. When you export glTF files from a tool like Blender, you can choose how many bone influences to include which determines how many glTF buffers are made:

Based on the discussion here and the existence of https://github.com/mrdoob/three.js/pull/28863, I think the three.js maintainers want to rework how extensions like this can be added so I doubt this would go in any time soon.