Skinned meshes

[Adngel]

Feature: Adding new skinned meshes to TEN engine

Descriptions:

The objective of this project is to introduce skinned meshes into the TEN engine, enhancing character models' realism and animation quality. Unlike traditional models (with the skin joints method), skinned meshes allow for better blending and more natural animations, as seen in the upcoming Tomb Raider remaster.

(images from the future Tomb Raider 1,2,3 remaster)

An idea to accomplish this, we would have to work on different areas: TEN code, Shaders, WadTool and Tomb Editor.

TEN Engine

• Modify the item class to accommodate skinned meshes optionally (like a new container list).
• Enable the engine to detect and use skinned meshes for rendering if available; otherwise, continue using bone meshes.

Shaders

• Create a new vertex shader capable of interpreting bone weight data for mesh blending. (For example: the vertex would start with the transformation of the first bone, and then it would go adding transformations of their other linked bones, scaled by their weight, until get the final transformation).

WadTool

This should be critical for the resource management. Ideally, these new system should be adapted to can be imported using the existing WadTool. There may be many ways to do it. The idea I propose is create a new layer for the existing objects to separate the animation skeleton from the rendered model. (But using still the same object slot).

Currently, Lara uses different slots some for her animations, and others for her outfits (SKIN, SKIN_JOINTS, HAIR). Using this method for every enemy or item in the game, could make things very long lists and ardous.

So implementing a layer system could organize and simplify options for the users

• whole object (for visual and collision boxes).
  • its skeleton (for animations, bones position and sphere colliders).
  • its render (for the bones meshes, or if it has, skinned meshes).

Also each layer could have import/export options related to them (so we can clean the crowded population of buttons in the interface).

Tomb Editor

• Update the Tomb Editor to render skinned meshes when applicable, replacing bone meshes.

Motivation:

• Anticipate user expectations due to the upcoming Tomb Raider Remaster, which includes updated models.
• Improve animation quality, enabling artists to create more realistic and immersive character animations.

Key Considerations:

• Implement this feature gradually to ensure a smooth transition.
• Avoid disrupting existing workflows and keep legacy compatibility.
• Seamless integration into existing tools is a priority.

Plan Overview

1. Update Item Structures

• Create a new list for the skinned meshes loaded in the level. (A list of Skinned Mesh class objects).
• Modify the item class to accommodate skinned meshes as optional components. (surelly, a list of pointers). (be aware may change in runtime, must be compatible with save/load).
• Ensure backward compatibility by allowing items to have both skinned or bone meshes if the skinned meshes are not available.

2. Skinned Mesh Class

• Create a dedicated skinned mesh class capable
  • Develop data structures to manage vertex weighting and textures materials.
  • Implement functions for their management (swapping, destruction, getters and setters, etc).
  • Recreate functionalities needed from traditional bone meshes for these new meshes work too. (Render queque management)

3. Renderer Integration

• Update the rendering logic when enable the engine to detect and use skinned meshes for rendering if they are available for a particular item.
• Implement fallback mechanisms to continue using bone meshes for items without skinned equivalents.
• During drawing stage, ensure the new skinned meshes are working properly in the render queques (for opaques and alpha blending).

4. Level Flow load

• Modify the level load function to allow it read the new skinned meshes from the wad and include them into the level skinned meshes container. (So they can be instantiated by the item).

5. Vertex Shader

• Develop a custom vertex shader specifically for skinned mesh rendering.
• Implement vertex transformations that account for bone weights and hierarchy during vertex blending.
• Ensure seamless integration with the existing shader system in TEN.
• Ensure it can keeps using the existing pixel shader for items.

6. WadTool Update

• Integrate skinned meshes into the resource management system.
• Propose and implement a layer system within WadTool to categorize and organize skinned mesh data separately from animation skeletons and rendered models.
• Enhance asset import and export capabilities to handle skinned mesh data and provide more options (export skeleton or render meshes only, or combined).
• Design an intuitive user interface for easy access to different layers.

7. Tomb Editor Update

• Update the Tomb Editor to recognize and render skinned meshes when applicable.

Diagrams

Loading

Drawing Flow

WadTool

[Lwmte]

Several ideas from me:

1. Skinned mesh should be a separate category in Wad2, so that several objects may refer to the same skinned mesh. This will allow to easily attach same skinned meshes to different objects. Also in final compiled level, skinned meshes should be also kept separately from actual objects. This will allow to easily swap skinned meshes in runtime and somehow replicate old meshswap workflow.

2. There should be no separate skinned mesh editor. Original mesh editor should be just extended with a toolset to edit vertex weights.

3. boneWeights should actually be std::vector<std::map<BoneIndex, Weight>>.

4. Good question about bone mutators - should they affect skinned mesh at all? At first stage, I think we can ignore that, because mutators are usually used with detached meshes, such as rolling wheels, stretching spikes, etc. Later we can think how to adapt mutators to a skinned mesh too.