Yellow-Dog-Man
Blendshape Assets
Is your feature request related to a problem? Please describe.
When making an avatar, the process is usually to make two versions. One with all blendshapes, and one with most of them baked. If you have a NSFW avatar, you usually have to maintain a completely separate version as well, making it 3-4 avatars in total. When you baked a blendshape and want to add it back, you have to load the unbaked version and re-do the whole process.
Additionally, when you add new blendshapes to an avatar at a later point and import the mesh again, the bones may be in a completely different order. For complex avatar, re-assigning them correctly can be a lot of work.
I am not maintaining a NSFW version of my avatar since everything is just a system I can spawn in the world as an item to install. The only thing I can't modify is blendshapes. If I have two systems, which each need their own blendshapes, I can't just dynamically replace the mesh of the avatar due to the different possible combinations.
For me, baking blendshapes is a manual process because the customization options are driving blendshapes which should also be baked. This means I can't use the "bake non-driven blendshapes" button.
Describe the solution you'd like
It would be useful to me if I could dynamically add/remove blendshapes to/from a skinned mesh as long as the vertices match up.
This would also let the game deduplicate different baked avatars using the same base mesh and blendshapes, just having a different combination of them baked, saving on storage space, network, and VRAM.
Describe alternatives you've considered
Maintaining an unbaked version of an avatar and manually baking blendshapes after each change, and a separate baked NSFW avatar.
Additional Context
What would be the implications of this? Would this be viable to implement?
Requesters
Colin The Cat (colin.cat)
We could potentially offer some functionality to "merge" two meshes and add blendshapes from one to another if the vertices match up, but the result of this would be a different mesh altogether, so that wouldn't provide the benefits you're wanting - like deduplicating them.
Essentially what would need to happen is that the system would need to process the mesh in the background to "unbake" it with the data, which would significantly slow down its loading - unless it's just raw blendshape data, but this needs the mesh to be exact.
My concern with that would be that this functionality would be fairly brittle - potentially any mesh operations or changes could just end up reordering some vertices in the binary representation and result in the data just making no sense anymore, because the separated blendshapes become out of date.
We could potentially explore something with the asset variant system though - that would keep everything "in sync" with the same mesh, rather than separating them into their own assets - we could generate the base mesh data and then generate the blendshape data as individual variants and then let you specify which blendshapes you want to load on the asset provider - changing that would still require re-loading the mesh data, but I think this solution would be most viable and flexible.
There are still some caveats though:
- You'd loose performance benefits of any baked non-zero blendshapes - those would need to be computed realtime, because they can't be part of the base mesh data - even if they don't change - unless we bake them onload (which will increase CPU usage of loading and prevent certain other optimizations)
- It could potentially use more memory in practice, if people with the same base mesh load different sets of blendshapes - unless we combine those in the asset variant system, but that's a lot of extra complex engineering potentially
- You still might need separate NSFW version anyways? I'm not sure how this would work out.
From a private conversation with Froox, I think this would add to this discussion:
Context: Loading blendshapes as asset variants.
Biggest downside of that one would be that any static blenshapes that are non-zero would need to be kept as dynamic and re-computed, rather than baked into the geometry.
I didn't even think of this, but of course baking blendshapes modifies the base mesh, which would make proper deduplication impossible.
Option 1: Baked Blendshapes
An idea would be to store some metadata (not the original mesh, that's out of question, see #1028). This metadata could for example be a SHA512 of the original mesh and a dictionary from baked non-zero blendshape to value. This could let the cloud associate baked meshes to the original mesh data.
The problem would then be how to deduplicate loading two baked avatars in a more efficient way using those asset variants. I don't think this would be viable.
Option 2: Removed Blendshapes
Please correct me if this is specific to me. Most of the blendshapes on my avatar are just unused and can be removed without baking them. I have 300 blendshapes. Without facetracking, I can remove 45 of them. I don't need MMD, that's another 20. There are also just a bunch of misc blendshapes I'm not touching, around 100.
This means the asset variant system could deduplicate assets by, for example, hashing the base mesh and throwing all blendshapes into a "bucket" associated to the base mesh on the cloud, allowing only a part of them to be downloaded when needed since they're just separate buffers.
When multiple avatars using the same base mesh are loaded, which are not baked, but are only using some of the blendshapes, only one mesh would need to be loaded for all of them and only the blendshapes which are used by any of the avatars. If the asset cache saved blendshapes separately, then most of them could also be loaded from disk.
Additionally, if we get blendshape caching on the new renderer as stated as a "nice to have" in phase 1, this would come at no additional cost on the GPU.
So, if baking blendshapes was made obsolete by the new renderer, blendshape asset variants would be a good way to save on bandwidth.
This solution would then also allow me to copy a blendshape from one mesh to another as long as the base meshes have the same SHA512 hash (same "bucket"). If this could be done by just using the data model (perhaps a list), it would open up some cool possibilities.
You still might need separate NSFW version anyways? I'm not sure how this would work out.
I'm only concerned about blendshapes which would be added to the avatar mesh for certain features that don't require modifying the mesh. If I had NSFW blendshapes on my otherwise SFW avatar, I don't think I'd be able to use it in SFW sessions.
But of course, if the mesh data itself needed to be changed, then this feature request wouldn't help with deduplication.