fix baking compensate_scale on import

[ScanMountGoat]

The relevant code from ssbh_wgpu, which matches the results from testing in game: https://github.com/ScanMountGoat/ssbh_wgpu/blob/d1748b0e9663f47281c51d7f05cde1fbea5b00f6/ssbh_wgpu/src/animation.rs#L67-L83

[ScanMountGoat]

The main differences I can find between Blender and ssbh_wgpu is that Blender doesn't correctly handle getting the scale value from tracks with a single frame and does a change of basis operation to change the bone orientation. It's not clear what the problem is at the moment. ssbh_editor has accurate scaling on version 0.7.0 and fixes to anim decompression, which should make testing easier.

[ScanMountGoat]

SSBH Editor Reference:

Blender 3.0 before fast anim changes with compensate scale fixes applied:

Blender 3.3 on latest commit that already includes compensate scale fixes:

[ScanMountGoat]

It looks like it may have something to do with the new way keyframes are being applied @ssbucarlos. I understand needing to decompose the matrix to insert keyframes, but I'm not sure what all the basis and local matrix manipulation code is supposed to be doing. The old anim code in 3.0 had other scaling related issues, so I wouldn't expect fixing compensate scale to be enough to fix it.

[ssbucarlos]

I am not sure what i did to break it, my intended goal with these changes for fast export was to be able to use the FCurves's keyframe_points's foreach_set method, but the values needed are the matrix basis instead of the parent-relative (?) matrix that was used before.

[ssbucarlos]

SSBH Editor Reference:

Blender 3.0 before fast anim changes with compensate scale fixes applied:

Blender 3.3 on latest commit that already includes compensate scale fixes:

do u have a link to which commit this was where it looked better

[ScanMountGoat]

I haven't narrowed it down to a specific cause yet, but the scale compensation itself seems to be working as intended. The result of raw_matrix = mathutils.Matrix(tm @ scale_compensation @ rm @ scale_matrix) matches the output from ssbh_wgpu at that particular frame other than some rounding errors. I'll do a bit more investigating later and let you know if I find anything.

[ScanMountGoat]

SSBH Editor Reference: Blender 3.0 before fast anim changes with compensate scale fixes applied: Blender 3.3 on latest commit that already includes compensate scale fixes:

do u have a link to which commit this was where it looked better

I was looking at this one. Anything before the "fast anim export" will probably be similar. 1614e0dc102babc2371df738e5f738b4fe98b03b

[ScanMountGoat]

I simplified the code to use Blender's internal calculations for matrix_basis. The results are visually identical to the previous commit. The new code should be identical to the code before the fast animation change now other than how key frames are inserted. The keyframe setting should be the last thing I need to check. I'll need to make sure that decomposing the matrix and updating the fcurves values is working as intended.

[ScanMountGoat]

I'll need to make sure that decomposing the matrix and updating the fcurves values is working as intended.

Sadly, this is almost definitely the issue. This appears to be an issue with how Blender represents the bone transforms. Setting the bone transform matrix and then accessing it again doesn't always result in the same matrix. Blender is likely doing decomposition internally and just exposing matrices for convenience. This is most noticeable with animations with shearing due to non uniform scale. Animations without scale or uniform scaling don't seem to be affected.

ssbh_wgpu with each transform decomposed and then recomposed (matches Blender):

ssbh_wgpu:

[ScanMountGoat]

There may be a way to convert the animation to a format that Blender can actually represent, but that's now a mathematics problem rather than an animation problem. The question is whether there is a skeleton that produces identical accumulated transforms for each bone where each bone's individual transform is an affine transformation consisting only of translation, rotation, and scale. Simply decomposing the original transforms clearly doesn't work in general. It's likely that this can't be solved exactly, so we would need to find a way to at least reduce the errors. @ssbucarlos

[ssbucarlos]

thanks for isolating the issue. Now that you isolated the exact location of the issue, tomorrow ill take a look and see if i can find a solution. In the meantime, we can maybe present users an option to disable fast import for more accurate animations? If this is an acceptable workaround lmk and i can implement that tomorrow.

[ScanMountGoat]

The old importer isnt technically correct either due to the reasons I mentioned. I'll try and take a look tomorrow at seeing if there's a way to keep the fast importer code but have less errors in the imports. Setting the fcurves didn't introduce any additional error after Blender does it's thing behind the scenes when setting the matrix, so there may have been some other difference with the old code. Feel free to play around with the latest commit and see if you can make it look slightly less weird.

Thanks for being flexible and quick to respond about this kind of stuff. There's a really passive aggressive argument that helped me track down the issue that you can read here :)

[ssbucarlos]

honestly its been a while since i took a look at this so i wasn't 100% sure what the issues with the old importer were, i just saw and was like "oh i can just use the old importer and get this result" lol

[ssbucarlos]

Thanks for being flexible and quick to respond about this kind of stuff. There's a really passive aggressive argument that helped me track down the issue that you can read here :)

I try to be responsive, but the issue is that if i don't get around to responding when i get the notif, im simply going to forget :( The least i can do is be polite, which did not seem to be the goal of the issue submitter lol

[ssbucarlos]

There may be a way to convert the animation to a format that Blender can actually represent, but that's now a mathematics problem rather than an animation problem. The question is whether there is a skeleton that produces identical accumulated transforms for each bone where each bone's individual transform is an affine transformation consisting only of translation, rotation, and scale. Simply decomposing the original transforms clearly doesn't work in general. It's likely that this can't be solved exactly, so we would need to find a way to at least reduce the errors. @ssbucarlos

Ok today im gonna read up on transformation matrices, affine transformations, shear, etc and see if theres a possible workaround. If i end up giving up ill let you know

[ScanMountGoat]

Let me know if you find anything. The main issue is that Blender decomposes the bone transform into scale, rotation, and translation. This allows users to edit the translation, rotation, and scale instead of a matrix. The anim transformations usually only contain translation, rotation, and uniform scaling and can be decomposed and recomposed with essentially no error. $$T_{ultimate} \times R_{ultimate} \times S_{ultimate} = T_{blender} \times R_{blender} \times S_{blender}$$

I've found that compensate scale with non uniform scale results in a matrix that can't be represented as translation, rotation, and scale for similar reasons. The following equation isn't true for characters like Sonic due to non uniform scaling. Uniform scaling with scale compensation is still fine to decompose. Decomposing and recomposing every anim transform in ssbh_wgpu results in the same incorrect results as Blender.

$$T_{ultimate} \times SC_{ultimate} \times R_{ultimate} \times S_{ultimate} = T_{blender} \times R_{blender} \times S_{blender}$$

What I'm not sure about is how much these errors from incorrectly reconstructing the transform matrix accumulate along the bone chain. The current code just naively accumulates the Blender pose_bone.transform, which assumes the transforms are represented accurately in Blender. In theory, we could calculate the expected accumulated transforms for each bone in an initial pass and then assign the expected transforms to the Blender bones in a second pass. I wasn't able to get this working for some reason, so I wasn't able to test if it actually helps or not.

[ssbucarlos]

ok so i think i caught up and im not sure i fully understand, but is the issue that blender is fundamentally missing a 'shear' component (perhaps its called something else) for the bones/objects? but from messing around with test bones i see that its possible to create bone/object transforms that cannot be represented using the bone/objects loc/rot/scale alone, another object has to be created. It might be possible to create empties or dummy bones and scale/rotate those to provide the additional missing transform, ill let you know how that goes

[ssbucarlos]

admittedly this is pretty out of my comfort zone tbh so i think ill probably just move on after trying this workaround and chalk it up to "fundamental blender issue" for now 😓

[ScanMountGoat]

ok so i think i caught up and im not sure i fully understand, but is the issue that blender is fundamentally missing a 'shear' component (perhaps its called something else) for the bones/objects?

Yeah. Not all affine transforms can be represented using just scale, rotation, and translation.

admittedly this is pretty out of my comfort zone tbh so i think ill probably just move on after trying this workaround and chalk it up to "fundamental blender issue" for now 😓

I noticed you had dictionaries set up to cache the bone transforms as least as far back as daab14a0515421fda819aefad5d98639fd87eb12. The bone_to_matrix didn't seem to be actually used to set the bone transform though. This was the main thing I wanted to get working but wasn't able to. It would also be helpful for the future to document how the transforms work.

The issue with using empties, additional bones, etc is that it becomes harder to export the animations. If you can't figure anything out either, I wouldn't worry about it. People wanting to do wifi safe visibility animations for sonic's fsmash can just use your anim splicer. It's just helpful to know why the issue occurs for people wondering why "Blender broke their animation".

[ssbucarlos]

The issue with using empties, additional bones, etc is that it becomes harder to export the animations.

hmm ur right it would make the animation infeasible to edit/export anyways, so maybe its best to just document the issue, why it happens, that way ppl know that some vanilla anims wont import properly. i wonder if theres a cheap way to detect the issue and print a warning so users know to check the wiki for more info.

The bone_to_matrix didn't seem to be actually used to set the bone transform though

oh yeah i just had that since i needed to keep track of a bone's matrix to calculate it's child's matrix_basis.