fix: improved quantization error

[ddeadguyy]

Feel free to link this failed attempt to https://github.com/nfrechette/acl/issues/352 I doubt it can compete with status quo unless clip ranges are manipulated further, but once you start doing that, you're just introducing floating point error somewhere else.

[commit-lint[bot]]

Bug Fixes

• improved quantization error (8e6f28f)

Contributors

ddeadguyy

Commit-Lint commands

You can trigger Commit-Lint actions by commenting on this PR:

• @Commit-Lint merge patch will merge dependabot PR on "patch" versions (X.X.Y - Y change)
• @Commit-Lint merge minor will merge dependabot PR on "minor" versions (X.Y.Y - Y change)
• @Commit-Lint merge major will merge dependabot PR on "major" versions (Y.Y.Y - Y change)
• @Commit-Lint merge disable will desactivate merge dependabot PR
• @Commit-Lint review will approve dependabot PR
• @Commit-Lint stop review will stop approve dependabot PR

[ddeadguyy]

Yeah, I can clear that up.

Most quantization follows the (start, end) -> (-0.5 + max_error, 0.5 - max_error) style. I removed quantization that ACL doesn't use as an added bonus, and to conserve effort.

The vector3_u48 family of functions is used to compress constant channels within segments. I opted to preserve the (start, end) -> (-0.5, 0.5) style there, which is what the precise_endpoints naming convention signifies. Now that I think about it, perhaps the new style would work better. I was more fixated on the bone ranges.

The vector3_u24 family of functions compresses bone ranges within segments. This is the trickiest part. I wanted to preserve endpoints as well as I could, knowing that the quantization within segments introduces error there. I also wanted to compress the midpoint of ranges instead of their mins, to take maximum advantage of the signed normalized range(-0.5, 0.5). In order for this to work for any subrange, the requirements get a bit stricter. (start, middle, end) -> (-0.5, 0.0, 0.5) or (0.0, 0.5, 1.0). I sacrificed a value in the bitmask to achieve this. A similar result would be achieved by using a sign bit instead, but IMO this is simpler. Naming convention is precise_endpoints_midpoint. Stepping through its unit test should help explain the process.

In any case, it didn't work as well as I'd hoped, so if you take a closer look, hopefully you find a bug.

FWIW I considered eliminating segment-level quantization(raw constants and ranges) to see what would happen, but the toughest obstacle was the assumption that segment constants and segment ranges are always the same size.

[nfrechette]

Awesome, thanks! I'll keep the PR around for now and when I get to validate all this, I'll take a look on my end to see what I find.