getRotation returns the wrong world rotation if parent has non-uniform scale

[rafern]

Calling getRotation on an entity will return the right world rotation, unless a parent is scaled. Intuitively, entity.getRotation() should be equivalent to something like:

const parents = [];
let focus = entity;
while(focus) {
    parents.unshift(focus);
    focus = focus.parent;
}

const worldRot = new pc.Quat();
for(const parent of parents)
    worldRot.mul(parent.getLocalRotation());

(I know this is a horribly inefficient way to implement this)

Is the current behaviour intended? If so, would it be a good idea to add a new method that gets world rotation without scale and document the current behaviour?

Edit: The scaling issue only happens when the scale is uneven (e.g. (3, 1, 1) fails, but (3, 3, 3) works)

Edit^2: The solution above is still wrong for stretched entities, but closer to the real value. Maybe the real solution would be to implement a different matrix decomposition that takes stretch into account (assuming the current one doesn't)?

[rafern]

It turns out this is an unsolvable problem; you can't decompose matrix into its scale and rotation if the scale is non-uniform, which explains all the issues occurring. Apparently three.js has the same issue: https://github.com/mrdoob/three.js/issues/3845. I also tried decomposing this kind of matrix with gl-matrix and the issue still occurs. The only solution would be to document this edge case and not rely on scale/rotation decomposition; basically try to only use matrices if non-uniform scaling is possible. Here's what I used for testing with gl-matrix:

getWorldDecomposition(entity, translation = null, rotation = null, scale = null) {
    const glMat4 = glMatrix.mat4.clone(entity.getWorldTransform().data);
    const glVec3 = glMatrix.vec3.create();
    const glQuat = glMatrix.quat.create();

    if(translation) {
        glMatrix.mat4.getTranslation(glVec3, glMat4);
        translation.x = glVec3[0];
        translation.y = glVec3[1];
        translation.z = glVec3[2];
    }

    if(scale) {
        glMatrix.mat4.getScaling(glVec3, glMat4);
        scale.x = glVec3[0];
        scale.y = glVec3[1];
        scale.z = glVec3[2];
    }

    if(rotation) {
        glMatrix.mat4.getRotation(glQuat, glMat4);
        rotation.x = glQuat[0];
        rotation.y = glQuat[1];
        rotation.z = glQuat[2];
        rotation.w = glQuat[3];
    }
}

This might also be why rotate is buggy, but that's just a guess: https://github.com/playcanvas/engine/issues/3759

Edit: it turns out Playcanvas stores local position, rotation and scale in separate variables and builds the local transform out of them, instead of the other way around which I was assuming how it was done, so it might actually be possible to fix the ~~getRotation and getScale methods~~ getScale method (edit^2: maybe not getRotation because the child entity will be skewed if it has a non-uniform scaled parent; maybe if there is a new skew property on entities? I have no idea anymore, this problem is a mess), but it would make them very slow for children deep in the scene hierarchy (maybe caching the results would salvage the performance lost?). getPosition is still perfectly OK though

[mvaligursky]

Closing as current behaviour is expected.