Model picking does not find right intersections

[javagl]

(This is a spin-off from https://github.com/CesiumGS/cesium/issues/11814#issuecomment-1932619298 , where the info that is posted here has been removed)

This model is a (low-resolution) sphere that can be put into the /Apps/SampleData/models/CesiumAir/ for the test:

sphere_V60_radius_5.zip

Then one can run this sandcastle locally:

const viewer = new Cesium.Viewer("cesiumContainer", {
  globe: false
});

const urlA = "../../SampleData/models/CesiumAir/Cesium_Air.glb";
const urlB = "../../SampleData/models/CesiumAir/sphere_V60_radius_5.glb";
const scene = viewer.scene;

scene.camera.position = new Cesium.Cartesian3(0.0, 0.0, 75.0);
scene.camera.direction = Cesium.Cartesian3.negate(
  Cesium.Cartesian3.UNIT_Z,
  new Cesium.Cartesian3()
);
scene.camera.up = Cesium.Cartesian3.clone(Cesium.Cartesian3.UNIT_Y);
scene.camera.frustum.near = 0.01;
scene.camera.frustum.far = 500.0;

const modelA = await Cesium.Model.fromGltfAsync({
  url: urlA
});
viewer.scene.primitives.add(modelA);  

const modelB = await Cesium.Model.fromGltfAsync({
  url: urlB
});
viewer.scene.primitives.add(modelB);  

const handler = new Cesium.ScreenSpaceEventHandler(scene.canvas);
handler.setInputAction(function (movement) {
  const ray = scene.camera.getPickRay(movement.position);
  const pickedA = modelA.pick(ray, scene.frameState, 1.0, 0.0, new Cesium.Cartesian3());
  const pickedB = modelB.pick(ray, scene.frameState, 1.0, 0.0, new Cesium.Cartesian3());
  console.log("pickedA: "+pickedA);
  console.log("pickedB: "+pickedB);
  
  if (Cesium.defined(pickedA)) {
    viewer.entities.add({
      position: pickedA,
      point: {
        pixelSize: 8,
        color: Cesium.Color.YELLOW,
        disableDepthTestDistance: Number.POSITIVE_INFINITY,
      },
    });
  }
  if (Cesium.defined(pickedB)) {
    viewer.entities.add({
      position: pickedB,
      point: {
        pixelSize: 12,
        color: Cesium.Color.RED,
        disableDepthTestDistance: Number.POSITIVE_INFINITY,
      },
    });
  }
  
  
}, Cesium.ScreenSpaceEventType.LEFT_CLICK); 

It will load both models, and check for intersections while picking. Intersections with the plane will be shown in yellow. Intersections with the sphere will be shown in red. Here's what it does:

From what I can tell, there clearly are places where it does not pick the sphere although it should. And there clearly are places where it picks the plane although it shouldn't.

[javagl]

For the specific case here, one veeeery deep root problem is what I found after adding some ridiculous debug logging of the vertex coordinates of each triangle of a unit square that was supposed to be tested for intersections: Got (0, 0, 1) (0, 0, 1) (1, 0, 0) for 1 3 2

Wait, what? Yeah. The part at https://github.com/CesiumGS/cesium/blob/a47c95e1ad2601c329d367b0159dca00599b72d2/packages/engine/Source/Scene/Model/pickModel.js#L247 does not work for interleaved buffers.

When storing that unit square with non-interleaved buffers, it works.

I'll try to fix that, but will probably not be able to resist the urge do to a few other cleanups there as well. But I'll have to further wrap my head about some interdepdencies there - what's the role of the WebGL2 frame state and these typed arrays, or how attempting to fix the issue about interleaved buffers may be affected by the quantization. Eventually, it may include

• a fix for this issue
• a fix for https://github.com/CesiumGS/cesium/issues/11814
• avoiding a few million calls to defaultValue
• avoiding that the dequantization and applying the exaggeration are both done six times for each vertex (they are done on the fly in getVertexPosition...)
• having fewer than 300 lines in one function...

[jjhembd]

@javagl thanks for the test cases! Here's what I get from your sphere & airplane case after #11842