Custom Layer Interface is missing a mercatorMatrix3d

[olsen232]

The Custom Layer Interface documentation gives an example of how to make a custom 2D layer. It documents that you can choose whether the layer is 2D or 3D - but the effects of that choice are quite limited, possibly only enabling or disabling the depth buffer. Different transformational matrices are used for 2D and 3D rendering, but the Custom Layer Interface is always provided with a matrix called mercatorMatrix. This matrix is appropriate, I believe, for 2D rendering, but not for 3D rendering, since it is missing one of the translate steps that the matrices used for 3D rendering have. Specifically, it doesn't have this line of code, or any equivalent: mat4.translate(m, m, [0, 0, -this.elevation]); // elevate camera over terrain

Because it is missing this translation - which is not always the same amount, and depends in some way on the terrain height at the center of the camera - using this matrix to render a point in a 3D context causes the point to appear too high, by a small amount when the camera-center-terrain-height is near sea level, and by a large amount when the camera-center-terrain-height is high altitude.

maplibre-gl-js version: 4.1.0 or main @ 33457131

browser: Brave, but this logic is not browser dependent.

Steps to Trigger Behavior

Use a CustomLayerInterface to render points / objects in a 3D view, with terrain enabled, in a place with hilly terrain (effects of the bug will be minimal if the terrain is close to sea level).

Link to Demonstration

https://github.com/maplibre/maplibre-gl-js/pull/3796 - which is still a draft - shows the issue (the red point is too high), and shows a basic fix (the green point is in the correct location). To try it out, checkout these changes, run build-dev, and open test/examples/custom-layer-interface-bug.html Moving the view around will cause the red dot's height to jump to some other value whenever recalculateZoom is called, which happens at the end of each panning movement.

Here is a screenshot of the issue as shown by the #3796

Note that the green point has the right position, this is possible since I've modified src/geo/transform.ts to expose another matrix called mercatorMatrix3D.

Expected Behavior

• Red point should be shown where the green point is.
• A matrix should be provided to make it possible / straight-forward to show the red point in the right place.
• Documentation at https://maplibre.org/maplibre-gl-js/docs/API/interfaces/CustomLayerInterface/ should explain how to render in 2D and in 3D - or doing either one should work without modifying the given example code.

Actual Behavior

• Red point is in the wrong place.
• No matrix provided by CustomLayerInterface to get the red point in the right place.
• Documentation doesn't explain how to get the red point in the right place when rendering in 3D.