EXT_mesh_gpu_instancing with negative scale might break backface culling
Currently, EXT_mesh_gpu_instancing does not prohibit negative values for the "SCALE" attribute accessor. As a result, when a primitive is instanced, each instance may end up with a different winding order. As far as I know, none of the current graphics APIs provide a way to apply a cull mode per instance. Therefore, if the primitive is not double-sided, it seems difficult to render it according to the original intent of the extension.
Any thought on this?
Negative scaling is not prohibited in general, so this does not only apply to the instancing extension, but to core glTF as well. As such, this may be related to https://github.com/KhronosGroup/glTF/issues/2252
This is just a pointer. It might not be a real duplicate of that issue. It's not unlikely that a mix of positive and negative determinants within once instanced(!) draw operation raises further challenges or even possible ambiguities on the level of ~"Open/WebGL rendering specifications", so maybe someone wants to add further details here.
It feels like EXT_mesh_gpu_instancing should at the very least have a non-normative note saying that inconsistent scale sign (or maybe negative scale in general) is not supported well in renderers and is discouraged.
It's possible to make a renderer that can handle inconsistent signs - e.g. it's possible to use EXT_mesh_gpu_instancing merely as a storage compaction extension and create individual draw calls from each instance on load - or split the instances into two draw calls based on their sign. However I don't think this is how this extension is presently implemented, and I don't think it should be required of renderers to analyze the data.
Even if negative scale is used consistently, renderers will need to analyze the individual instance data for negative scale to work. This is doable, but contradicts the ideal of "GPU ready data can be loaded from buffers into GPU memory as is" and doesn't seem to be supported by three.js or Babylon.JS.