Custom Shader - detect `material.alpha` instead of `isTranslucent` flag?

[ptrgags]

Right now, when using a CustomShader with ModelExperimental, material.alpha only affects the translucency when CustomShader.isTranslucent is set true in the constructor.

This seems to be non-obvious. Another way this could be addressed is to check for the presence of material.alpha in the shader text (this should already be parsed in customShader.usedVariablesFragment.materialSet and if set, change the pass to translucent. This would replace the check in CustomShaderPipelineStage to something along the lines of this:

  if (materialSet.hasOwnProperty('alpha')) {
    alphaOptions.pass = Pass.TRANSLUCENT;
  } else {
    alphaOptions.pass = undefined;
  }

Note that this would not be able to detect that material.alpha = 1.0 would be opaque since in general the result will be an expression evaluated on the GPU.

[j9liu]

Tangentially related (though if this should be its own issue that's fine too), when #10549 fixed the improper handling of translucency in ModelExperimental, it exposed that custom shaders wouldn't work if they resulted in both opaque and translucent colors. isTranslucent basically marks the entire shader translucent, disabling the depth mask, so any opaque features will render without regard to depth.

We might need to mimic how styling works and use one opaque command and one translucent, with some extra work to avoid drawing opaque / translucent parts in the commands they don't belong to.

[ptrgags]

The one tricky thing about opaque and translucent for custom shader is it's harder to tell when to apply each, since the final color is the result of user-defined GLSL code (whereas CPU styling can expect a feature color to determine the translucency)

Certainly possible to do the pass filtering with the result, though whether that's a good idea (GPU divergence-wise) probably depends on the user's shader

[j9liu]

Certainly possible to do the pass filtering with the result, though whether that's a good idea (GPU divergence-wise) probably depends on the user's shader

That's a fair caveat; if we ever support both opaque and translucent rendering in the same CustomShader we should definitely warn the user about the performance implications.