Ideas for supporting `KHR_techniques_webgl` in the new `Model`

[ptrgags]

The new Model architecture is designed to be more modular than before. Already this has been quite helpful in adding new function like 3D Tiles Next Metadata. However, adding backwards compatibility for glTF 1.0/KHR_techniques_webgl poses challenges for the new architecture.

Background: Architectural differences

KHR_techniques_webgl/old Model	new Model
Access attributes directly (e.g. attribute vec3 a_position)	Pre-processes the attributes and stores them in a ProcessedAttributes struct that can be passed to shader functions.
Very little assumptions can be made about attributes (especially for glTF 1.0!). attribute types can be anything, NORMAL doesn't have to be normalized, etc. Flexible for the shader author, but hard to integrate with other features.	Based on glTF 2.0, so attributes have predictable types, NORMAL has unit length, etc. ProcessedAttributes stores attributes after applying matrices, normalizing normals, etc. Less flexible, but it makes it easier to add new features.
defines a main() function. modifications require renaming this function and adding a new wrapper function.	Each pipeline stage only adds functions/structs, a fixed function main() function (ModelVS.glsl/ModelFS.glsl) with #ifdefs determines which stages to run
Writes to gl_Position/gl_FragColor directly	stages may return values or modify out parameters, but only ModelVS.glsl/ModelFS.glsl sets gl_Position/gl_FragColor so there are no surprises.

Option A: Make a simplified pipeline for techniques

Description

This method would involve creating a TechniquePipelineStage that is intended to be run as the only stage in the pipeline. The pipeline stage would do the following:

• Create a shader that includes only the technique shader code. Even ModelVS/ModelFS would be ignored since the technique includes its own main() function.
• Take the list of attributes from the technique and match them up to attributes from primitive.attribute. The results would become the VertexArray at the end of the pipeline
• Uniforms would be added to the uniform map. Some would come from the extension on the material, others would come from CesiumJS builtins (e.g. czm_modelView).

Pros

• This method would be straightforward to implement
• It would not require any expensive operations like post-processing the shader text.

Cons

• Many Model features that modify the shader would not be compatible with this due to the architectural differences. We'd either have to disable most features, or hack together workarounds on a case-by-case basis. The latter might lead to code duplication
• Possible duplication of attribute processing code from GeometryPipelineStage. However, there might be some differences, as glTF 1.0 does not restrict attribute types like in glTF 2.0 (see table in this section)

Option B: Wrap the technique in a new pipeline stage

Description

This method would also define a TechniquePipelineStage, but it would be designed more like the other Model pipeline stages.

• The technique GLSL code would have to be modified in a few ways:
  • the main() function would need to be renamed since we have our own. Perhaps techniqueMain()
  • The attribute declarations would have to be removed, as the geometry stage already declares attributes at the top of the shader
  • Attribute variables would have to be renamed in the body of the technique code to match what the geometry stage uses. E.g. a_pos (technique) -> a_positionMC (geometry stage)
  • (optional) varying/uniform declarations could be extracted and added to the top of the shader via shaderBuilder.addUniform()/shaderBuilder.addVarying(). This is more for consistency and readability
• We'd create a new stage wrapper function, similar to how the custom shader stage works. here's one rough idea of how this might look:

// Vertex Shader:

// Note: this stage would not use ProcessedAttributes, because the technique
// would not be expecting that. It will use the attributes (e.g. a_positionMC)
// directly.
// Not 100% sure what the signature would be, this is just one idea
void techniqueStage(out vec4 position, out float pointSize) {
	// Call the technique code. This will modify gl_Position and possibly
	// gl_PointSize
	techniqueMain();
	position = gl_Position;
	pointSize = gl_PointSize;
}

// Fragment Shader:
void techniqueStage(inout czm_modelMaterial material) {
    // call the technique code from the glTF. This will modify
    // gl_FragColor
	techniqueMain();
	material.diffuse = gl_FragColor.rgb;
	material.alpha = gl_FragColor.a;
}

Pros

• This method would have much better compatibility with (most) other stages. If the same material struct is used, for example, then CustomShader or styling could override the color.
• This much better matches the architecture and intent of the Model pipeline

Cons

• If the technique uses unexpected attribute types (e.g. we've seen oct-encoded vec2 normal), this could confuse other stages that expect more typical values (e.g. vec3 normalized normals). In such a case, how would the geometry stage produce a vec3 normalMC that other stages might be expecting?
• Depending on the amount of re-writing necessary, this option could possibly get expensive in terms of regex substitutions/parsing, especially for long shaders.
• Unexpected formatting/code that's commented out might make some regex parsing tricky to implement

Option C: Transcode the technique to a CustomShader

Description

This would be similar to Option B, but instead of making a new pipeline stage, it would just create a CustomShader on the fly. Some details:

• The main() function would need to be renamed, and a vertexMain()/fragmentMain() wrapper function with the appropriate signature would need to be added at the end.
• Attribute declarations would have to be removed like in Option B
• Attributes used in the technique body would either need to be renamed to something from Processed Attributes (e.g. a_position -> attributes.positionMC) or to the attributes declared by the geometry stage (a_position -> a_positionMC). The former might be error prone, matrices might get double-applied in the vertex shader.
• The technique GLSL code would be added as vertexShaderText/fragmentShaderText
• Uniform declarations would be removed from the shader and added to the uniforms dictionary.

Pros

• This method would make use of existing abstractions

Cons

• Either now we'd need to support multiple custom shaders in sequence, or we'd have to disallow applying a custom shader to this model if there are techniques.
• This feels like a misuse of CustomShader. A technique may make different assumptions about the input than custom shaders does, and by setting the custom shader internally, it's unclear what should happen if the user adds one of their own.

Option D: Support multiple CustomShaders as an alternative

Description

Techniques have one advantage over the current CustomShader - you can apply multiple techniques within a single glTF. If you could apply multiple custom shaders to the same Model, maybe it could compensate for the lack of techniques.

• We'd need a way to identify primitives. You could certainly use EXT_mesh_features, though that sounds like overkill. material.name is one option, albeit this is not required by the glTF spec...
• Then in the constructor for a tileset/model, pass in a dictionary of shaderId -> CustomShader instead of a single shader.
• Then when running the CustomShaderPipelineStage, the only change necessary would be to look up the primitive's shader ID, and grab the corresponding CustomShader.

Pros

• While this may not be the right solution for solving the lack of techniques, it could be useful in general, as real-world data might require
• Since this works on a per-primitive level, it avoids the divergence of selecting a shader via an if statement over featureId.

Cons

• Determining how to identify primitives within a model is non-trivial, as a tileset may have thousands of models, each with several primitives.
• This method would still require re-processing many tilesets that are currently glTF 1.0/KHR_techniques_webgl

---

Appendix A: Changes to GltfLoader

In order to do any of the above options, the techniques would need to be parsed and cached. This is best done with GltfLoader (and other related loaders)

• Create loaders for technique sources, programs and techniques. Note that sources will require async loading since they may be in a separate glsl file.
• Also update ResourceCache to cache these resources, as they are often shared between primitives, or even across tiles in a tileset
• Update ModelComponents to include structs for these things. shader sources should be stored here as a string of GLSL code, as it will typically be post-processed in most of the above options
• ModelComponents should also have a struct for TechniqueUniform - as materials may provide constant values for these fields.
• Materials should now include a reference to the relevant technique and any uniforms that are necessary
• Material loading will now need to parse the KHR_techniques_webgl extension. This may involve additional texture loading when SAMPLER_2D uniforms are used
• GltfLoader will now need a step to parse the root-level extension.

[ptrgags]

As for my opinion:

I think Option B shows the most promise in terms of providing backwards compatibility for techniques. It's most aligned with the architecture and design goals of Model, and is the one most likely to play nice with other pipeline stages.

That said, it's not the most trivial to implement as it involves post-processing the technique shader code. I'm also concerned about how we want to handle non-typical attributes (e.g. vec2 normals that are oct-decoded in the technique), as it may conflict with the regular pipeline stages (e.g. GeometryPipelineStage). It would be good to research what corner cases to expect in practice and see if there are any restrictions we can make that would make it easier to implement

@lilleyse what are your thoughts on these options?

[ptrgags]

Just to cross-link the conversation, this discussion started in the forum in this thread about glTF 1.0 support

[ptrgags]

Yesterday I opened https://github.com/CesiumGS/cesium/issues/10821 for the vec2 oct-encoded normals, as that could be fixed independently of adding support for KHR_techniques_webgl

[katSchmid]

Has any decision been made on this?

[ggetz]

Hi @katSchmid, we're unlikely to prioritize work on backwards compatibility for 1.0 at this time. However, we would be happy to review a PR if you have the bandwidth to contribute.

[ggetz]

Given that we're unlikely to prioritize work on this, I'll close the issue. But as I said above, we'd be happy to review a PR.

[xtassin]

Hi,

I am a bit surprised this issue was closed as I believe managing several shaders per model is a very much required feature.

After having read the proposed solutions, I think being able to apply multiple customShaders that could match the model's materials would be a very suitable option.

Any chance this could be reconsidered?