Support for multiple shader objects per stage in opengl

[cstoitner]

When renderdoc tries to process the shaders of my application, I get the following error:

Error   - Couldn't make separable shader program for shader. Errors:
Vertex info
-----------
0(25) : error C3002: call to undefined function "void transformPosition(vec3, vec4);"
0(26) : error C3002: call to undefined function "vec3 transformNormal_worldSpace();"

My application links multiple shader objects per shader stage into a gl shader program (for code reuse). The missing functions are defined in a second vertex shader object. From a quick look at the code I can not tell if this is just not supported in renderdoc, or if there is a different problem with patching my shaders for use in a separable program.

[baldurk]

Yes I think this is just plain unsupported by renderdoc right now, I didn't think anyone was wild enough to actually make use of this opengl craziness :).

[ajurcik]

I am thinking about trying to implement this feature. I have only very shallow knowledge of RenderDoc's design so that I would appreaciate you feedback on my idea.

My first impression was that RenderDoc creates a separable shader program per shader. What about creating the separable shader program per shader stage, i.e., at a one level higher granularity? Please, do you see any possible design issues here?

[baldurk]

RenderDoc does create a separable shader program per shader stage already, since it's used for reflecting out per-stage information as well as for replacing individual stages and things like that. I think maybe you're misinterpreting because there's an implicit assumption that only one shader is used per stage, which is the core of this issue of not supporting multiple shaders per stage.

That assumption is present in quite a lot of places, not just that part which only happens during replay. I don't know all the code off hand that would need to change necessarily since I'd never thought about abusing the API in this way.

[ajurcik]

Yes, I think your statement

there's an implicit assumption that only one shader is used per stage

more clearly describes what also I meant.

Please, what do you mean by

abusing the API?

I googled a little bit and found that D3D enables similar functionality: https://docs.microsoft.com/en-us/windows/desktop/direct3dhlsl/using-shader-linking. But it seems that D3D's approach is less error prone in terms of not including all necessary shader objects.

From my application's point of view, it is good that each shader source is associated with a version directive. In this manner it is at first sight obvious whether I can include a specific common shader code or I need to take an alternative pathway due to, e.g., a feature unavailability. Sure, you can do this using comments also or by implementing your own framework. But if the functionality is already in the OpenGL API it seems to me as a good idea to use it first. It also feels to me a more natural way to have the common shader pieces well defined as compilation units (standalone shaders). They can be easily checked by the compiler in this manner.

Currently, I don't see it as a good option to change my app's design towards including the common shader code using additional shader sources. Therefore I played a little with RenderDoc's code and currently I am able to replay a capture from the app which I attached to #1079. If you are interested you can check this here: https://github.com/ajurcik/renderdoc/tree/glsl-multi-shader. Please note that it is only a prototype and there are known unsolved bugs and unimplemented parts. Basically, I defered the creation of the separable (per-stage) shader program until glLinkProgram when the all the captured shaders are already set up. The main unimplented part is when glsllang is used but I think this could be done in a very similar way.

I can continue on this feature (trying the best in C++ as I can do:) as I think it would be great to have it in RenderDoc even with probable limitations compared to not beeing able to replay, e.g., post VS data. Please, let me know your ideas about this feature.

[baldurk]

D3D is not really comparable because it has a sensible system where shader compilation happens offline to an IR which is then passed to the API at runtime. I don't think many projects use that shader linking system, though I don't know exactly how many.

The reason it's awkward is that it adds a fair amount of complexity to track and can be done equivalently with multiple shader sources as is more typical. On GL especially going off the beaten path like this especially with shaders exposes you to more chances for driver variation and bugs. The only thing the driver can do up front is front-end lowering to an internal IR, the majority of compilation is still deferred to program link time in the driver.

There are quite a few edge-cases in GL which vary in difficulty to handle and obtuseness. In an ideal world I would like to support every possible legal use of the API, but it's unfortunately not always worth the development effort to get working.

If you want to PR support that would be fine, but it would need to be complete - I think at the moment this is only doing just enough to cover your use case. Unfortunately with GL because there are so many of these rabbit holes where you can do unexpected weird things with the API there are a lot of potential intersections and complexities. E.g. you mention you have different versions per shader and I hadn't even thought of that possibility - that makes for complexities in GLES when you need to version-match any shader replacements in the overlay and things.

The code you have there is a start but as you mention it doesn't cover Post VS and overlays, nor does it cover the shader patching necessary to make shaders legal to be separable. There's also the case where currently attaching and detaching a shader from a program - currently I sidestep that problem with an ever-expanding list and the loop just picks up the last one bound, but in your code this will break. You'd need to start handling arbitrary attaching and detaching properly which is quite complex in its own right.

As a side note I assume you don't want nitpicks this early, but if you do continue with this branch I'd recommend you check out the contributing guide to ensure you're formatting commits correctly - it's much easier to do that from the start than retrofit it.