Different out/inout behaviour during fragment shader execution

[Makogan]

By suggestion of @nvining-work (https://github.com/nvining-work) I am filing an issue here. He also wants to say hi to @dneto0.

I ran into the current situation, i was building a BVH raytracer using vulkan on Ubuntu, compiling with shaderc.

In this shader I declared the function:

int best_box;
float triangle_distance;
int IntersectBVH(out Ray ray, int id)
{
    best_box = 0;
    triangle_distance = 1e30f;

    uint stack[256];
    int sp = 0;

    stack[sp] = id;
    sp++;

    int safety = 0;
    while(sp > 0)
    {
        sp--;
        uint b_id = stack[sp];

        bool collides = IntersectAABB(ray, nodes[b_id].aabbMin, nodes[b_id].aabbMax);
        if(safety++ > 200) return collides ? int(b_id) : -1;
        if(nodes[b_id].triCount > 0 && collides)
        {
            // return int(b_id); 
            for (uint i = 0; i < nodes[b_id].triCount; i++ )
            {
                const uint triangle_id = tri_idxs[nodes[b_id].leftFirst + i];
			    IntersectTri( 
                    ray, 
                    triangles[triangle_id + 0].position.xyz,  
                    triangles[triangle_id + 1].position.xyz, 
                    triangles[triangle_id + 2].position.xyz);
                
                if(ray.t < triangle_distance)
                {
                    triangle_distance = ray.t;
                    best_box = int(b_id);
                }
            }

        }
        else if (collides)
        {
            stack[sp++] = nodes[b_id].leftFirst;
            stack[sp++] = nodes[b_id].leftFirst + 1; 
        }
    }

    return -1;
}

With this signature the program behaved strangely, depending on things like wheter or not I declared a variable or I called seemingly inconsequential functions, the program would sometimes work, sometimes it would output a blank screen.

Upon modifying the signature to int IntersectBVH(inout Ray ray, int id) all seemingly illogical behaviour stopped and my program followed my expectations.

The call to this function was:

    Ray ray = Ray(ray_pos, normalize(r), 1e30f);
    int val = IntersectBVH(ray, 0);

I have attached the broken fragment shader as a reference. In this file changing the out parameter qualifiers as specified above should result in different behaviours. I can provide more information if required.

broken.txt

[ubc-nvining]

(I'm not sure if this is your guys' bug, or if it needs to be referred to Khronos. Either way, it seems like undefined behaviour should be flagged somewhere down the line.)

[michaelnikelsky]

To be honest: int IntersectBVH(out Ray ray, int id) is a wrong use of the out parameter. Specifying the ray as an out parameter means: I don´t care about the value of the parameter before the function and I make sure the parameter is written to in the function. This doesn´t happen here, the Ray parameter should have an "in" specifier (or no specifier at all). If you have a C++ background think of it like this: in: const reference parameter inout: reference parameter out: function return parameter.