Raytracing

[bjornbytes]

I think we can do this!?

• Add Raytracer object. You can add Mesh or Model objects to it, or raw triangles/boxes. You can add things with a custom transform, and maybe add multiple copies of them cheaply (instancing).
• You can send the raytracer to shaders with Pass:send, just like a Texture or Buffer.
• In the shader, you can use ray queries to shoot a ray somewhere and figure out if it hit something.

Note that we would NOT be adding support for ray tracing pipelines, where you render the whole screen with raytracing and fancy materials. We would only be supporting ray queries to start. But these are still useful for implementing shadows, ambient occlusion, etc. If ray queries make it easier to do shadows compared to shadow mapping, that would be awesome.

I don't know very much about raytracing yet, so I'm still learning and figuring stuff out!

[bjornbytes]

Here's roughly what a ray query looks like in shader code:

uniform accelerationStructureEXT raytracer;

vec4 lovrmain() {
  rayQueryEXT query;
  vec3 rayPos = vec3();
  vec3 rayDir = vec3();
  float tmin = 0;
  float tmax = 1;
  
  rayQueryInitializeEXT(query, raytracer, gl_RayFlagsTerminateOnFirstHitEXT, 0xFF, rayPos, tmin, rayDir, tmax);
  rayQueryProceedEXT(query);
  if (rayQueryGetIntersectionTypeEXT(query, true) != gl_RayQueryCommittedIntersectionNoneEXT) {
    // hit something
  } else {
    // didn't hit anything
  }
}

(From Vulkan Samples)

[DonaldHays]

This would be very neat!

I started looking at raytracing shaders myself for the first time on my light mapping sample a few months back. Lovr, of course, didn't have any support for it at the time. The sample just does CPU-based raytracing.

Raytracing is better suited for higher-end hardware. And it usually requires temporal techniques to denoise. So it opens a bit of a can of worms, especially for VR, where temporal artifacts can be particularly bothersome. But it would be fun to play with.

[bjornbytes]

Quest 3/3S surprisingly support ray queries. They probably aren't particularly fast, but yes, would be fun to have it available to experiment with.

[bjornbytes]

The low-level Vulkan code for ray queries and acceleration structures has been added on a raytracer branch.

Now it just needs a Lua API and an implementation in the graphics module!

Current idea for Lua API:

-- `capacity` is max number of objects in raytracer, `options` table has some performance hints
raytracer = lovr.graphics.newRaytracer(capacity, { options })

-- adds a Mesh to raytracer with a given transform.  A mesh can be added multiple times,
-- which will instance it (reuse its data).  returns an ID that can be used to move the instance.
id = raytracer:add(mesh, ...transform)

-- Maybe you should also be able to add a raw vertex/index buffer?
id = raytracer:add(vertexBuffer, indexBuffer, ...transform)

-- Maybe you can add a Model?  The semantics of this are a little murky -- are you adding all
-- the meshes in the model under a single ID?  Or does it consume multiple IDs?  What if you want
-- to move *one* mesh in the model?
id = raytracer:add(Model, ...transform)

-- Maybe you can also add geometry from a table, for convenience?
-- This one would not really support instancing though.  Each time you add the data, you're creating
-- a new copy of the geometry data and computing a new BVH for it!
id = raytracer:add({vertices}, {indices}, ...transform)

-- Might want a different names that makes it clear you're changing an object, not the raytracer
raytracer:setTransform(id, ...transform)
raytracer:setMask(id, 0xff) --> objects can have an 8 bit mask which acts like a layer or filter

raytracer:clear() --> remove everything from the raytracer
raytracer:remove(id) --> I think you'll be able to remove individual objects

raytracer:getCapacity() --> how many objects can I hold?
raytracer:getCount() --> get the number of objects currently in the raytracer

raytracer:rebuild() --> manually rebuild the raytracer (not normally needed.  could pick up changes to meshes?)

Pass:send('var', raytracer) --> send to a shader to do ray queries against it!

Main sticking points are

• How Model interacts with raytracer:add
• What happens if you modify geometry data after adding it to a raytracer (e.g. Mesh:setVertices, or animated models for that matter), how BLAS gets rebuilt
• Can we actually do instancing when adding vertex/index buffers? It seems hard to associate the BLAS with <vertexBuffer, indexBuffer> pairs.

[bjornbytes]

Decision: You can add a Model to a Raytracer. It only consumes a single ID, since it would be way too confusing if objects consumed variable numbers of IDs. We can generate a single BLAS for the Model by using multiple geometries -- each one can have its own transform (and instanced meshes can point to the same vertex/index buffer addresses). This might be more efficient (packs all the model geometry in a single BLAS, only uses 1 TLAS instance), and you can always add the model's meshes individually if you want.

For initial version, it isn't necessary to be able to add Buffers or Lua tables to the Raytracer. We'll just start with Mesh and Model.