Ray tracing and multi-gpu/ doman

[IMbackK]

I noticed that there are various bits in the code that disable the raytraced free surface visualization when multiple domains are in use. What is the technical reason for this limitation? Also mild feature request.

[ProjectPhysX]

Hi @IMbackK,

so with multi-GPU, each GPU only has its own domain kN memory and can't see the others. For rasterization rendering, each GPU can still independently render its own domain to its own frame buffer, with the 3D offset of the domain applied before camera transformations. The frames from all GPUs are then sent to CPU and overlayed with their accompanying z-buffers, such that always the pixel from the domain closest to the camera gets drawn to the composite frame.

Raytracing works very differently: shoot rays outward from the camera, make them traverse the domain grid, and reflect/refract off the water surface multiple times. With multi-GPU, if a ray goes through one domain and into another, there is no easy way to tell the other domain; some GPU-->GPU data transfer would be needed for every single ray, ans since reflected/refracted rays go in chaotic directions, this is very difficult to handle.

So for now I've just disabled raytracing for multi-GPU.

Kind regards, Moritz

[IMbackK]

Hi,

Thanks, that makes sense. I guess as a hack i could copy all the domains onto one gpu and do the raytraceing there, usually this would be pretty bad but on infinity fabric connected mi100 gpus there should be enough bandwidth to pull this off for smallish domains.

[ProjectPhysX]

@IMbackK the problem here is that one GPU cannot fit the other domains in memory anymore. An option here would be to copy all domains to CPU and then optionally to an extra rendering GPU not assigned to simulation, and render it either on CPU or that extra GPU.