[🐛 bug report] Debug level OPTIX_COMPILE_DEBUG_LEVEL_FULL requires optimization level OPTIX_COMPILE_OPTIMIZATION_LEVEL_0.

[BachiLi]

Summary

I'm helping someone (@antipa) building/installing Mitsuba 2. When we run in the gpu_autodiff_rgb variant and load a scene file, we get COMPILE ERROR: Debug level OPTIX_COMPILE_DEBUG_LEVEL_FULL requires optimization level OPTIX_COMPILE_OPTIMIZATION_LEVEL_0. It fails at rt_check(): OptiX API error = 7001 (Invalid value) in ...\mitsuba2\src\librender\scene_optix.inl:109.

System configuration

• Platform: Windows 10 10.0.19041 Build 19041
• Compiler: Visual Studio 2019
• Python version: 3.8.3
• Mitsuba 2 version: master
• Compiled variants:
  • gpu_autodiff_rgb
• GPU: Maxwell Titan X with drivers 456.38, CUDA 10.2

Description

See summary.

Steps to reproduce

Run the following python script

import mitsuba
mitsuba.set_variant('gpu_autodiff_rgb')
from mitsuba.core import Thread
from mitsuba.core.xml import load_file

Thread.thread().file_resolver().append('../../Scene_Files/cbox')
scene = load_file('../../Scene_Files/cbox/cbox.xml')

Output

2020-10-23 10:09:57 INFO  main  [xml.cpp:1221] Loading XML file "..\..\Scene_Files\cbox\cbox.xml" ..
2020-10-23 10:09:57 INFO  main  [xml.cpp:1222] Using variant "gpu_autodiff_rgb"
2020-10-23 10:09:57 INFO  main  [xml.cpp:355] "..\..\Scene_Files\cbox\cbox.xml": in-memory version upgrade (v0.4.0 -> v2.2.1) ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\srgb.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\srgb_d65.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\path.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\ldsampler.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\box.dll" ..
cuda_eval(): launching kernel (n=1, in=0, out=19, ops=25)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
cuda_eval(): launching kernel (n=1, in=0, out=1, ops=7)
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\hdrfilm.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\perspective.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\diffuse.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\area.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\uniform.dll" ..
2020-10-23 10:09:57 INFO  main  [PluginManager] Loading plugin "plugins\obj.dll" ..
2020-10-23 10:09:57 DEBUG main  [OBJMesh] Loading mesh from "cbox_luminaire.obj" ..
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_luminaire.obj": read 2 faces, 4 vertices (120 B in 0ms)
cuda_eval(): launching kernel (n=2, in=5, out=24, ops=358)
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_luminaire.obj": computed vertex normals (took 23ms)
2020-10-23 10:09:57 DEBUG main  [OBJMesh] Loading mesh from "cbox_floor.obj" ..
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_floor.obj": read 2 faces, 4 vertices (120 B in 0ms)
cuda_eval(): launching kernel (n=4, in=4, out=3, ops=31)
cuda_eval(): launching kernel (n=2, in=5, out=24, ops=358)
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_floor.obj": computed vertex normals (took 24ms)
2020-10-23 10:09:57 DEBUG main  [OBJMesh] Loading mesh from "cbox_ceiling.obj" ..
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_ceiling.obj": read 4 faces, 8 vertices (240 B in 0ms)
cuda_eval(): launching kernel (n=4, in=9, out=27, ops=389)
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_ceiling.obj": computed vertex normals (took 22ms)
2020-10-23 10:09:57 DEBUG main  [OBJMesh] Loading mesh from "cbox_back.obj" ..
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_back.obj": read 2 faces, 4 vertices (120 B in 0ms)
cuda_eval(): launching kernel (n=8, in=4, out=3, ops=31)
cuda_eval(): launching kernel (n=2, in=5, out=24, ops=358)
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_back.obj": computed vertex normals (took 16ms)
2020-10-23 10:09:57 DEBUG main  [OBJMesh] Loading mesh from "cbox_greenwall.obj" ..
2020-10-23 10:09:57 DEBUG main  [OBJMesh] "cbox_greenwall.obj": read 2 faces, 4 vertices (120 B in 0ms)
cuda_eval(): launching kernel (n=4, in=4, out=3, ops=31)
cuda_eval(): launching kernel (n=2, in=5, out=24, ops=358)
2020-10-23 10:09:58 DEBUG main  [OBJMesh] "cbox_greenwall.obj": computed vertex normals (took 17ms)
2020-10-23 10:09:58 DEBUG main  [OBJMesh] Loading mesh from "cbox_redwall.obj" ..
2020-10-23 10:09:58 DEBUG main  [OBJMesh] "cbox_redwall.obj": read 2 faces, 4 vertices (120 B in 1ms)
cuda_eval(): launching kernel (n=4, in=4, out=3, ops=31)
cuda_eval(): launching kernel (n=2, in=5, out=24, ops=358)
2020-10-23 10:09:58 DEBUG main  [OBJMesh] "cbox_redwall.obj": computed vertex normals (took 16ms)
2020-10-23 10:09:58 DEBUG main  [OBJMesh] Loading mesh from "cbox_smallbox.obj" ..
2020-10-23 10:09:58 DEBUG main  [OBJMesh] "cbox_smallbox.obj": read 12 faces, 24 vertices (720 B in 1ms)
2020-10-23 10:09:58 DEBUG main  [OBJMesh] Loading mesh from "cbox_largebox.obj" ..
2020-10-23 10:09:58 DEBUG main  [OBJMesh] "cbox_largebox.obj": read 12 faces, 24 vertices (720 B in 1ms)
2020-10-23 10:09:58 INFO  main  [Scene] Building scene in OptiX ..
[ 2][COMPILE FEEDBACK]: COMPILE ERROR: Debug level OPTIX_COMPILE_DEBUG_LEVEL_FULL requires optimization level OPTIX_COMPILE_OPTIMIZATION_LEVEL_0.

rt_check(): OptiX API error = 7001 (Invalid value) in ......\src\librender\scene_optix.inl:109.
        Log: cuda_shutdown()
cuda_shutdown(): variable 4084 is still live.
cuda_shutdown(): variable 10 is still live.
cuda_shutdown(): variable 4066 is still live.
cuda_shutdown(): variable 11 is still live.
cuda_shutdown(): variable 12 is still live.
cuda_shutdown(): variable 13 is still live.
cuda_shutdown(): variable 14 is still live.
cuda_shutdown(): variable 15 is still live.
cuda_shutdown(): variable 16 is still live.
cuda_shutdown(): variable 17 is still live.
(skipping remainder)
cuda_shutdown(): 78 variables were still live at shutdown.

[merlinND]

Hi @BachiLi,

Assuming this is in Debug mode, does it work in Release mode? (At least this would give you a workaround).

The error is surprising since it seems we do set OPTIX_COMPILE_OPTIMIZATION_LEVEL_0 :

https://github.com/mitsuba-renderer/mitsuba2/blob/b87c11ace78e110b094b23b83ac5902a4773202d/src/librender/scene_optix.inl#L80-L81

Could you check this is indeed set as we expect?

[wjakob]

This is apparently a known ABI issue with the latest version of OptiX. Some constants defined in the header file changed their meaning (uh oh). Try setting the CMake option MTS_USE_OPTIX_HEADERS, which uses the official optix headers, which would have the right constant for that specific version.

[BachiLi]

Makes a lot of sense. We will try this. Thanks!

[garrido-pablo]

I had this issue when I upgraded my Nvidia drivers to one of the latest drivers a few weeks ago. I had to downgrade to 445.75 (old driver I installed in April/May this year) and mitsuba2 worked like a charm. In my case, it made sense as I still have Optix 6.5 installed. Hope this helps.

[antipa]

Thanks all. I am without power here in the Bay Area. @BachiLi and I will try these suggestions once power is restored and report back.

[BachiLi]

Thanks all for the help! It's working for us. We have to fix two more problems:

1. Nick's GPU only has compute capability of 5.2, so we have to change https://github.com/mitsuba-renderer/mitsuba2/blob/master/resources/ptx/optix_rt.ptx#L10 to sm_52.
2. As mentioned in https://github.com/mitsuba-renderer/mitsuba2/issues/254, with Optix 7.1 it crashes in Debug mode with nan/inf checks.

I will let y'all decide whether to close this issue or not. Thanks again!