instant-ngp
instant-ngp copied to clipboard
NVlabs
Could not initialize Vulkan and NGX. DLSS not supported
Linux, Debian stable Git build from this morning.
I actually had instant-ngp working on this system months ago, but now get only errors. My system has a 1050 Ti (4gb) and two K80's (one K80 has two 12gb cards). (There could have been other changes, like nvidia drivers maybe put an update in debian or something), but I had it working at commit d793984b0). I've seen some similar errors in other posts, but I'm not sure if it's the same issue.
With the two types of cards, the K80's come after the 4gb 1050, so I use: export CUDA_VISIBLE_DEVICES=1,2
19:36:32 INFO Loading NeRF dataset from
19:36:32 INFO data/pulseox/transforms.json
19:36:32 SUCCESS Loaded 98 images after 0s
19:36:32 INFO cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:36:33 INFO Loading network config from: configs/nerf/small.json
19:36:33 INFO Loading parent network config from: configs/nerf/base.json
19:36:33 INFO GridEncoding: Nmin=16 b=1.5874 F=2 T=2^15 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performan
ce, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performan
ce, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performan
ce, raise the target GPU architecture to 75+.
19:36:33 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
19:36:33 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
19:36:33 INFO total_encoding_params=960848 total_network_params=9728
19:36:33 WARNING Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:36:33 WARNING Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
Got cutlass error: Error Internal at: 346
Could not free memory: /home/yippie/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:465 cudaDeviceSynchronize() failed with error operation not permitted when stream is capturing
$ nvidia-smi
Sun Jul 17 19:44:36 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03 Driver Version: 460.91.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 105... On | 00000000:09:00.0 On | N/A |
| 31% 38C P8 N/A / 75W | 617MiB / 4039MiB | 16% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 On | 00000000:0C:00.0 Off | 0 |
| N/A 40C P8 27W / 149W | 8MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 Tesla K80 On | 00000000:0D:00.0 Off | 0 |
| N/A 42C P8 30W / 149W | 8MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
I went ahead and tested with the frequency config (no clue what that is, but it defaults to Cutlass):
19:45:50 INFO Loading NeRF dataset from
19:45:50 INFO data/pulseox/transforms.json
19:45:50 SUCCESS Loaded 98 images after 0s
19:45:50 INFO cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:45:51 INFO Loading network config from: configs/nerf/frequency.json
19:45:51 INFO Loading parent network config from: configs/nerf/base.json
19:45:51 INFO Density model: 3--[Frequency]-->96--[CutlassMLP(neurons=256,layers=9)]-->1
19:45:51 INFO Color model: 3--[Frequency]-->24+16--[CutlassMLP(neurons=256,layers=3)]-->3
19:45:51 INFO total_encoding_params=0 total_network_params=434176
19:45:51 WARNING Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:45:51 WARNING Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
19:45:51 ERROR Uncaught exception: /home/jag/opt/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:567 cuMemCreate(&m_handles.back(), n_bytes_to_allocate, &prop, 0) failed with error CUDA_ERROR_OUT_OF_MEMORY
Could not free memory: /home/jag/opt/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:473 cuMemRelease(handle) failed with error CUDA_ERROR_INVALID_VALUE
And when I drop frequency.json's n_neurons to 16, to test:
19:47:08 INFO Loading NeRF dataset from
19:47:08 INFO data/pulseox/transforms.json
19:47:08 SUCCESS Loaded 98 images after 0s
19:47:08 INFO cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:47:09 INFO Loading network config from: configs/nerf/frequency.json
19:47:09 INFO Loading parent network config from: configs/nerf/base.json
19:47:09 INFO Density model: 3--[Frequency]-->96--[CutlassMLP(neurons=16,layers=9)]-->1
19:47:09 INFO Color model: 3--[Frequency]-->24+16--[CutlassMLP(neurons=16,layers=3)]-->3
19:47:09 INFO total_encoding_params=0 total_network_params=4096
19:47:09 WARNING Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:47:09 WARNING Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
Got cutlass error: Error Internal at: 346
Could not free memory: /home/jag/opt/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:465 cudaDeviceSynchronize() failed with error operation not permitted when stream is capturing
I have not yet tested reverting to that old revision but do plan to.
Update: With an aabb_scale of 2 or 4 in data/pulseox/transforms.json, without a specified config, it does pop up the gui finally:
19:51:45 INFO Loading NeRF dataset from
19:51:45 INFO data/pulseox/transforms.json
19:51:45 SUCCESS Loaded 98 images after 0s
19:51:45 INFO cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:51:45 INFO Loading network config from: configs/nerf/base.json
19:51:45 INFO GridEncoding: Nmin=16 b=1.51572 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
19:51:45 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
19:51:45 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
19:51:45 INFO total_encoding_params=13074912 total_network_params=9728
19:51:45 WARNING Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:51:45 WARNING Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
With an aabb_scale of 8 it gives the stream is capturing error:
19:54:06 INFO Loading NeRF dataset from
19:54:06 INFO data/pulseox/transforms.json
19:54:06 SUCCESS Loaded 98 images after 0s
19:54:06 INFO cam_aabb=[min=[-0.444531,-1.33551,-1.07948], max=[3.62585,2.37042,4.1108]]
19:54:07 INFO Loading network config from: configs/nerf/base.json
19:54:07 INFO GridEncoding: Nmin=16 b=1.5874 F=2 T=2^19 L=16
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 37. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
19:54:07 INFO Density model: 3--[HashGrid]-->32--[FullyFusedMLP(neurons=64,layers=3)]-->1
19:54:07 INFO Color model: 3--[Composite]-->16+16--[FullyFusedMLP(neurons=64,layers=4)]-->3
19:54:07 INFO total_encoding_params=13288400 total_network_params=9728
19:54:07 WARNING Vulkan error: loaderValidateLayers: Layer 0 does not exist in the list of available layers
19:54:07 WARNING Could not initialize Vulkan and NGX. DLSS not supported. (/home/jag/opt/src/nerf/mesh/instant-ngp/src/dlss.cu:220 vkCreateInstance(&instance_create_info, nullptr, &vk_instance) failed)
Got cutlass error: Error Internal at: 346
Could not free memory: /home/jag/opt/src/nerf/mesh/instant-ngp/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:465 cudaDeviceSynchronize() failed with error operation not permitted when stream is capturing
I'm not closing this since I'm not sure what's going on, or if I'm doing something wrong still, and I'd appreciate suggestions.
Same here. Please let me know if anyone has solved this! May thanks!!! @jaggzh @psakamoori
Same here. I think the problem is rooted from my graphics card. It has only 4GBs of memory, however, I am unsure as well.
