warp icon indicating copy to clipboard operation
warp copied to clipboard

[QUESTION] Catch an cuda OOM error

Open yzx9 opened this issue 8 months ago • 2 comments

How can I catch an cuda OOM (Out of Memory) error when trying to allocate a very large VDB using wp.Volume.allocate_by_tiles?

Warp CUDA error 2: out of memory (in function alloc_device_async, /builds/omniverse/warp/warp/native/warp.cu:588)
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to allocate grid buffer on the device

yzx9 avatar Apr 01 '25 08:04 yzx9

Looks like there's a C++ exception thrown by NanoVDB which goes unhandled and crashes the program. We can do a better job there and convert it to a Python exception so that you can catch it.

nvlukasz avatar Apr 01 '25 15:04 nvlukasz

@gdaviet provided a fix for this in 4108a52804c5cc94169fa860f7ce0f330c99ec92, which should be available for testing by building from source or starting with tonight's nightly build. It will be made available in Warp 1.8.0.

shi-eric avatar Apr 28 '25 21:04 shi-eric