cupoch icon indicating copy to clipboard operation
cupoch copied to clipboard

Getting cudaErrorMemoryAllocation out of memory

Open quicktwit opened this issue 2 years ago • 6 comments

Hi This library is awesome! Thank you, author. I am working on big-size point clouds (in ply format). I am trying this code (https://github.com/neka-nat/cupoch/blob/master/examples/python/advanced/pointcloud_outlier_removal.py) to filter point cloud noises. On a few MBs or around 1Gbs size point cloud file works like a charm. But when I try on a bigger point cloud file(5GB - approx 97million points) then I after remove_statistical_outlier I am getting cudaErrorMemoryAllocation out of memory.

cl, ind = voxel_down_pcd.remove_statistical_outlier(nb_neighbors=20, std_ratio=2.0)
Memory Error: std::bad_alloc: CUDA error at /home/path/cupoch/cupoch/third_party/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:69: cudaErrorMemoryAllocation out of memory

I really wanted to use this library as it is much faster than other alternatives. Can you help me with how to solve this issue? I really appreciate this. Thank you again in advance!! P.S. System: Ubuntu 20.04 RTX3090 24GB I installed cupoch using pip

quicktwit avatar Jun 30 '22 05:06 quicktwit

Hi, Thanks for the report! If possible, could you please share the pointcloud file with the error?

neka-nat avatar Jul 01 '22 07:07 neka-nat

Hi @neka-nat Thank you for your reply. Some data I cannot put publicly. So I’m finding some workaround so that l can share here easily. I will update soon. Thank you so much again.

quicktwit avatar Jul 02 '22 08:07 quicktwit

Hi @neka-nat I tested one point cloud data. You can use the file called L003.ply from here: https://1drv.ms/u/s!Amlc6yZnF87psX6hKS8VOQllVvj4?e=yWhrYX This file L003.ply has more than 39million points. Screen Shot 2022-07-03 at 19 29 11 Same error I am getting for this L003.ply also:

cl, ind = voxel_down_pcd.remove_statistical_outlier(nb_neighbors=20, std_ratio=2.0)
MemoryError: std::bad_alloc: CUDA error at: /home/runner/work/cupoch/cupoch/third_party/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:69: cudaErrorMemoryAllocation out of memory

quicktwit avatar Jul 03 '22 10:07 quicktwit

Hi @neka-nat Do you have any advice on how do I fix this? I tried too much downsampling but it's affecting the result. Let me know if you have any suggestions. Thank you in advance.

quicktwit avatar Jul 07 '22 23:07 quicktwit

I am still investigating the problem, but I found that it was failing to allocate memory during flann tree construction. It may take some time to solve the problem.

neka-nat avatar Jul 10 '22 13:07 neka-nat

Hi @neka-nat Thank you for the update. I will wait for your update. Thank you once again.

quicktwit avatar Jul 11 '22 01:07 quicktwit