Error 700 illegal memory access encountered when training on custom data

[erikfredin]

Hi! I'm currently trying to use this repository to train a Voxel-RCNN on my own data, and used the tutorial on this repository to do that. When attempting to train, my code shows the error message pasted below. I also tried training another model and got the same result. How do I resolve this?

The error: RuntimeError: sync /tmp/pip-build-env-l_y4vd57/overlay/lib/python3.11/site-packages/cumm/include/tensorview/profile/cuda_profiler.h:168 cuda failed with error 700 an illegal memory access was encountered. use CUDA_LAUNCH_BLOCKING=1 to get correct traceback.

The error occurs when running the following line: File "/home/microlab/Erik/OpenPCDet/pcdet/models/backbones_3d/spconv_backbone.py", line 149, in forward x = self.conv_input(input_sp_tensor)