VoxFormer icon indicating copy to clipboard operation
VoxFormer copied to clipboard

Published 20 hours ago • verified publisher icon NVlabs

stage2 training error

Open ziming-liu opened this issue 1 year ago • 3 comments

File "/home/ziliu/anaconda3/envs/semantics/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 186, in new_func return old_func(*args, **kwargs) File "/data/acentauri/user/ziliu/data/voxformer/projects/mmdet3d_plugin/voxformer/modules/deformable_cross_attention.py", line 143, in forward index_query_per_img = mask_per_img[0].sum(-1).nonzero().squeeze(-1) RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

ziming-liu avatar Aug 27 '23 20:08 ziming-liu

Hello, I have the same problem, may I ask you to solve this problem? If so, can you provide some solutions

Sxiansheng24 avatar Mar 13 '24 13:03 Sxiansheng24

Hello, I have the same problem, may I ask you to solve this problem? If so, can you provide some solutions

no, I didn't move forward.

ziming-liu avatar Mar 17 '24 21:03 ziming-liu

Hello, after completing the first phase of training, how should I proceed with the second phase? Are there any additional steps required, or do I only need to enter the training command in the terminal? Directly entering the training command results in an error, as shown in the picture. Snipaste_2024-05-17_16-20-11

codeimdy avatar May 17 '24 08:05 codeimdy