Error numba.cuda.cudadrv.error.NvvmError: Failed to compile, while testing SECOND model with pre-trained model and Config file

[ammaryasirnaich]

I am trying to test the SECOND model with pretrained model and config file, but when i run the test.py script i am getting the below error. I will appreciate if can get help with it .

2022-06-27 14:28:25,122   INFO  cfg.EXP_GROUP_PATH: workspace/OpenPCDet/tools/cfgs/kitti_models
2022-06-27 14:28:25,123   INFO  Loading KITTI dataset
2022-06-27 14:28:25,279   INFO  Total samples for KITTI dataset: 3769
2022-06-27 14:28:26,408   INFO  ==> Loading parameters from checkpoint /workspace/working_dir/second_7862.pth to GPU
2022-06-27 14:28:26,427   INFO  ==> Done (loaded 163/163)
2022-06-27 14:28:26,436   INFO  *************** EPOCH 7862 EVALUATION *****************
eval: 100%|████████████████████████████| 943/943 [00:59<00:00, 15.97it/s, recall_0.3=(0, 16665) / 17558]
2022-06-27 14:29:25,499   INFO  *************** Performance of EPOCH 7862 *****************
2022-06-27 14:29:25,499   INFO  Generate label finished(sec_per_example: 0.0157 second).
2022-06-27 14:29:25,499   INFO  recall_roi_0.3: 0.000000
2022-06-27 14:29:25,499   INFO  recall_rcnn_0.3: 0.949140
2022-06-27 14:29:25,499   INFO  recall_roi_0.5: 0.000000
2022-06-27 14:29:25,499   INFO  recall_rcnn_0.5: 0.891047
2022-06-27 14:29:25,499   INFO  recall_roi_0.7: 0.000000
2022-06-27 14:29:25,500   INFO  recall_rcnn_0.7: 0.665053
2022-06-27 14:29:25,502   INFO  Average predicted number of objects(3769 samples): 14.227
Traceback (most recent call last):
  File "/workspace/OpenPCDet/tools/test.py", line 200, in <module>
    main()
  File "/workspace/OpenPCDet/tools/test.py", line 196, in main
    eval_single_ckpt(model, test_loader, args, eval_output_dir, logger, epoch_id, dist_test=dist_test)
  File "/workspace/OpenPCDet/tools/test.py", line 62, in eval_single_ckpt
    eval_utils.eval_one_epoch(
  File "/workspace/OpenPCDet/tools/eval_utils/eval_utils.py", line 110, in eval_one_epoch
    result_str, result_dict = dataset.evaluation(
  File "/workspace/OpenPCDet/pcdet/datasets/kitti/kitti_dataset.py", line 357, in evaluation
    from .kitti_object_eval_python import eval as kitti_eval
  File "/workspace/OpenPCDet/pcdet/datasets/kitti/kitti_object_eval_python/eval.py", line 6, in <module>
    from .rotate_iou import rotate_iou_gpu_eval
  File "/workspace/OpenPCDet/pcdet/datasets/kitti/kitti_object_eval_python/rotate_iou.py", line 263, in <module>
    def rotate_iou_kernel_eval(N, K, dev_boxes, dev_query_boxes, dev_iou, criterion=-1):
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/decorators.py", line 135, in kernel_jit
    return Dispatcher(func, sigs, bind=bind,
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/compiler.py", line 811, in __init__
    self.compile(sigs[0])
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/compiler.py", line 952, in compile
    kernel.bind()
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/compiler.py", line 576, in bind
    self._func.get()
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/compiler.py", line 446, in get
    ptx = self.ptx.get()
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/compiler.py", line 415, in get
    ptx = nvvm.llvm_to_ptx(self.llvmir, arch=arch,
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/cudadrv/nvvm.py", line 548, in llvm_to_ptx
    ptx = cu.compile(**opts)
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/cudadrv/nvvm.py", line 236, in compile
    self._try_error(err, 'Failed to compile\n')
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/cudadrv/nvvm.py", line 254, in _try_error
    self.driver.check_error(err, "%s\n%s" % (msg, self.get_log()))
  File "/opt/conda/lib/python3.8/site-packages/numba/cuda/cudadrv/nvvm.py", line 144, in check_error
    raise exc
numba.cuda.cudadrv.error.NvvmError: Failed to compile

<unnamed> (91, 19): parse expected comma after load's type
NVVM_ERROR_COMPILATION

[XMing0124]

I encountered the same problem,do you fix it?

[ammaryasirnaich]

I encountered the same problem,do you fix it?

Not yet, waiting for help!

[ammaryasirnaich]

I think I have solved the issue by only recreating the development environment. I am sharing information about my environment, @XMing0124 hope it works for you also.

Python: 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05) [GCC 9.3.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 3080
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.3, V11.3.109
GCC: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
PyTorch: 1.9.0a0+c3d40fd
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2019.0.5 Product Build 20190808 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.1.2 (Git Hash N/A)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86
  - CuDNN 8.2.1
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.1, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 
TorchVision: 0.10.0a0
OpenCV: 4.5.5
spconv2.0: True

[ammaryasirnaich]

try to re-install numba with version=0.53.0 and check if it works

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[mrprajesh]

@ammaryasirnaich On a different note, how did you collect the development environment details or which commands to use?