RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

[k1234567jkldhd]

Input the command: python demo.py tracking,ddd --load_model ../models/nuScenes_3Dtracking.pth --dataset nuscenes --pre_hm --track_thresh 0.1 --demo ../videos/nuscenes_mini.mp4 --test_focal_length 633 result: /home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/sklearn/utils/linear_assignment_.py:22: FutureWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead. FutureWarning) Running tracking Using tracking threshold for out threshold! 0.1 Fix size testing. training chunk_sizes: [32] input h w: 448 800 heads {'hm': 10, 'reg': 2, 'wh': 2, 'tracking': 2, 'dep': 1, 'rot': 8, 'dim': 3, 'amodel_offset': 2} weights {'hm': 1, 'reg': 1, 'wh': 0.1, 'tracking': 1, 'dep': 1, 'rot': 1, 'dim': 1, 'amodel_offset': 1} head conv {'hm': [256], 'reg': [256], 'wh': [256], 'tracking': [256], 'dep': [256], 'rot': [256], 'dim': [256], 'amodel_offset': [256]} Creating model... Using node type: (<class 'model.networks.dla.DeformConv'>, <class 'model.networks.dla.DeformConv'>) Warning: No ImageNet pretrain!! loaded ../models/nuScenes_3Dtracking.pth, epoch 70 out_name nuscenes_mini.mp4 Initialize tracking! Traceback (most recent call last): File "demo.py", line 128, in demo(opt) File "demo.py", line 83, in demo ret = detector.run(img) File "/home/h/CenterTrack/src/lib/detector.py", line 119, in run images, self.pre_images, pre_hms, pre_inds, return_time=True) File "/home/h/CenterTrack/src/lib/detector.py", line 339, in process output = self.model(images, pre_images, pre_hms)[-1] File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/h/CenterTrack/src/lib/model/networks/base_model.py", line 75, in forward feats = self.imgpre2feats(x, pre_img, pre_hm) File "/home/h/CenterTrack/src/lib/model/networks/dla.py", line 631, in imgpre2feats x = self.base(x, pre_img, pre_hm) File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/h/CenterTrack/src/lib/model/networks/dla.py", line 307, in forward x = self.base_layer(x) File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 343, in forward return self.conv2d_forward(input, self.weight) File "/home/h/anaconda3/envs/CenterTrack/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 340, in conv2d_forward self.padding, self.dilation, self.groups) RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

The code was tested on Ubuntu 20.04, with [Anaconda] Python 3.6, CUDA 10.0, and [PyTorch]v1.10.2.

[unbeliveyu]

Have you solved this problem

[bryanbocao]

@unbeliveyu Try using Docker: https://github.com/bryanbocao/CenterTrack/blob/master/readme/Docker.md https://github.com/xingyizhou/CenterTrack/pull/290

Run CenterTrack using Docker

DCNv2 Compilation Issue Fixed

docker pull bryanbocao/centertrack
docker run -d --ipc=host --shm-size=16384m -it -v /:/share --gpus all --network=bridge bryanbocao/centertrack /bin/bash
docker ps -a

Check <CONTAINER_ID>:

CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS                       PORTS                NAMES
89bb79551ccb   bryanbocao/centertrack   "/usr/local/bin/nvid…"   49 seconds ago   Up 38 seconds                6006/tcp, 8888/tcp   competent_northcutt

docker exec -it <CONTAINE_ID> /bin/bash

docker exec -it 89bb79551ccb /bin/bash

Inside the container:

cd /root/CenterTrack/src/lib/model/networks/DCNv2
python3 setup.py build develop
cd /root/CenterTrack/src/
python3 demo.py tracking,ddd --load_model ../models/nuScenes_3Dtracking.pth --dataset nuscenes --pre_hm --track_thresh 0.1 --demo ../videos/nuscenes_mini.mp4 --test_focal_length 633

The DCNv2 is built from MatthewHowe_DCNv2. Thanks MatthewHowe for the contribution. Then you should be able to see the results.

Go to /share to transfer data between the container and your root system.