mmdeploy icon indicating copy to clipboard operation
mmdeploy copied to clipboard

Published 20 hours ago • verified publisher icon open-mmlab

Error generating model from ONNX to Tensorrt on Jetson AGX Orin 32GB with centerpoint model

Open H329-2 opened this issue 3 years ago • 6 comments
trafficstars

I using the deploy.py in tools to get a TensorRT engine of centerpoint from Pytorch model on Jetson AGX Orin 32GB, and there are 'nan' in the result of engine _inference.

But when I deploy the model with same method on PC with Nvidia RTX, I get correct outputs. and the ONNX from jetson can convert to TensorRT engine correctly on PC.

Reproduction

I use command below to get the engine :

python tools/deploy.py \ ./configs/mmdet3d/voxel-detection/voxel-detection_static.py \ ../mmdetection3d/configs/centerpoint/centerpoint_02pillar_second_secfpn_4x8_cyclic_20e_nus.py \ ./work_dirs/centerpoint/epoch_20.pth /home/iair/Documents/mmdeploy/work_dirs/centerpoint \ /n015-2018-11-21-19-38-26+0800__LIDAR_TOP__1542801007446751.pcd.bin --work-dir \ ./work_dirs/centerpoint/ \ --device cuda:0 \ --show

There was no bbox in visualization.Then I used the TensorRT engine to run inferencing and get nan in the result, while the ONNX model could get a correct result.

Environment

sys.platform: linux Python: 3.8.13 (default, Mar 28 2022, 10:59:05) [GCC 10.2.0] CUDA available: True GPU 0: Orin CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 11.4, V11.4.239 GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 PyTorch: 1.11.0 PyTorch compiling details: PyTorch built with:

  • GCC 9.4
  • C++ Version: 201402
  • OpenMP 201511 (a.k.a. OpenMP 4.5)
  • LAPACK is enabled (usually provided by MKL)
  • NNPACK is enabled
  • CPU capability usage: NO AVX
  • CUDA Runtime 11.4
  • NVCC architecture flags: -gencode;arch=compute_72,code=sm_72;-gencode;arch=compute_87,code=sm_87
  • CuDNN 8.4.1
    • Built with CuDNN 8.3.2
  • Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CUDA_VERSION=11.4, CUDNN_VERSION=8.3.2, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=open, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=ON, USE_NCCL=0, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.12.0 OpenCV: 4.6.0 MMCV: 1.6.0 MMCV Compiler: GCC 9.4 MMCV CUDA Compiler: 11.4 MMDeploy: 0.7.0+83b11bc Backend information onnxruntime: 1.12.1 ops_is_avaliable : False tensorrt: 8.4.1.5 ops_is_avaliable : True ncnn: None ops_is_avaliable : False pplnn_is_avaliable: False openvino_is_avaliable: False snpe_is_available: False Codebase information mmdet: 2.25.1 mmseg: 0.27.0 mmcls: 0.23.2 mmocr: None mmedit: None mmdet3d: 1.0.0rc4 mmpose: None mmrotate: None

H329-2 avatar Aug 22 '22 13:08 H329-2

"But when I deploy the model with same method on PC with Nvidia RTX, I get correct outputs." Is the TensorRT version on your PC the same as that on Orin?

lvhan028 avatar Aug 23 '22 02:08 lvhan028

Yes. I use jetpack 5.0.2 on the Jetson AGX Orin, the corresponding TensorRT version is 8.4.1.5. And I have tried to use the same version of TensorRT on my PC, which can get the correct outputs.

H329-2 avatar Aug 23 '22 03:08 H329-2

I built an engine of faster_rcnn with mmdeploy on Jetson AGX Orin this morning and it conducted a correct result, but the engine building for centerpoint was still error. @lvhan028

H329-2 avatar Aug 23 '22 06:08 H329-2

We suppose that scatterND opr abnormal:

  • issue author split centerpoint into two parts, scatterND indices and updates are all available
  • use polygraphy to save scaterND input/output during inference the whole .engine model, then load input with base64 and numpy, scatterND indices out of range

tpoisonooo avatar Aug 31 '22 07:08 tpoisonooo

I have borrowed a jetson orin, centerpoint works.

Here is my env (jetpack5.0.2 & trt8.4.1.5), hope it helps.

$ python3 tools/check_env.py
(py38) i@ubuntu:~/konghuanjun/mmdeploy$ python3 tools/check_env.py 
2022-09-14 19:24:12,482 - mmdeploy - INFO - 

2022-09-14 19:24:12,483 - mmdeploy - INFO - **********Environmental information**********
2022-09-14 19:24:12,907 - mmdeploy - INFO - sys.platform: linux
2022-09-14 19:24:12,908 - mmdeploy - INFO - Python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:56:18) [GCC 10.3.0]
2022-09-14 19:24:12,908 - mmdeploy - INFO - CUDA available: True
2022-09-14 19:24:12,908 - mmdeploy - INFO - GPU 0: Orin
2022-09-14 19:24:12,908 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda-11.4
2022-09-14 19:24:12,908 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.4, V11.4.239
2022-09-14 19:24:12,908 - mmdeploy - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2022-09-14 19:24:12,908 - mmdeploy - INFO - PyTorch: 1.11.0
2022-09-14 19:24:12,908 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 9.4
  - C++ Version: 201402
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - CUDA Runtime 11.4
  - NVCC architecture flags: -gencode;arch=compute_72,code=sm_72;-gencode;arch=compute_87,code=sm_87
  - CuDNN 8.4.1
    - Built with CuDNN 8.3.2
  - Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CUDA_VERSION=11.4, CUDNN_VERSION=8.3.2, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=open, TORCH_VERSION=1.11.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=ON, USE_NCCL=0, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

2022-09-14 19:24:12,908 - mmdeploy - INFO - TorchVision: 0.11.1
2022-09-14 19:24:12,908 - mmdeploy - INFO - OpenCV: 4.6.0
2022-09-14 19:24:12,908 - mmdeploy - INFO - MMCV: 1.5.2
2022-09-14 19:24:12,909 - mmdeploy - INFO - MMCV Compiler: GCC 9.4
2022-09-14 19:24:12,909 - mmdeploy - INFO - MMCV CUDA Compiler: 11.4
2022-09-14 19:24:12,909 - mmdeploy - INFO - MMDeploy: 0.8.0+b310ba8
2022-09-14 19:24:12,909 - mmdeploy - INFO - 

2022-09-14 19:24:12,909 - mmdeploy - INFO - **********Backend information**********
2022-09-14 19:24:13,671 - mmdeploy - INFO - onnxruntime: None   ops_is_avaliable : False
2022-09-14 19:24:13,719 - mmdeploy - INFO - tensorrt: 8.4.1.5   ops_is_avaliable : True
2022-09-14 19:24:13,746 - mmdeploy - INFO - ncnn: None  ops_is_avaliable : False
2022-09-14 19:24:13,748 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-09-14 19:24:13,749 - mmdeploy - INFO - openvino_is_avaliable: False
2022-09-14 19:24:13,775 - mmdeploy - INFO - snpe_is_available: False
2022-09-14 19:24:13,776 - mmdeploy - INFO - ascend_is_available: False
2022-09-14 19:24:13,778 - mmdeploy - INFO - coreml_is_available: False
2022-09-14 19:24:13,778 - mmdeploy - INFO - 

2022-09-14 19:24:13,778 - mmdeploy - INFO - **********Codebase information**********
2022-09-14 19:24:13,783 - mmdeploy - INFO - mmdet:      2.25.0
2022-09-14 19:24:13,783 - mmdeploy - INFO - mmseg:      0.25.0
2022-09-14 19:24:13,783 - mmdeploy - INFO - mmcls:      0.23.2
2022-09-14 19:24:13,783 - mmdeploy - INFO - mmocr:      None
2022-09-14 19:24:13,783 - mmdeploy - INFO - mmedit:     None
2022-09-14 19:24:13,783 - mmdeploy - INFO - mmdet3d:    1.0.0rc4
2022-09-14 19:24:13,783 - mmdeploy - INFO - mmpose:     None
2022-09-14 19:24:13,783 - mmdeploy - INFO - mmrotate:   None

$ sudo apt-cache show nvidia-jetpack
[sudo] password for i: 
Package: nvidia-jetpack
Version: 5.0.2-b231
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 5.0.2-b231), nvidia-jetpack-dev (= 5.0.2-b231)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_5.0.2-b231_arm64.deb
Size: 29304
SHA256: b1268b2cb969e677163f291967bc7542371a29d536379df3f7dfa1f247ff3fab
SHA1: 7ff288a771b83eec8f80a41ccf0eec490f32e10a
MD5sum: 5cc57807b33630d8edb249e53daf58ed
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

tpoisonooo avatar Sep 15 '22 02:09 tpoisonooo

mmdeploy commit-id :

$ git log
commit b310ba8d3ebf44f0a542ad2953681af3bb7e7ccb (grafted, HEAD -> master, tag: v0.8.0, origin/master, origin/HEAD)

tpoisonooo avatar Sep 15 '22 02:09 tpoisonooo

@tpoisonooo I tried the same environment as yours on Jetson NX (jetpack5.0.2 & trt8.4.1.5).

(mmdeploy) jz@jz-desktop:~/yxb/mmdeploy$ python tools/check_env.py
2022-09-27 11:42:41,232 - mmdeploy - INFO -

2022-09-27 11:42:41,232 - mmdeploy - INFO - **********Environmental information**********
2022-09-27 11:42:42,401 - mmdeploy - INFO - sys.platform: linux
2022-09-27 11:42:42,403 - mmdeploy - INFO - Python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:56:18) [GCC 10.3.0]
2022-09-27 11:42:42,404 - mmdeploy - INFO - CUDA available: True
2022-09-27 11:42:42,404 - mmdeploy - INFO - GPU 0: Xavier
2022-09-27 11:42:42,404 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda-11.4
2022-09-27 11:42:42,404 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.4, V11.4.239
2022-09-27 11:42:42,405 - mmdeploy - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2022-09-27 11:42:42,405 - mmdeploy - INFO - PyTorch: 1.12.0a0+2c916ef.nv22.3
2022-09-27 11:42:42,405 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 9.4
  - C++ Version: 201402
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - CPU capability usage: NO AVX
  - CUDA Runtime 11.4
  - NVCC architecture flags: -gencode;arch=compute_62,code=sm_62;-gencode;arch=compute_72,code=sm_72;-gencode;arch=compute_87,code=sm_87
  - CuDNN 8.4.1
    - Built with CuDNN 8.3.2
  - Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CUDA_VERSION=11.4, CUDNN_VERSION=8.3.2, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=open, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=0, USE_NNPACK=0, USE_OPENMP=ON, USE_ROCM=OFF,

2022-09-27 11:42:42,405 - mmdeploy - INFO - TorchVision: 0.11.1
2022-09-27 11:42:42,405 - mmdeploy - INFO - OpenCV: 4.6.0
2022-09-27 11:42:42,406 - mmdeploy - INFO - MMCV: 1.6.0
2022-09-27 11:42:42,406 - mmdeploy - INFO - MMCV Compiler: GCC 9.4
2022-09-27 11:42:42,406 - mmdeploy - INFO - MMCV CUDA Compiler: 11.4
2022-09-27 11:42:42,406 - mmdeploy - INFO - MMDeploy: 0.8.0+b310ba8
2022-09-27 11:42:42,406 - mmdeploy - INFO -

2022-09-27 11:42:42,407 - mmdeploy - INFO - **********Backend information**********
2022-09-27 11:42:44,297 - mmdeploy - INFO - onnxruntime: None   ops_is_avaliable : False
2022-09-27 11:42:44,688 - mmdeploy - INFO - tensorrt: 8.4.1.5   ops_is_avaliable : True
2022-09-27 11:42:44,893 - mmdeploy - INFO - ncnn: None  ops_is_avaliable : False
2022-09-27 11:42:45,038 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-09-27 11:42:45,045 - mmdeploy - INFO - openvino_is_avaliable: False
2022-09-27 11:42:45,114 - mmdeploy - INFO - snpe_is_available: False
2022-09-27 11:42:45,261 - mmdeploy - INFO - ascend_is_available: False
2022-09-27 11:42:45,268 - mmdeploy - INFO - coreml_is_available: False
2022-09-27 11:42:45,269 - mmdeploy - INFO -

2022-09-27 11:42:45,269 - mmdeploy - INFO - **********Codebase information**********
2022-09-27 11:42:45,281 - mmdeploy - INFO - mmdet:      2.25.2
2022-09-27 11:42:45,281 - mmdeploy - INFO - mmseg:      0.27.0
2022-09-27 11:42:45,281 - mmdeploy - INFO - mmcls:      0.23.2
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmocr:      None
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmedit:     None
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmdet3d:    1.0.0rc4
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmpose:     None
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmrotate:   None

(mmdeploy) jz@jz-desktop:~/yxb/mmdeploy$ sudo apt-cache show nvidia-jetpack
[sudo] password for jz:
Package: nvidia-jetpack
Version: 5.0.2-b231
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 5.0.2-b231), nvidia-jetpack-dev (= 5.0.2-b231)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_5.0.2-b231_arm64.deb
Size: 29304
SHA256: b1268b2cb969e677163f291967bc7542371a29d536379df3f7dfa1f247ff3fab
SHA1: 7ff288a771b83eec8f80a41ccf0eec490f32e10a
MD5sum: 5cc57807b33630d8edb249e53daf58ed
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

then I use the command: python ./tools/deploy.py configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic-nus-20x5.py ../mmdetection3d/configs/centerpoint/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus.py ../mmdetection3d/checkpoints/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus_20201004_170716-a134a233.pth ../mmdetection3d/demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402936198962.pcd.bin --work-dir mmdet3d_centerpoint_deploy_8.4 --device cuda:0

There is no bbox in visualization and get nan in the result. Is there any thing that makes a difference?

speshowBUAA avatar Sep 27 '22 03:09 speshowBUAA

Have you tried pointpillars which output_pytorch works fine but output_tensorrt remains Nan equally?

python ./tools/deploy.py configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic-nus-64x4.py ../mmdetection3d/configs/pointpillars/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d.py ../mmdetection3d/checkpoints/hv_pointpillars_secfpn_sbn-all_4x8_2x_nus-3d_20210826_225857-f19d00a3.pth ../mmdetection3d/demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402936198962.pcd.bin --work-dir mmdet3d_nus_deploy/ --device cuda:0

speshowBUAA avatar Sep 27 '22 03:09 speshowBUAA

@tpoisonooo I tried the same environment as yours on Jetson NX (jetpack5.0.2 & trt8.4.1.5).

(mmdeploy) jz@jz-desktop:~/yxb/mmdeploy$ python tools/check_env.py
2022-09-27 11:42:41,232 - mmdeploy - INFO -

2022-09-27 11:42:41,232 - mmdeploy - INFO - **********Environmental information**********
2022-09-27 11:42:42,401 - mmdeploy - INFO - sys.platform: linux
2022-09-27 11:42:42,403 - mmdeploy - INFO - Python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:56:18) [GCC 10.3.0]
2022-09-27 11:42:42,404 - mmdeploy - INFO - CUDA available: True
2022-09-27 11:42:42,404 - mmdeploy - INFO - GPU 0: Xavier
2022-09-27 11:42:42,404 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda-11.4
2022-09-27 11:42:42,404 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.4, V11.4.239
2022-09-27 11:42:42,405 - mmdeploy - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2022-09-27 11:42:42,405 - mmdeploy - INFO - PyTorch: 1.12.0a0+2c916ef.nv22.3
2022-09-27 11:42:42,405 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 9.4
  - C++ Version: 201402
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - CPU capability usage: NO AVX
  - CUDA Runtime 11.4
  - NVCC architecture flags: -gencode;arch=compute_62,code=sm_62;-gencode;arch=compute_72,code=sm_72;-gencode;arch=compute_87,code=sm_87
  - CuDNN 8.4.1
    - Built with CuDNN 8.3.2
  - Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CUDA_VERSION=11.4, CUDNN_VERSION=8.3.2, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=open, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=0, USE_NNPACK=0, USE_OPENMP=ON, USE_ROCM=OFF,

2022-09-27 11:42:42,405 - mmdeploy - INFO - TorchVision: 0.11.1
2022-09-27 11:42:42,405 - mmdeploy - INFO - OpenCV: 4.6.0
2022-09-27 11:42:42,406 - mmdeploy - INFO - MMCV: 1.6.0
2022-09-27 11:42:42,406 - mmdeploy - INFO - MMCV Compiler: GCC 9.4
2022-09-27 11:42:42,406 - mmdeploy - INFO - MMCV CUDA Compiler: 11.4
2022-09-27 11:42:42,406 - mmdeploy - INFO - MMDeploy: 0.8.0+b310ba8
2022-09-27 11:42:42,406 - mmdeploy - INFO -

2022-09-27 11:42:42,407 - mmdeploy - INFO - **********Backend information**********
2022-09-27 11:42:44,297 - mmdeploy - INFO - onnxruntime: None   ops_is_avaliable : False
2022-09-27 11:42:44,688 - mmdeploy - INFO - tensorrt: 8.4.1.5   ops_is_avaliable : True
2022-09-27 11:42:44,893 - mmdeploy - INFO - ncnn: None  ops_is_avaliable : False
2022-09-27 11:42:45,038 - mmdeploy - INFO - pplnn_is_avaliable: False
2022-09-27 11:42:45,045 - mmdeploy - INFO - openvino_is_avaliable: False
2022-09-27 11:42:45,114 - mmdeploy - INFO - snpe_is_available: False
2022-09-27 11:42:45,261 - mmdeploy - INFO - ascend_is_available: False
2022-09-27 11:42:45,268 - mmdeploy - INFO - coreml_is_available: False
2022-09-27 11:42:45,269 - mmdeploy - INFO -

2022-09-27 11:42:45,269 - mmdeploy - INFO - **********Codebase information**********
2022-09-27 11:42:45,281 - mmdeploy - INFO - mmdet:      2.25.2
2022-09-27 11:42:45,281 - mmdeploy - INFO - mmseg:      0.27.0
2022-09-27 11:42:45,281 - mmdeploy - INFO - mmcls:      0.23.2
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmocr:      None
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmedit:     None
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmdet3d:    1.0.0rc4
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmpose:     None
2022-09-27 11:42:45,282 - mmdeploy - INFO - mmrotate:   None

(mmdeploy) jz@jz-desktop:~/yxb/mmdeploy$ sudo apt-cache show nvidia-jetpack
[sudo] password for jz:
Package: nvidia-jetpack
Version: 5.0.2-b231
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 5.0.2-b231), nvidia-jetpack-dev (= 5.0.2-b231)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_5.0.2-b231_arm64.deb
Size: 29304
SHA256: b1268b2cb969e677163f291967bc7542371a29d536379df3f7dfa1f247ff3fab
SHA1: 7ff288a771b83eec8f80a41ccf0eec490f32e10a
MD5sum: 5cc57807b33630d8edb249e53daf58ed
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

then I use the command: python ./tools/deploy.py configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic-nus-20x5.py ../mmdetection3d/configs/centerpoint/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus.py ../mmdetection3d/checkpoints/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus_20201004_170716-a134a233.pth ../mmdetection3d/demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402936198962.pcd.bin --work-dir mmdet3d_centerpoint_deploy_8.4 --device cuda:0

There is no bbox in visualization and get nan in the result. Is there any thing that makes a difference?

trt8.5 has fixed it.

tpoisonooo avatar Sep 28 '22 05:09 tpoisonooo

Close this issue for NVIDIA has fixed this bug here https://github.com/NVIDIA/TensorRT/issues/2338. If you have more question, please open a new issue.

tpoisonooo avatar Oct 17 '22 06:10 tpoisonooo