[Feat] add monodet task

[sunjiahao1999]

Motivation

Add monocular detection task for mmdet3d

Modification

Add a new monocular detection task and monocular detection model & corresponding unit tests , including data and config for testing.

[tpoisonooo]

Need update doc https://github.com/open-mmlab/mmdeploy/blob/master/docs/zh_cn/03-benchmark/benchmark.md

[tpoisonooo]

fix lint:

$ python3 -m pip install pre-commit
$ cd /path/to/mmdeploy
$ pre-commit install
$ pre-commit run --all-files
..
$ git commit ..

[sunjiahao1999]

Step 1 : Recompile to add new bev_nms trt op. Step 2 : Download checkpoints in mmdet3d. Step 3 : Test command

python tools/deploy.py \
configs/mmdet3d/monocular-detection/monocular-detection_tensorrt_static-928x1600.py \
${PATH_TO_MMDET3D}/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py \
${PATH_TO_MMDET3D}/checkpoints/centerpoint_02pillar_second_secfpn_circlenms_4x8_cyclic_20e_nus_20210816_064624-0f3299c0.pth \
tests/test_codebase/test_mmdet3d/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg \
--work-dir mmdeploy-models/fcos3d \
--device cuda:0 \
--show

[tpoisonooo]

Please give the whole command you are using for tools/deploy.py.

[Isaac-xie]

When I get the onnx of FCOS3D with the following command, an error is reported.

python mmdeploy/tools/deploy.py mmdeploy/configs/mmdet3d/monocular-detection/monocular-detection_onnxruntime_dynamic.py mmdetection3d/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py mmdetection3d/checkpoints/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth mmdetection3d/demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg --work-dir mmdeploy-models/fcos3d

Error message.

"/media/data2/software/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmdeploy/utils/constants.py", line 15, in get
    raise KeyError(f'Cannot get key by value "{value}" of {cls}')
KeyError: 'Cannot get key by value "MonocularDetection" of <enum \'Task\'>'
2022-10-19 14:34:59,775 - mmdeploy - ERROR - `mmdeploy.apis.pytorch2onnx.torch2onnx` with Call id: 0 failed. exit.

[sunjiahao1999]

@Isaac-xie Have you updated mmdeploy to the latest version? You can check if MonocularDetection in your mmdeploy/utils/constants.py. If not, please pull the latest version to your own repo.

[chegnyanjun]

@sunjiahao1999 ,hi,i met the same problem with Isaac-xie,and i check the mmdeploy/utils/constants.py file ,only find the MonocularDetection in Task，not in SDK_TASK_MAP,which it means only can run fcos3d in python , not in sdk?,Does the mmdeploy has the plan to run fcos3d with sdk? Or if i want to run fcos3d in sdk, i must to recompile the sdk in source by change some codes?

[Isaac-xie]

@sunjiahao1999 Hi, when I inferred the onnx of FCOS3D with the following command, the inference was very slow, about 20 minutes per item.

- python tools/test.py configs/mmdet3d/monocular-detection/monocular-detection_onnxruntime_static.py ../../MM3D_for_nuscenes/mmdetection3d/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune.py --model work_dir/end2end.onnx --out work_dir/out.pkl --device cuda:0

[sunjiahao1999]

@Isaac-xie It possibly caused by GroupNormalization operation. ONNX add GN in opset 18 recently link. Currently it is better to use TensorRT.

[Isaac-xie]

@sunjiahao1999 Thanks for your reply, I found that replacing dcn with conv in the model will improve the inference speed more, about 10s per item. dcn causes slow inference speed. is it a bug?

[ZengyuanYu]

@sunjiahao1999 Hi, when I inferred the onnx of FCOS3D with the following command, the inference was very slow, about 20 minutes per item.

- python tools/test.py configs/mmdet3d/monocular-detection/monocular-detection_onnxruntime_static.py ../../MM3D_for_nuscenes/mmdetection3d/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune.py --model work_dir/end2end.onnx --out work_dir/out.pkl --device cuda:0

hi, Why I can't deploy FCOS3D use below cmd:

 mmdeploy/configs/mmdet3d/monocular-detection/monocular-detection_tensorrt_dynamic-320x320-1600x1600.py \
    mmdetection3d/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py \
    checkpoints/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_20210715_235813-4bed5239.pth \
    mmdetection/demo/demo.jpg \
    --work-dir mmdeploy_model/fcos3d \
    --device cuda \
    --dump-info

error is :

Traceback (most recent call last):
  File "mmdeploy/tools/deploy.py", line 304, in <module>
    main()
  File "mmdeploy/tools/deploy.py", line 125, in main
    export2SDK(
  File "/data/FCOS3D/mmdeploy/mmdeploy/backend/sdk/export_info.py", line 456, in export2SDK
    deploy_info = get_deploy(deploy_cfg, model_cfg, work_dir, device)
  File "/data/FCOS3D/mmdeploy/mmdeploy/backend/sdk/export_info.py", line 372, in get_deploy
    cls_name = task_map[task]['cls_name']
KeyError: <Task.MONOCULAR_DETECTION: 'MonocularDetection'>

[ZengyuanYu]

@Isaac-xie Have you updated mmdeploy to the latest version? You can check if MonocularDetection in your mmdeploy/utils/constants.py. If not, please pull the latest version to your own repo.

hi, i notice mmdeploy/utils/constants.py include MonocularDetection in Class Task, not include in SDK_TASK_MAP. I meet this error. as info below:

Traceback (most recent call last):
  File "mmdeploy/tools/deploy.py", line 304, in <module>
    main()
  File "mmdeploy/tools/deploy.py", line 125, in main
    export2SDK(
  File "/data/FCOS3D/mmdeploy/mmdeploy/backend/sdk/export_info.py", line 456, in export2SDK
    deploy_info = get_deploy(deploy_cfg, model_cfg, work_dir, device)
  File "/data/FCOS3D/mmdeploy/mmdeploy/backend/sdk/export_info.py", line 372, in get_deploy
    cls_name = task_map[task]['cls_name']
KeyError: <Task.MONOCULAR_DETECTION: 'MonocularDetection'>

I pull mmdeploy latest version.

[happyday-lkj]

@Isaac-xie It possibly caused by GroupNormalization operation. ONNX add GN in opset 18 recently link. Currently it is better to use TensorRT.

If I want to deploy Fcos3D, whether the mmdeploy version must less than 1.0???

[ycdhqzhiai]

Use the following command to deploy fcos3d to report errors：

python tools/deploy.py  \
configs/mmdet3d/monocular-detection/monocular-detection_tensorrt_static-928x1600.py  \
../mmdetection3d/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py \ 
../mmdetection3d/checkpoints/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d_finetune_20210717_095645-8d806dc2.pth  \
../mmdetection3d/demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__CAM_BACK__1532402927637525.jpg \
--work-dir mmdeploy-models/fcos3d
 --device cuda:0

AttributeError: 'FCOSMono3DHead' object has no attribute 'bbox_coder'

environment： mmdeploy：0.13.0 mmdetection3d: 0.18.1

[RocketFlash]

Why this feature disappeared in versions >= 1.0.0 ?

[RunningLeon]

Why this feature disappeared in versions >= 1.0.0 ?

yes. Something is missing @sunjiahao1999 hi, could spare some time to create a New PR from this PR and align with mmdeploy(main branch), mmdet3d(main branch)?

[achbogga]

@sunjiahao1999 @RunningLeon Any update on adding this FCOS3D to the main branch?

[achbogga]

Tagging @tpoisonooo @grimoire @Isaac-xie

[evercherish]

@ycdhqzhiai 你好我也遇到了一模一样的问题，请问你最后解决了吗？