[WIP][Feature] Support TransFusion

[wep21]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Please describe the motivation of this PR and the goal you want to achieve through this PR.

import TransFusion as project

Modification

Please briefly describe what modification is made in this PR. add transfusion implementation

BC-breaking (Optional)

Does the modification introduce changes that break the back-compatibility of the downstream repos? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[CLAassistant]

All committers have signed the CLA.

[JingweiZhang12]

@wep21 Thanks for your awesome contribution, and we're excited to invite you to join core developer community of mmdet3d. Just add our assistant using the WeChat ID: openmmlabwx. When sending the friend request, remember to include the remark "mmsig + Github ID". We will invite you to MMDet3D developer group.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (0ef13b8) 47.26% compared to head (2587d4f) 47.26%.

Additional details and impacted files

@@           Coverage Diff            @@
##           dev-1.x    #2547   +/-   ##
========================================
  Coverage    47.26%   47.26%           
========================================
  Files          277      277           
  Lines        23415    23415           
  Branches      3655     3655           
========================================
  Hits         11067    11067           
  Misses       11635    11635           
  Partials       713      713           

Flag	Coverage Δ
unittests	47.26% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[JingweiZhang12]

Could you update your progress of implementation of TransFusion according to the bellow items:

• [ ] Support inference
• [ ] Align the precision of inference
• [ ] Support training
• [ ] Align the precision of training
• [ ] Provide docs

[JingweiZhang12]

@wep21 Hi, have you aligned the inference precision of TransFusion(only LiDAR modality)?

[wep21]

@JingweiZhang12

Hi, have you aligned the inference precision of TransFusion(only LiDAR modality)?

I'm sorry, but not yet. I've tested with only nuscenes-mini. Currently, I don't have enough gpu environment, it may take a time to train and test with nuscenes full dataset.

[JingweiZhang12]

@JingweiZhang12

Hi, have you aligned the inference precision of TransFusion(only LiDAR modality)?

I'm sorry, but not yet. I've tested with only nuscenes-mini. Currently, I don't have enough gpu environment, it may take a time to train and test with nuscenes full dataset.

That's OK. Suggest you can make sure the results of one forward are the same as that of the official implementation firstly.

[sunjiahao1999]

@wep21 Hi, any progress on TransFusion so far?

[wep21]

@sunjiahao1999 I'm sorry for late reply. This is the evaluation result with nuscenes dataset after 20 epoch training.

Evaluating bboxes of pred_instances_3d
mAP: 0.5562                                                                                                                                                                                                                                                                  
mATE: 0.3236
mASE: 0.2586
mAOE: 0.3608
mAVE: 0.3053
mAAE: 0.1887
NDS: 0.6344

[wep21]

20230819_092709.log

[wep21]

This figure comes from the original paper of transfusion. The second row shows the score in the same condition as my experiment.

[wep21]

inference demo

python demo/pcd_demo.py demo/data/nuscenes/n015-2018-07-24-11-22-45+0800__LIDAR_TOP__1532402927647951.pcd.bin projects/TransFusion/configs/transfusion_lidar_pillar02_second_secfpn_8xb2-cyclic-20e_nus-3d.py ~/Downloads/mmdet3d_epoch20.pth --show --score-thr 0.05

[JingweiZhang12]

@wep21 Please fix the lint.

[wep21]

@JingweiZhang12 I fixed lint at https://github.com/open-mmlab/mmdetection3d/pull/2547/commits/2cb5f7876fcb58a5ce0182581a85174056265b3c.

[wep21]

@JingweiZhang12 @sunjiahao1999 friendly ping

[friendship1]

I was looking for this transfusion code. Thank you.

Just to double-check, are the 54.4 and 64.7 values in the table in the paper you posted (https://github.com/open-mmlab/mmdetection3d/pull/2547#issuecomment-1690360882), 2nd row, left column, the ones you're comparing to (https://github.com/open-mmlab/mmdetection3d/pull/2547#issuecomment-1690340922)? In other words, is it correct that your experiment yielded higher numbers than the paper (just for pillar based, lidar-only)?

Also, I know that the original Transfusion paper used a "fading strategy", and I was wondering if this was applied in your code or in your experiments.

Thanks the great work!

[friendship1]

I just checked that the fading strategy is implemented as a custom hook.

But what you implemented doesn't seem to use a 2-stage learning method: pretrain only the LiDAR model first and then further train this model on additional modality with camera input added. is that correct?