[WIP]: Refactor DETR and Deformable DETR on 3.x

[Li-Qingyun]

Motivation

We (me, @jshilong, @LYMDLUT, and @KeiChiTse) are going to refactor DETR-like models, to enhance the usability and readability of our codebase.

[Li-Qingyun]

Short-term Plan

We plan to implement a base detector class for Transformer-based detector, TransformerDetector, move the transformer-related part in the original head module to the new detector module, and use TransformerEncoder and TransformerDecoder directly in the detector.

NOTE we are still exploring better refactoring solutions, and the short-term plans are only tried temporarily. It is welcome to give suggestions for refactor and report bugs, in the comments or reviews, or contact me. (wechat:18545525156 or email: [email protected]).

[CLAassistant]

All committers have signed the CLA.

[Li-Qingyun]

Short-term Plan

We decided to refactor the forward process of Transformer in the TransformerDetector class. Specifically, the whole forward process is wrapped with forward_Transformer, which is divided into about four parts: pre_transformer, forward_encoder, pre_decoder, forward_decoder.

NOTE we are still exploring better refactoring solutions, and the short-term plans are only tried temporarily. It is welcome to give suggestions for refactor and report bugs, in the comments or reviews, or contact me. (wechat:18545525156 or email: [email protected]).

[jshilong]

The overall design LGTM now, this pr may be merged after benchmarking the DETR and DeformDETR

[Li-Qingyun]

The design LGTM. Only need to fix minor comments +1

Thank you very much for the review and the suggestions! The comments have been fixed! @Tai-Wang

[RangiLyu]

The whole design LGTM.

[Li-Qingyun]

The whole design LGTM.

Thanks very much for your review and the suggestions! The comments have been replied and modifications have been committed. @RangiLyu