transformers icon indicating copy to clipboard operation
transformers copied to clipboard

Add DETA

Open NielsRogge opened this issue 2 years ago • 6 comments

What does this PR do?

This PR adds DETA. DETA is a slight change to Deformable DETR by using traditional IoU-based assignment as opposed to the Hungarian matching used in the original DETR, and incorporating NMS (non-maximum suppression) in the postprocessing.

Note: this model has a torchvision dependency for NMS.

To do:

  • [ ] transfer checkpoints

NielsRogge avatar Jan 03 '23 13:01 NielsRogge

cc @alaradirik this PR is in a ready state, except for 2 things:

  • [x] whether or not we leverage torchvision's batched_nms => the CI is currently failing because this library is not installed. Will also ask for @sgugger and @LysandreJik's opinion here
  • [ ] the post_process_object_detection method might require an in-depth look

NielsRogge avatar Jan 04 '23 10:01 NielsRogge

There is no problem with the model requiring torchvision to be installed. We have many models with specific dependencies, some of which you ported yourself ;-). Just protect the import between if is_torchvision_available() and have a the first line in the init of the models be a require_backends(["torchvision"]).

sgugger avatar Jan 04 '23 13:01 sgugger

The documentation is not available anymore as the PR was closed or merged.

@sgugger I've addressed all comments, except for adding support for the custom kernel.

Could we perhaps add support for the custom kernel for the 3 models (Mask2Former, OneFormer and DETA) in a separate PR?

NielsRogge avatar Jan 25 '23 14:01 NielsRogge

In this case, remove the code trying to load the custom kernels in the modeling file and we can add it back in the PR that will deal with custom kernels.

sgugger avatar Jan 25 '23 15:01 sgugger

@sgugger ok, feel free to approve :)

NielsRogge avatar Jan 26 '23 12:01 NielsRogge

Failing test is unrelated/flaky, merging.

NielsRogge avatar Jan 31 '23 09:01 NielsRogge