Grounding DINO missing custom kernels

[sam-ulrich1]

System Info

Latest including master as of 5/11/24 Ubuntu, 4090 GPU

Who can help?

@amyeroberts

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction

When running any GroundingDINO model you get the following. I have confirmed the kernels do not exist in the transformers repo:

Could not load the custom kernel for multi-scale deformable attention: [Errno 2] No such file or directory: '/.../transformers-main/src/transformers/kernels/grounding_dino/vision.cpp'".
Could not load the custom kernel for multi-scale deformable attention: [Errno 2] No such file or directory: '.../transformers-main/src/transformers/kernels/grounding_dino/vision.cpp'".

Expected behavior

Either the kernels are not necessary and should be removed or they are necessary and causing performance issues. GroundingDINO does seem to under perform in transformers but this is purely anecdotal