Issue updating to `torchvision==0.12.0`

[dnth]

We are targeting to integrate FCOS from torchvision==0.12.0 which depends on torch==1.11.0.

But at the same time we need torch==1.10.0 because mmdet==2.17.0 depends on it.

We could not update mmdet to a higher version because VFNet breaks from version 2.17.0 onward. The VFNet error is shown in https://github.com/open-mmlab/mmdetection/issues/6851.