Have you tried using non-ImageNet-pretrained backbones?

[vadimkantorov]

E.g. resnet50-swav?

[shenyunhang]

I did not try the unsupervised or self-supervised backbones. It could be an interesting work.