Very low mAP on binary classification

[aeroned]

I am training YOLOX-s model for binary classification. The dataset consists of 7000 images with 1 object in the center. On ResNet, the accuracy is over 90%, but with YOLOX it's only 55%. These are the configurations in my exp file:

self.num_classes = 2
self.no_aug_epochs = 20
self.max_epochs = 20
self.data_num_workers = 8
self.input_size = (512, 512)
self.test_size = (512, 512)
self.mosaic_prob = 0
self.hsv_prob = 0
self.mixup_prob = 0
self.enable_mixup = False
self.multiscale_range = 0

def random_resize(self, data_loader, epoch, rank, is_distributed):
    return self.input_size

Is there any suggestion/improvement to my situation?