Training the network gives high AUC but low ACC

[erikbabu]

Thanks for open-sourcing this solution. I am using it as the backbone to evaluate a Federated Learning approach to medical imaging for my final year individual undergraduate project.

I trained the model (not using the pre-saved weights) on a 20% sample of the data, which follows the same distribution as the overall training set, and although the AUC scores are quite good, the ACC scores are quite low for some of the observations. I was wondering if you had some insight on what might be causing this. What Acc scores were you getting for your best model?

Atelectasis - AUCROC: 0.871, Acc: 0.375 Cardiomegaly - AUCROC: 0.834, Acc: 0.670 Consolidation - AUCROC: 0.908, Acc: 0.860 Edema - AUCROC: 0.888, Acc: 0.790 Pleural Effusion - AUCROC: 0.901, Acc: 0.335

Also, I am using the CheXpert downsampled dataset (11GB). The config file I am using is similar to the provided one except I have had to reduce the batch size to 8 due to GPU memory constraints.

Thanks!

[aakash-saboo]

Hey the threshold used to calculate accuracy is 0.5 you can change the threshold to reach desired accuracy. You should focus on AUC and not accuracy as accuracy changes with threshold but AUC NOT

[yil8]

@yww211 @deadpoppy I guess we could also consider open sourcing our calibration module that exactly deals with the accuracy/threshold issue lol

[deadpoppy]

Wow! It must be the most powerful tool for this problem. @yil8

[Danie1Hayes]

@yww211 @deadpoppy I guess we could also consider open sourcing our calibration module that exactly deals with the accuracy/threshold issue lol

Hi man, i'm facing the same problem now, higher AUC with lower ACC. I've tried to calculate the cutoff of ROC as the optimal threshold , but that didn't work. Could you share your secret about the calibration module you mentioned? Thanks a lot, best wishes