mhcnuggets scores on TESLA samples

[vladimirkovacevic]

I was very curious to test the performance of mhcnuggets 2.3 on the latest dataset with TESLA validated neoantigen candidates published in Cell (Key Parameters of Tumor Epitope Immunogenicity Revealed Through a Consortium Approach Improve Neoantigen Prediction, Table S4 and S7). In TESLA they used flow cytometry and microscopy to confirm (VALIDATE) neoantigen candidates. More info is available in the paper. Here are ic50 scores obtained by mhcnuggets. The distributions of confirmed and not confirmed candidates are in a very similar range: Comfirmed candidates(041): Median=5598.78, Mean=7441.88(+-9090.58) Not confirmed candidates(871): Median=5605.33, Mean=9153.18(+-10031.29) When mhcnuggets scores are normalized (with added minus sign since smaller ic50 is better) and compared against VALIDATED AUC score is 0.513, precision is 0.0596, here is the output comparison figure:

Here is the table with obtained scores as a reference and command used for one of the mhcnuggets runs: python /mhcnuggets/mhcnuggets/src/predict.py -c I --allele HLA-C*05:01 --peptides merged.fasta -o mhcnuggets_HLA-C*05:01.peps

Are these results expected? Could it be that this use case is out of the scope for mhcnuggets or it requires some additional tuning?