LGM-Net icon indicating copy to clipboard operation
LGM-Net copied to clipboard

high performance is caused by testing label peeked through data augmentation?

Open terryphillip87 opened this issue 4 years ago • 0 comments

hi, I have gone through your code and found some issues regarding the extraordinarily high performance.

1.in this line, https://github.com/likesiwell/LGM-Net/blob/55c035fb0f4d4a06c6af7dbf76381107580e9894/experiment_builder.py#L186 the data augmentation is enabled during testing, and in https://github.com/likesiwell/LGM-Net/blob/55c035fb0f4d4a06c6af7dbf76381107580e9894/data.py#L139, images from the same class are assigned with the same rotation angle. In this case, the testing query images are rotated aligned with the ground truth class, which means the query images already knows the correct label through rotations. I have run your code by disabling the data augmentation during testing, and the performance drops over 20%. It is counterintuitive that a data-augmentation trick in the testing phase can have a significant impact on the performance. I would like the author to make some clarifications regarding this issue.

terryphillip87 avatar Mar 27 '20 05:03 terryphillip87