frederick0329
About the question in FAQ: How can I use TrackIn to identify mislabelled examples?
In your FAQ page, you reply this question as
Aggregating Opponents over several test examples: The premise is that mislabelled training examples will oppose predictions of correctly labelled test points in their vicinity. We therefore expect mislabelled examples to recur in the top opponent lists of several test examples, and aggregating the opponents across a test set will tend to identify incorrectly labelled examples.
I also think to handle it in this way. The problem I encountered here is, top opponent or proponent lists of different test examples are all identical. So, I obtain the exactly same list of opponents for each test examples. The reason of this is because only the loss of prediction point (test point) changes in the score calculation formula for different test points. When we sort TracinCP scores, we actually sort it based on the loss of training point. So we get exactly same lists of top opponents/proponent for each test point.
In the code, scores.append(np.sum(trackin_train['loss_grads'][i] * loss_grad)
Can you confirm the case above? If this is true, what do you suggest? Thanks.
Similar experience. I obtained similar lists of opponents for different test examples. They visually don't make sense.
One of the primary reasons you see the same list of opponents/proponents for different test examples is that the mislabeled/atypical training instance have gradient magnitudes that are significantly larger than other training instances.
Thinking about this intuitively, two largely orthogonal components determine the magnitude of the dot product you show. First is the alignment of the gradient vectors. The second is the (training) gradient magnitudes. Either or both can cause the dot product to have a large magnitude.
This paper discusses this problem extensively and proposes two different fixes (disclaimer: I am one of the authors). The simple fix is to normalize by the training and test gradient magnitudes. We term this modified version of TracIn GAS.
If you have any questions, you can let me know.
