Clarification on in-between class samples results

[pmgautam]

In the paper, Figure 5b (Analysis for in-between class samples) is explained as the probability to predict neither two classes by varying the combination ratio lambda. Does it mean that if the sample is created by mixing cat and dog, then the model should predict labels other than either of the cat or dog? What is the significance of this result? Maybe I am missing something here. Also, there are separate plots for Mixup and Cutmix methods and in each plot, various methods are compared. How to interpret the results. Maybe the question is basic but I'm not able to understand these results. Hope someone can answer my queries. I've attached the Figure for reference. Thanks in advance!

[SanghyukChun]

Hi, thanks for your question.

if the sample is created by mixing cat and dog, model should predict labels other than either of the cat or dog?

Yes, we first mix two images and make a prediction by models with different training techniques. There are two different strategies to mix two images, Mixup and CutMix (of course, we worked with this paper three years ago, so now there are more mixing strategies)

The left figure denotes the model accuracies by the mixup-based mixing strategy. (the right figure is for CutMix-based mixing strategy)

You need a full figure to understand the main argument of this section: "CutMix shows very robust generalization performances compared to Mixup and Cutout in various scenarios"

We can observe four observations here

1. Cutout is not helpful for improving in-between sample prediction (but it helps occluded sample accuracies, because it is trained by occluded samples)
2. Mixup is not helpful for improving performances under occlusions, while CutMix shows almost similar performances to Cutout.
3. Naturally CutMix-trained model will show the best CutMix in-between sample prediction because it is trained by CutMix-ed samples. And so does Mixup. (Each of them shows the best or the second-best performances in each figure)
4. However, CutMix shows very robust performances (the errors are not increased too much) while Mixup shows the very low performance when the mixing ratio is high (0.5).

Not included in our paper, but CutMix works well in other types of robustness benchmarks: ImageNet-A https://arxiv.org/abs/1907.07174

Also, if you stack more training techniques (such as ReLabel https://arxiv.org/abs/2101.05022), you will get better generalization results in various robustness benchmarks.

[pmgautam]

Thank you