[FEATURE] ReLabel ImageNet support

[hiyyg]

[rwightman]

Doesn't really make sense as it was built for eval. The 'real' labels are only for validation set.

[hiyyg]

I mean ReLabled ImageNet, it improved the labels of training set, from a new CVPR 2021 paper.

[rwightman]

I didn't realize that existed and assumed you were referring to 'ReaL', link for reference https://github.com/naver-ai/relabel_imagenet

I'll take a take a closer look at some point, and consider adding support for training with ReLabel. The current weights available don't look too compelling, but maybe with better aug and regularization they would be.

[hellbell]

@hiyyg @rwightman Thank you for having interests in our ReLabel-ImageNet work. Indeed, I have conducted our ReLabel technic on this repository (timm) with EfficientNet variants, and their results are like this,

Arch	Vanilla train	ReLabel train
EfficientNet-B0	77.4	78.0 (+0.6)
EfficientNet-B1	79.2	80.3 (+1.1)
EfficientNet-B2	80.3	81.0 (+0.7)
EfficientNet-B3	81.7	82.5 (+0.8)

Maybe I can share how to add ReLabel training on timm :) Thanks!

[rwightman]

@hellbell I'd like to explore this but I have other priorities right now. One comment though. This technique leverages additional data, so it's reasonable to expect improved performance.

Any model based on input that leverages additional data (JFT-300M, ImageNet21k, Instagram 1B, etc) or additional human labelling/cleaning for training can no longer longer be considered as 'just ImageNet-1k.' It puts them in a different class of result, along side models that are pretrained on larger datasets, or models using techniques like Noisy Student, Meta Pseudo Label, etc that leverage additional data (labeled or unlabeled) in semi-supervised learning.

[hellbell]

@rwightman I agree. The techniques including ours, semi-supervised learning, knowledge distillation, pseudo-labeling methods should be distinct from other ImageNet-1k-training methods. Anyway, our method barely needs additional computation costs compared to other methods.

[liu09114]

@hiyyg @rwightman Thank you for having interests in our ReLabel-ImageNet work. Indeed, I have conducted our ReLabel technic on this repository (timm) with EfficientNet variants, and their results are like this,

Arch Vanilla train ReLabel train EfficientNet-B0 77.4 78.0 (+0.6) EfficientNet-B1 79.2 80.3 (+1.1) EfficientNet-B2 80.3 81.0 (+0.7) EfficientNet-B3 81.7 82.5 (+0.8) Maybe I can share how to add ReLabel training on timm :) Thanks!

Here is my result. Greedy network enlarging

[Jesthut]

Hi @hellbell, ReLabel is very interesting, here is another solution for efficient knowledge distillation (https://github.com/szq0214/fkd), what do you think?