Does intermediate fine-tuning on IN21K improve downstream performance?

[TempleX98]

For example, Merged-38M MIM pretraining -> IN21K finetuning -> downstream task finetuning

[Yuxin-CV]

On ADE20K, IN-21K intermediate finetuning slightly degenerates the performance.