May I ask which transfer accuracy used in LogME paper?

[luzai]

Hi,

In LEEP paper, there is two transfer accuracy depending on which transfer learning method is used, re-train head or fine-tune (whole model). Is LogME correlated well with the transfer accuracy of fine-tuning whole model?

If we have a large target-domain dataset and finetune for long enough epoch, the knowledge in pretrained model will be forgot. All pretrained models will have similar accuracy. So we should expect the logME for different models also be similar. Will this similar logME scores happen?

Thank you!

Best, Luzai

[youkaichao]

Hi, we only use the accuracy of fine-tuning.

If you have a large target-domain dataset and finetune for long enough epoch, the knowledge in pretrained model will be forgotten. And if you evaluate LogME scores on these fine-tuned models, I think they would have similar LogME scores.

[luzai]

Hi youkaichao,

Thank you so much for your reply! Do you mean "if you evaluate LogME scores on the pretrained models before finetuning, they would have similar LogME scores"?

I guess it is possible that LogME score is low for one model because the extract feature is not compatible with target domain labels. But after finetuning on large dataset for long epochs, the feature extractor improves and transfer accuracy improves.

I think there is some scenarios that LogME works the best. May I ask about the scenarios? Is it for example target-domain dataset is medium size?

Thank you!

Best, Luzai