Linear prob evaluation on ImageNet

[vtddggg]

Thanks for your wonderful work!

I have some questions about linear prob evaluation on ImageNet:

1. how do you conduct linear prob evaluation on ImageNet? using sklearn.linear_model.LogisticRegression or finetuning fc layer like this script
2. if answer of 1) is Yes. I found do LogisticRegression on ImageNet is very costly. Can I ask for your time spend on ImageNet linear prob evaluation experiment ?

[jongwook]

Hi,

1. Yes, but we later found that a PyTorch version can work as equivalently on linear probes.
2. Please see https://github.com/openai/CLIP/issues/64#issuecomment-804444364 for more details

[mingtan2]

a PyTorch version can work as equivalently on linear probes. Hi @jongwook , could you share the details or script for the PyTorch version? Thanks.

[jyuan1118]

Did you found the equivalently pytorch verson?

[jongwook]

These both worked well for us:

• PyTorch's stock LBFGS: https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html
• A third-party implementation that has more features: https://github.com/hjmshi/PyTorch-LBFGS