Mitchell Wortsman

Not at the moment.. It's definitely a possible extension!

Hi, Sorry but I'm super confused about this issue, do you mind elaborating? Thank you, Mitchell

My guess is also a shuffling issue with webdataset when these were run

Looking at this again I wonder if it is caused by the `scale` param, which also exhibits stair-like behaviour. I would expect that stair-like `scale` => stair-like loss. But I...

Interesting. I wonder how accuracy/loss would be impacted if this learnable param was replaced by a scheduled param---something like 100 - k*cosine_decay(iteration).

I feel like there's a better solution to this so that you don't have to add that to PYTHONPATH -- anybody know it?

Then I will re-open this issue and mark as `good first issue` so that someone can submit a PR with this change.

This is already supported by https://github.com/mlfoundations/wise-ft , would be nice to merge

Hello, is this issue helpful https://github.com/mlfoundations/open_clip/issues/18 TL;DR: use `--openai-pretrained`

Depends what your aim is, what exactly are you fine-tuning on? We have another repository for fine-tuning pre-trained CLIP on things like ImageNet/CIFAR etc. https://github.com/mlfoundations/wise-ft and an associated preprint https://arxiv.org/abs/2109.01903.