Ross Wightman

get_reduction_mask can be used https://github.com/mlfoundations/open_clip/blob/d0befe114486a51b109bfc307fa3fbcaa3283b8b/src/open_clip/tokenizer.py#L163 SigLipTokenizer was used to import sentencepiece vocab files into HF T5 tokenizer, not currently used by the models but would be needed for another import...

possibly a corrupt checkpoint? although not at all clear what's being done here? loading in c++ via libtorch?

@nikifori @vkrishnamurthy11 there is already https://github.com/mlfoundations/open_clip/pull/523 ... it is looking reasonable but I have not found the time to sit down and test carefully, if anyone else is willing to...

So, supporting mps and other non cuda/cpu devices worthwhile goal, not sure 'this' is the best approach though. For autocast, should we rely on the amp (precision) arg to determine...

hmm, I would have thought this works as long as you don't lock the full image or text towers... but perhaps not, it may not be good idea to checkpoint...

Re the remove MHA from attn pooler, I don't recall what the motivation for that was? It doesn't lool like it's doing anything that isn't supported by MHA at this...

@royg27 guess that one fell through the cracks re getting it online, the performance of those weights is pretty unspectacular vs the laion2B ones, do you need it for any...

@rsomani95 thanks for the contribution For this to work in all current cases, the bulk of the logic needs to be in a free function because this should work with...

@ErwannMillon thanks for the PR, most of the meta device code/logic is in torch itself, I believe (correct me if I'm wrong) the lines of code to implement the rest...

@ErwannMillon k, it can definitely be useful, especially as the models get larger. Aside from test failing, there are a few things I want to verify and could probably clean...