Ross Wightman

And also having the two cpe modules in a ModuleList instead of cpe1/cpe2 instantiated relative to the other modules in they order they are used in fwd would be preferred...

@fffffgggg54 changes looking good, a few minor formatting things I'll take care of and will do some testing, I'll likely combine it with polishing up FocalNet and do in same...

@fffffgggg54 a number of the changes are just small style nits that I often do for consistency... I can point out a few examples. For the downsample, I think I'd...

@fffffgggg54 thanks again for PR, I've been fiddling with the models and I'm falling short of the official classification results trying to validate, I've run through a few different crop...

@fffffgggg54 thanks, merged, I did a bit of refactoring, I didn't really feel their 'patch embed' was a patch embed anymore given it was being used like a stride 4...

@yazdanbakhsh loading jax .npz checkpoints is integrated into the model for original Google jax implementations (big vision support being merged today) * https://github.com/google-research/vision_transformer * https://github.com/google-research/big_vision However, I don't have support...

@yazdanbakhsh just dumped some hparams, but as usual, they need adapting to your scenario / specific network https://gist.github.com/rwightman/943c0fe59293b44024bbd2d5d23e6303

@lorenzbaraldi MIM (BEiT and MAE style) support is on the todo list but requires some careful experimentation, no sure what the timeline is...

@hellojialee with the new multi-weight support it's easy to add the old weights back, but I'm not sure there was ever tiny w/o 21k pretrain in timm. I don't think...

There are definitely some augreg (how to train your vit) 1k tiny weights but I never them in any timm release. EDIT yes, the original ViT release in JAX never...