can you provide your pre-trained weight?

[NEUdeep]

Hello, author, your vit code is very concise and excellent. I found that the pre-training weight provided by "timm" is very inconsistent with your code. The key and value are not in line with your code, can you provide your pre-trained weight? Thank you

[robinzhaorr]

Hi, NEUdeep. I am also trying to implement pre-training weight in my model, have you found a solution? Thanks in advance.

[forever208]

this pre-trained feature gonna save lives...

[linYDTHU]

Hi, I also need the pre-trained feature on some famous datasets! Thanks!

[NEUdeep]

Hi, NEUdeep. I am also trying to implement pre-training weight in my model, have you found a solution? Thanks in advance.

[NEUdeep]

Hi, NEUdeep. I am also trying to implement pre-training weight in my model, have you found a solution? Thanks in advance. sorry ,not yet

[vietvo89]

Hello guys

I have an issue with pre-trained model from this repo. I am trying to follow Cat and Dog notebook to load "pre-trained" model as below

import torch
from vit_pytorch.efficient import ViT
from linformer import Linformer

efficient_transformer = Linformer(
    dim=128,
    seq_len=49+1,  # 7x7 patches + 1 cls-token
    depth=12,
    heads=8,
    k=64
)

# load model
model = ViT(
    dim=128,
    image_size=224,
    patch_size=32,
    num_classes=2,
    transformer=efficient_transformer,
    channels=3,
).cuda()
model.eval()
...
testiter = iter(testloader)
x_test, y_test = testiter.next()
output = model(x_test.cuda())

Data I used for this is ImageNet2012, size = 224x224 But it seems it does not work and totally fails to classify. Do you have any idea?

Thanks

[vietvo89]

Hello guys

I also follow another code in readme to load "pre-trained" model as below

import torch
from vit_pytorch import ViT

v = ViT(
    image_size = 256,
    patch_size = 32,
    num_classes = 1000,
    dim = 1024,
    depth = 6,
    heads = 16,
    mlp_dim = 2048,
    dropout = 0.1,
    emb_dropout = 0.1
).cuda()
model.eval()
...
testiter = iter(testloader)
x_test, y_test = testiter.next()
output = model(x_test.cuda())

Data I used for this is ImageNet2012, size = 224x224 But it totally misclassified all too.

Thanks

[khawar-islam]

@forever208 Is it possible? If we use timm pre-trained weights in this code. The problem is that training takes alot of time.

[abhigoku10]

@khawar512 can you pls share the code to load it from timm it will be helpful !!!

[khawar-islam]

@abhigoku10 Yes, y not https://github.com/rwightman/pytorch-image-models

[DegardinBruno]

@forever208 Is it possible? If we use timm pre-trained weights in this code. The problem is that training takes alot of time.

Could you use those pre-trained weights with this implementation of lucidrains?

[liyiersan]

@DegardinBruno I hope this works for you. I provide an example of loading a pre-trained vit-base. Please refer to https://github.com/liyiersan/MSA/blob/22243186133369941bb78bbd93e6e2cd04317f66/models/vit.py#L133-L211. or you can refer to https://github.com/Sebastian-X/vit-pytorch-with-pretrained-weights.