Trouble replicating results on Roaming-Imagenet for pretrained models

[sebamenabar]

Hi!

I'm not able to replicate the performance for the pretrained online-protonet on the Roaming-Imagenet dataset. With the provided checkpoint I get the reported metrics, but if I pretrain the model I'm getting ~21AP on testing and ~22AP on validation.

To pretrain I use the following command:

python -m fewshot.experiments.pretrain --config configs/models/roaming-imagenet/pretrain.prototxt --env configs/environ/roaming-imagenet-docker.prototxt --seed 0 --tag pretrained

On the pretrain.py file I modified this line: dataset = get_data(env_config) and put this instead: dataset = get_data_fs(env_config, load_train=True), because the method get_data does not exist on the fewshot.experiments.utils file.

Finally, for finetunning/evaluating the pretrained model I run: python -m fewshot.experiments.oc_fewshot --config configs/models/roaming-imagenet/online-protonet.prototxt --data configs/episodes/roaming-imagenet/roaming-imagenet-150.prototxt --env configs/environ/roaming-imagenet-docker.prototxt --tag the_tag --pretrain results/oc-fewshot/tiered-imagenet/pretrained_model/weights-40000

I hope that you can helps me with this and thanks for the code!

[sebamenabar]

any chance you had the time to look at this @renmengye ?

[renmengye]

Could you try passing the pretrain folder path instead of the checkpoint path?

On Thu, Oct 28, 2021 at 10:54 Sebastian Amenabar @.***> wrote:

any chance you had the time to look at this @renmengye https://github.com/renmengye ?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/renmengye/oc-fewshot-public/issues/1#issuecomment-953925664, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIVUFLNIFR362NBQSWIG4LUJFPZFANCNFSM5GAGGYFQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Sent from my iPhone