A considerable gap from the SOTA

[Brave-peng]

trafficstars

I am trying to reimplement the OTAM and using the default arguments, this is options and the result in ssv2。

Options: Namespace( dataset='data/ssv2_256x256q5', learning_rate=0.001, tasks_per_batch=16, checkpoint_dir='checkpoint_dir', test_model_name='checkpoint_best_val.pt', training_iterations=60000, resume_from_checkpoint=False, way=5, shot=5, query_per_class=5, query_per_class_test=1, val_iters=[], num_val_tasks=1000, num_test_tasks=10000, print_freq=100, seq_len=8, num_workers=8, backbone='resnet50', opt='sgd', save_freq=5000, img_size=224, num_gpus=4, sch=[1000000], method='otam', pretrained_backbone=None, val_on_test=False, trans_linear_in_dim=2048)

test accuracies: data/ssv2_256x256q5: 44.5+/-0.4

This is a considerable gap from the SOTA. Instead, I also testing in the kinetics and using the TRX, only get the 63.2% accuracy(5-way-1-shot). Could you tell me what might be causing this?

[tobyperrett]

It's been a while since I've used this, but I seem to remember that OTAM works best with tasks_per_batch=1 (they don't mention gradient accumulation in their paper). Others who've used this code to replicate the OTAM results used that and it seemed to work OK. Hope that helps.

[tobyperrett]

In fact, that's what the setting is in scripts/examples.md. So just try running that command.

[Brave-peng]

Thank you very much for your quick reply, which is very helpful to me