代码中query的顺序没有打乱，打乱后准确率显著下降。The order of the query in the code is not shuffled, and the accuracy dropped significantly after being shuffled.

[EternalWang]

作者您好，我和朋友在运行这份代码的时候发现这样一个问题： query的输入顺序是和support的顺序固定一致的，此时训练后的模型在验证集上准确率能有85.684%，比较高。然而在把query的顺序打乱的情况下训练模型，只能在验证集上得到77.042%的准确率。公认的是，query的顺序应该被打乱。我们认为是您的图神经网络出现了过拟合，学习到了query的固定顺序。您能对此作出解释吗？感谢。（上述准确率均是在一块2080ti GPU上训练得到的结果，由于显存的原因，我将train_batch_size和test_batch_size改小到了32和16） Hello author, friends and I found this problem when running this code: The input order of query is consistent with the order of support. At this time, the accuracy of the trained model on the validation set can be 85.684%, which is relatively high. However, when the query order is shuffled, the model can only obtain 77.042% accuracy on the validation set. It is generally accepted that the order of queries should be recognized. We think that your BGNN has overfitted and learned a fixed order of queries. Can you explain this? Thanks. (The above accuracy rates are obtained by training on a 2080ti GPU. Due to limited GPU memory, I changed the train_batch_size and test_batch_size to 32 and 16)

[Luoyadan]

Hi EternalWang,

I just checked the data loader and there is something wrong with the query order. I directly use the same script in https://github.com/khy0809/fewshot-egnn and did not shuffle it... I will fix the error and update the source code and the results in arxiv soon. Thanks for pointing out.

Cheers, Yadan

[Luoyadan]

Hi EternalWang,

I re-ran the experiments and I found the results do not have a substantial change. But it did take a longer time for convergence. You may need to set a larger batch size to reach the same performance.

Cheers, Yadan

[EternalWang]

Hi Luoyadan,

I also think that batch size will affect accuracy, but its impact is acceptable, relative to whether the order of the query is shuffled. Could you please update the source code?

Cheers, EternalWang

[Luoyadan]

Hi all,

I will update the source code by the next week, since I am currently staying in hospital without my laptop. Thanks for your understanding.

Cheers, Yadan

Tang Hao [email protected]于2020年2月23日 周日下午2:12写道：

Hi EternalWang,

I re-ran the experiments and I found the results do not have a substantial change. But it did take a longer time for convergence. You may need to set a larger batch size to reach the same performance.

Cheers, Yadan

Hi, @Luoyadan https://github.com/Luoyadan : I also found the problem mentioned by @EternalWang https://github.com/EternalWang in the code. When I re-implemented the code, I found that the accuracy of your project was greatly reduced. I hope you can face up to this problem and update the code in time. Looking forward to your reply

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Luoyadan/BGNN-AAAI/issues/1?email_source=notifications&email_token=AEB3WPGH22XQRZEG4FDGPKLREHZS7A5CNFSM4KWJXFJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVQ7YY#issuecomment-590024675, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEB3WPBOT42YBCCYWEY4OBLREHZS7ANCNFSM4KWJXFJA .

--

Yadan Luo

School of Information Technology and Electrical Engineering

University of Queensland

Brisbane, Australia

[Luoyadan]

Dear all,

The query shuffle is enabled in the updated version. The CML-BGNN-3 can achieve 87.05% on the 5-way 1-shot classification task of the tiered-imagenet dataset, which is 1% lower than the results we reported.

Cheers, Yadan

[Duan-JM]

Dear all,

The query shuffle is enabled in the updated version. The CML-BGNN-3 can achieve 87.05% on the 5-way 1-shot classification task of the tiered-imagenet dataset, which is 1% lower than the results we reported.

Cheers, Yadan

Thank you for updating the code. However, this shuffle setting only shuffles the sequence inside the class, without shuffle the whole query sets. eg. For only one query scenario, the query in the first position always belongs to the first class.

Could we shuffle the query like below:

# before shuffle 
# query label
[0, 1, 2, 3, 4]

# after shuffle
# query label
[4, 2, 1, 3 0] # specifically random for each task