Some issues with the implementation of GANA

[RapidsLNT]

Dear Authors:

There are some issues with the implementation of GANA: Non-reproducible results, In-Train setting rather than Pre-Train setting, and inconsistencies between paper and code.

1. Reproducibility The results are not reproducible. Dependencies such as PyTorch version, CUDA version, and versions of the other Python libraries are not provided. Also, if we directly use the environment provided in MetaR, we get the following results on NELL-One (1-shot):

	MRR	Hit@10	Hit@5	Hit@1
GANA (Paper)	0.307	0.483	0.409	0.211
GANA (Reproduce)	0.265	0.453	0.357	0.163
MetaR (GitHub)	0.308	0.475	0.406	0.216

We tuned GANA using the following parameters: max_neighbor: 50, 100, 200 lstm_hiddendim: 100, 200, 600, 700 The results are much lower than the reported results in MetaR.

2. Pre-Train Setting In the paper, it is mentioned that GANA utilizes pretrained embeddings from GMatching. However, in main.py, tail is set to _in_train, which is different from the paper's descriptions. It seems that the implementation is based on the "In-Train" setting, not the "Pre-Train" setting.

3. Inconsistencies between the Paper and the Implementation

• In the paper, the hyperplane vector is updated using the gradient of the hyperplane vector. However, in the code, the hyperplane vector is updated using the gradient of the relation embedding. (In model_gana.py, line 239)
• The paper describes GAT, but the implementation is GATv2. (In model_gana.py, lines 172-174)

Hope to get a reply. Thank you.

Best Regards, RapidsLNT