DeepGOPlus code vs DeepGOWeb server inconsistencies

[InesGP]

I've been comparing the results generated from the code in this repository and the results generated from the DeepGoWeb server and I noticed some inconsistencies in regards to the GO classes and scores computed. I believe I am getting these inconsistencies due to an incorrect alpha value. I'm including a Jupyter notebook detailing a few test runs with different alpha values and their Jaccard similarity score as well as the Dockerfile I used to create the environment to run the DeepGOPlus code. I also attached below a table of some alpha values I tried out and was wondering if you could provide the ones you use in the DeepGoWeb server for version 1.0.1. I also created a PR request with a potential fix for the alpha values elaborating on the issue and the code I changed.

Dockerfile.txt DeepGOPlus Alpha Experimentation.pdf

@glatard

[coolmaksat]

Hi, I think you are comparing different models here. Version 1.0.1 and 1.0.5 use different versions of UniprotKB to train and evaluate the models and they also have different alphas

[InesGP]

Thanks for the reply, I had tested data version 1.0.1 as recommended, but still encountered the same issue but with decreased Jaccard similarity scores between the code and server results. Could there be differences in the code between versions?

[Soymethylation]

Hi! I have also encountered the same problem. I used version 1.0.13 in DeepGOWeb and the GO dataset of data-1.0.13.tar.gz in DeepGOPlus, and found that the results of the two were almost not completely similar. At the same threshold, the results of DeepGOPlus are generally lower than those of DeepGOWeb. I would like to know if there have been changes to the web version of the algorithm, or if there are any other reasons. If someone already knows the reason, I would be happy to know and would greatly appreciate the answer.

[z4668640]

I have the same problem now. The same data version runs with the same data on the Web version and locally, and the results are almost completely inconsistent. Can someone tell me why? Best wish.

[ferzcam]

Thank you for noticing this issue. There was indeed an issue regarding the alpha values and also a discrepancy between experimental and propagated annotations. We have updated the DeepGOWeb server and release a new DeepGOPlus package (v.1.0.2).