About data leakage on zero-shot classification?

[LTEnjoy]

Hello!

Thanks for your great work! I have tested the zero-shot classification given your released checkpoint and it did a good performance. But I am confused that whether there exists some data leakage problem? Your model was fine-tuned on Swiss-Prot database and the DeepLoc dataset was also constructed from UniProt database. Did you do some filtering when you tested zero-shot performance?

Looking forward to your reply! Thanks in advance!

[KatarinaYuan]

Hi, Thank you being interested in our work!

Please see the pre-training dataset https://github.com/DeepGraphLearning/ProtST/blob/db53a76ed2430eb66dd9c8134ace99fd60980fb3/protst/dataset.py#L22. It does not expose test labeled data of each benchmark dataset that has not been observed during multimodal pre-training nor downstream fine-tuning.

[LTEnjoy]

Hi,

Thank you for the reply and I'll check it out!