Questions about the test set

[luoxindi]

Hello, I would like to know how to get the correct answers of the questions in test set and their corresponding sparql statements? Do I only need to upload my model to the website you provided to get the results each time when I change the model? In other words, how do I get this test set with complete information?

[entslscheia]

Hi Xindi,

We didn't release the labels for test set on purpose for fair comparison (just like many other QA benchmarks did). To evaluate your model on the test set, you can follow the official instructions here. It's actually quite simple; you only need to submit the predictions rather than the model per se. To debug your model locally, we strongly recommend you using the dev set of GrailQA, which shares a similar distribution with the test set in terms of generalization levels.

Feel free to let me know if you have any further questions!

Best, Yu

[luoxindi]

I got it, thanks

[lgxccc]

May I ask why this website is not accessible now. I was able to access this website a few months ago. May I ask if you can log in to this website now?

[entslscheia]

@lgxccc Looks like Codalab was down days ago but it's back now. You can also directly send your predictions to my email if you want