EvalAI icon indicating copy to clipboard operation
EvalAI copied to clipboard

Submission status stays at "submitted" and no results show up on the leaderboard

Open MicroSTM opened this issue 2 years ago • 0 comments

We are hosting a challenge here: https://eval.ai/web/challenges/challenge-page/1788/overview It is a prediction-based challenge (submitting a single result file) and it has been approved. However, after submitting results using 2 different user accounts, we found that the submissions did not generate any result entry on the leaderboard. Their status stayed at "submitted," and it did not seem like any scores were generated.

Here is the worker log:

[2022-08-25 16:51:18] INFO WORKER_LOG Using /tmp/tmp_t7n3iky as temp directory to store data
[2022-08-25 16:51:18] INFO No custom requirements for challenge 1788
[2022-08-25 16:51:38] INFO WORKER_LOG Processing message body: {"challenge_pk": "1788", "phase_pk": "3492", "is_static_dataset_code_upload_submission": false, "submission_pk": 219300}
[2022-08-25 16:51:38] INFO SUBMISSION_LOG [x] Received submission message {"challenge_pk": "1788", "phase_pk": "3492", "is_static_dataset_code_upload_submission": false, "submission_pk": 219300}

Here is the screenshot of the submissions. (I reran one of the submissions so there was a canceled entry) Screen Shot 2022-08-25 at 12 59 33 PM

I have written and tested the evaluation script. Are there missing steps from my end? I understand that I do not need to run any submission worker on my own server if the submissions are not code-based. Is that right?

MicroSTM avatar Aug 25 '22 17:08 MicroSTM