waymo-open-dataset icon indicating copy to clipboard operation
waymo-open-dataset copied to clipboard

Evaluation on server not working

Open dsx0511 opened this issue 2 years ago • 1 comments

Hi,

I submitted my results for the validation set of the occupancy flow prediction challenge on the website two days ago. However, it seems that the evaluation is not working, and I still get the following error message: image

From my experience before the challenge deadline, this message shows when the evaluation is running on the server. Once it finishes (after about 10 min), we should see the result. Is there any problem with the evaluation server, or is it shut down after the challenge is finished?

dsx0511 avatar Jul 15 '22 09:07 dsx0511

Hi, I still can't get the evaluation result on the evaluation server. According to the homepage, the leaderboard is still open for submission: image. I also see some new submissions on other tasks after the end of the challenges. Could you please fix the problem for the occupancy and flow prediction task? We want to use the new evaluation result in a potential publication.

Best regards

dsx0511 avatar Aug 19 '22 09:08 dsx0511

Have the same issue with my submissions earlier this week: image Is there any plan to fix the problem? Thanks in advance.

jnan1 avatar Dec 15 '22 15:12 jnan1

Bumping this, my submisions to occupancy flow aren't getting processed either. I accidentally submitted one to Motion Prediction and recieved quick feedback that it was in the wrong format.

5had3z avatar Jan 27 '23 00:01 5had3z

The metrics server issue should be fixed. Please let us know if you are having any issues.

scott-ettinger avatar Jan 30 '23 19:01 scott-ettinger