waymo-open-dataset icon indicating copy to clipboard operation
waymo-open-dataset copied to clipboard

Published 20 hours ago • verified publisher icon waymo-research

Is the sim agents benchmark still running?

Open jonahthelion opened this issue 1 year ago • 4 comments

Hello,

I made a submission to the validation eval for the sim agents benchmark about 2 days ago and it hasn't finished evaluating yet. I wanted to check if the evaluation server is down?

I see this message when I view the submission: Screen Shot 2023-11-11 at 7 13 14 PM

Thank you!

jonahthelion avatar Nov 12 '23 00:11 jonahthelion

Hi,

thanks for reaching out. Our evaluation server has some stability issues, I've just triggered a reprocessing of your submission. The results should be ready within ~1 day. Apologize for the inconvenience.

alexgorban avatar Nov 13 '23 17:11 alexgorban

Thank you very much for the prompt response!

I've just uploaded a new submission to the sim agents test server. I was wondering if it would be possible to verify that the evaluation is running on that submission now? I'm guessing it is, but I want to be absolutely sure there isn't a problem given there are conference deadlines coming very soon.

Thank you!

Update: I see it has already finished evaluating! We're all good now.

jonahthelion avatar Nov 15 '23 18:11 jonahthelion

Hello,

I made a submission to the sim agents test server 12 hours ago but it still says it's processing. Is the server down again?

jonahthelion avatar Jan 11 '24 15:01 jonahthelion

Hello, I submitted to the sim agents test server 26 hours ago but it says it's still processing (I was struggling to upload results and ended up submitting 2 that are the same by accident, sorry about that). I wanted to check if the server is down?

jonahthelion avatar Mar 11 '24 22:03 jonahthelion