waymo-open-dataset icon indicating copy to clipboard operation
waymo-open-dataset copied to clipboard

Sim agent challenge V1 metric testing score

Open Ariostgx opened this issue 8 months ago • 2 comments

Hi there! I am currently working on a project using Sim agent challenges.

To form a more complete comparison with prior works, I would like to compare my method with 2023 challenge methods (e.g., Trajeglish). However, as the submission for the 2023 challenge is closed and the metric for 2024 challenge changed, I could not get a testing score to compare with the methods of the 2023 challenge.

Is there a chance that I can get V1 metrics for my current testing submission to 2024 challenge?

On the other hand, if that's not possible, how far would the validation score for V1 metrics (which I can compute offline) be from the testing scores? (If they are not too far away, I can have a rough comparison using validation scores for 2023 challenge methods).

Thank you so much for your help in advance!

Ariostgx avatar May 29 '24 18:05 Ariostgx