Motion forecasting: Focal agent not always observered over the full scenario length

[SchDevel]

Hey everyone,

I had a look into the motion forecasting dataset and there seems to be an issue with the trajectories of the focal agent. According to the paper, the focal agent should always be observed over the full 11 seconds, which then corresponds to 110 observations: "Within each scenario, we mark a single track as the “focal agent". Focal tracks are guaranteed to be fully observed throughout the duration of the scenario and have been specifically selected to maximize interesting interactions with map features and other nearby actors (see Section 3.3.2)"

However, this is not the case for some scenarios (~3% of the scenarios). One example: Scenario '0215552f-6951-47e5-8cf6-3d1351d28957' of the validation set has a trajectory with only 104 observations.

Can you reproduce my problem? Is this intended or can we expect this to be fixed in the near future?

Looking forward hearing from you!

Best regards

SchDevel

[ZikangZhou]

I also found that the ground-truth future of the focal agent is not always complete. This causes some problems in the calculation of metrics. I think those invalid time steps should be masked when calculating ADE. On the other hand, sometimes the final time step is invalid. Perhaps we should consider the last visible time step as the "final time step" when calculating FDE and MissRate?

[wqi]

Hi @SchDevel and @ZikangZhou, thanks for your comments on this issue!

There are indeed a small minority of scenarios that are missing object states for the focal agent; this is unintentional.We are aware of the problem and are actively working to release a patched version of the dataset.

We'll likely have it out for release by next week. If you are building submissions for the challenge, please continue to generate predictions for all test set scenarios - assuming a 60 timestep prediction horizon.

[narsisn]

Hi @ZikangZhou, thank you for your comment. could you give me more details about how you recognized the invalid final time steps?

best regards

[wqi]

Hey everyone, thanks for your patience with this dataset issue.

The Argoverse 2 Motion Forecasting dataset was bumped to v2.0.1 a few weeks back and is available to download directly from S3 or via the download links on the Argoverse website. All focal agents and scored actors are now guaranteed to have a full 110 observations.

The new data is backwards compatible with all previous tooling designed for data from our initial release.

The AV2 MF challenge has also been updated to reflect the latest version of the dataset.

[MasterIzumi]

Hi @wqi, I find that the number of scenarios in each split (for the current version of dataset) is not the same as described in the paper: test 24984 | train 199908 | val 24988 Is this because of the fix of this issue (since I find no missing state in the focal track anymore)? Will that influence the evaluation of the test split? Thanks!

[wqi]

The updated number of scenarios won't cause any issues with the test set evaluation on EvalAI. The evaluation server and computed results have been adjusted on the backend to account for the problematic scenarios which were removed.

[MasterIzumi]

The updated number of scenarios won't cause any issues with the test set evaluation on EvalAI. The evaluation server and computed results have been adjusted on the backend to account for the problematic scenarios which were removed.

I get it. Thank you so much!