nuPlan paper table comparison

[perone]

Reposting here as suggested in (https://forum.nuscenes.org/t/nuplan-paper-comparison-table/797):

Hello ! I was reading the nuPlan paper (https://arxiv.org/abs/2106.11810) where the Table 1 (shown below):

Is showing that the Lyft dataset doesn't have Closed-loop evaluation and that it has a "prediction" in the Type column.

As a suggestion, I would review the table because in L5kit there is a closed-loop evaluation with many closed-loop metrics available since 2021 (before the nuPlan paper publication). Not only there is a simulator but also a RL environment where rollouts can be done and use rewards from closed-loop metrics as well. There is also an entire section about Planning where there was even a paper released using it for training a planner in closed-loop. Anyway, the table doesn't seem to reflect the current state of the framework.

[patk-motional]

Hi @perone,

Sorry for the super late reply. Thank you for pointing this out. We are in the process of reviewing the paper to make it more accurate

[perone]

No problem, thanks a lot for the reply @patk-motional.

[perone]

Hi @patk-motional, can we reopen this issue ? It has been more than 1 year that the paper doesn't reflect the correct information about the framework and other datasets, that doesn't seem a fair comparison.

[patk-motional]

Hi @perone,

Sorry for the delay, does something like this suffice?

[perone]

Thanks for the quick reply @patk-motional, this sounds fine. I think the type can also be Pred+Plan (also for nuPlan no ?) because it can be used for agent prediction tasks as well, but up to you.

[nightrome]

After looking into this we have decided to update the paper to state that your dataset provides a closed-loop planning tutorial, but no official benchmark, which is really the emphasize of nuPlan.