Questions on Carla Evaluation

[byaman14]

Hi, I have few questions regarding Carla results.

1. For the paper, have you trained VAD on Carla and then reported the evaluation results or used your open loop checkpoint (pretrained on nuscenes) for evaluation.
2. Can you please share your closed loop evaluation script (such as carla_agent.py as in stp3) so we can reproduce the closed loop results?
3. Can we directly use an open loop checkpoint in your evaluation script to see closed loop evaluation results?

I would appreciate your quick response on this matter.

Best, Burhan

[rb93dett]

Hi Burhan,

Sorry for the late reply.

1. We trained VAD on Carla and then reported the evaluation results.
2. The closed-loop scripts will be open-sourced along with VADv2 as soon as possible.
3. We haven't tried this, but I think the results won't be satisfactory due to the domain gap.

If you have any other questions, please feel free to discuss them with us.

[jiaxiaosong1002]

@byaman14 Hi, we are the authors of Bench2Drive and we would like to introduce that we have implemented VAD (non-official) in CARLA with code, checkpoints, and closed-loop results open sourced. Feel free to try it. https://github.com/Thinklab-SJTU/Bench2DriveZoo/tree/uniad/vad

[byaman14]

Hi @jiaxiaosong1002, sounds great, thanks a lot!!

[xiaodongww]

@jiaxiaosong1002 ,hi, do you reproduce the results successfully in carla? It seems that your reported driving score is much lower that the score reported in the paper of VADv2.