Evaluation on Town-05 Long benchmark

[dingli-dean]

Hi, team. Thanks for releasing the exceptional work. I try to evaluate the released model (llava-v1.5) on town05 long benchmark (with leaderboard/data/evaluation_routes/routes_town05_long.xml), and sadly observe that the results are significantly lower than current SOTA methods.

Did you evaluate LMDrive model on town-05 long benchmark? If so, can you show the performance comparison between LMDrive and other methods?

Thanks again for your attention, and look forward to your reply.

[deepcs233]

Hi!

We haven't tested LMDrive on the other benchmarks. Our framework only tasks as input the instruction based on natural language. It's harder compared with other frameworks that can directly obtain the target point/ command. Hence, there's no sense in comparing with the traditional methods on other benchmarks.