nuScenes Results

[ZwwWayne]

Hi there,

This code repo is a graet work. But I have some question about the nuScenes results.

In your current newest version of code, we prepare the data and train a model using the config you provided, and we found that the AP is much lower than that in KITTI. Do you meet the same situation? If so, is this because there are some small bugs of data processing or just because the dataset is much harder?

Thanks

[yzheng97]

I'm also working on this dataset. Do you get the results on nuScene validation split? Are there bugs coming up for the operations below rather than the provided trainval&test split?(Of course after modifying the nuscenes_data_prep() function)

python create_data.py nuscenes_data_prep --data_path=NUSCENES_TRAIN_DATASET_ROOT --version="v1.0-train" --max_sweeps=10 python create_data.py nuscenes_data_prep --data_path=NUSCENES_TRAIN_DATASET_ROOT --version="v1.0-val" --max_sweeps=10

[yzheng97]

And from my point of view, low performance is pretty normal for a new challenging dataset. You could refer to the paper https://arxiv.org/pdf/1903.11027.pdf.

[traveller59]

The nuscenes dataset is much harder than KITTI. you should focus on nusc dist AP rather than kitti AP to match nuscenes paper result. @yzheng97 supported version is "v1.0-trainval", "v1.0-mini" and "v1.0-test". "v1.0-val" and "v1.0-train" is invalid.

[traveller59]

To understand why nuscenes is hard: In this picture, all cars will be count as false negatives (their bounding boxes is not empty) if your detector can't detect them but it is obviously impossible to detect them.

[yzheng97]

The nuscenes dataset is much harder than KITTI. you should focus on nusc dist AP rather than kitti AP to match nuscenes paper result. @yzheng97 supported version is "v1.0-trainval", "v1.0-mini" and "v1.0-test". "v1.0-val" and "v1.0-train" is invalid.

Thanks for your reply.

[ghimiredhikura]

@traveller59 ,

Do you have results in nuscenes dataset in numbers. I was wondering how much mAP you got till now.

[traveller59]

@ghimiredhikura car Nusc dist [email protected], 1.0, 2.0, 4.0 57.95, 65.08, 68.03, 71.30

worse than baseline in nusc paper: 76% dist AP (thresh=2.0)

[ZwwWayne]

@traveller59 Thanks for your reply, it helps a lot

[traveller59]

@WERush no, I am working on car-only network. some important parameters: detection range (especially z range), nms parameters. The major problem of nuscenes AP is too much false negatives. increase det range and modify nms param can decrease false negatives and greatly increase AP performance.

[yzheng97]

@traveller59 Could you please update the result of all.pp.config: 50 epoch, 1/8 dataset ? The result in README.md misses the barrier category. The result in NUSCENES-GUIDE.md is the same with all.pp.lowa.config.

[yinjunbo]

@traveller59 Is all.pp.config released in the latest git version? All I get in nuscenes sub folder are all.pp.mida.config, all.pp.mhead.config, all.pp.lowa.config, all.pp.largea.config, all.pp.deprecated.config, all.fhd.config. Besides, I'd like to know which config gives the best results on nuscenes. Thanks !

[yinjunbo]

@ghimiredhikura car Nusc dist [email protected], 1.0, 2.0, 4.0 57.95, 65.08, 68.03, 71.30

worse than baseline in nusc paper: 76% dist AP (thresh=2.0)

Does the results obtain from the model trained with the full dataset?

[yinjunbo]

@WERush Hello, are you still working on nuScenes? Do you obtain a better result with the latest config files? Thanks

[caroltk]

To understand why nuscenes is hard: In this picture, all cars will be count as false negatives (their bounding boxes is not empty) if your detector can't detect them but it is obviously impossible to detect them.

Hello @traveller59 , thanks for your great work! I am wondering how did you generate this visualize result (window)?