Error in nuscenes test data

[Morizb]

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py $ckpt_path$ 2 --eval bbox The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

When I modify the read test set directly and run it again, the following two errors are reported: KeyError: Caught KeyError in DataLoader worker process 0. KeyError: 'num_lidar_pts'

I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

[Morizb]

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py ckptpath 2 --eval bbox The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

When I modify the read test set directly and run it again, the following two errors are reported: KeyError: Caught KeyError in DataLoader worker process 0. KeyError: 'num_lidar_pts'

I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

My problem seems to be in line with this one: https://github.com/SxJyJay/MSMDFusion/issues/9#issuecomment-1519281181 The error reported is: FileNotFoundError: [Errno 2] No such file or directory: '...... /MSMDFusion/data/nuscenes/samples/FOREGROUND_MIXED_6NN_WITH_DEPTH/n015-2018-07-25-16-15-50+0800__LIDAR_TOP__1532506680948053.pcd. bin.pkl.npy'

[SxJyJay]

However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

Generally, results submitted to the leaderboard to get test set predictions are trained on train+val sets, therefore the results on the test set are higher than val set.

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py ckptpath 2 --eval bbox The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set. When I modify the read test set directly and run it again, the following two errors are reported: KeyError: Caught KeyError in DataLoader worker process 0. KeyError: 'num_lidar_pts' I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

My problem seems to be in line with this one: #9 (comment) The error reported is: FileNotFoundError: [Errno 2] No such file or directory: '...... /MSMDFusion/data/nuscenes/samples/FOREGROUND_MIXED_6NN_WITH_DEPTH/n015-2018-07-25-16-15-50+0800__LIDAR_TOP__1532506680948053.pcd. bin.pkl.npy'

It seems that we don't successfully upload the test set virtual points in the given link. I will check them.

[Morizb]

Thank you very much for your fast response!

[sun-yue2002]

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py ckptpath 2 --eval bbox The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

When I modify the read test set directly and run it again, the following two errors are reported: KeyError: Caught KeyError in DataLoader worker process 0. KeyError: 'num_lidar_pts'

I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

Hi! I met the same error. Have you find out the reason of this issue and resolution on how to deal with it?

[SxJyJay]

https://github.com/SxJyJay/MDU_preprocess

You can refer to this repo and generate the required virtual points for nuScenes test set using the very first command

python virtual_depth_mapping_multi_proj.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl  MODEL.WEIGHTS centernet2_checkpoint.pth

[sun-yue2002]

https://github.com/SxJyJay/MDU_preprocess

You can refer to this repo and generate the required virtual points for nuScenes test set using the very first command

python virtual_depth_mapping_multi_proj.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl  MODEL.WEIGHTS centernet2_checkpoint.pth

Thanks a lot!!!!

[Morizb]

@SxJyJay Hi, I am testing on the test set （nuscenes_infos_test.pkl ）and have followed the above code to generate virtual points on the test set, which solves the previous problem. But it reports a new error: AssertionError: Error: You are trying to evaluate on the test set but you do not have the annotations!

The command I run when testing is: ./tools/dist_test.sh sh ./tools/dist_test.sh ./configs/MSMDFusion_nusc_voxel_LC.py /work_dirs/20240429/epoch_6.pth 2 --eval bbox --cfg-options'test_evaluator.jsonfile_prefix=/work_dirs/20240429/results_eval'

Do you know what it is, please? How can I test it on a test set?

[Libraaer]

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py c k p t p a t h 2 --eval bbox The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

When I modify the read test set directly and run it again, the following two errors are reported: KeyError: Caught KeyError in DataLoader worker process 0. KeyError: 'num_lidar_pts'

I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

Hello, have you trained this model? I'm using A800 here but still not enough to train. If you've trained, can you give me an answer to the reason for my lack of memory here, thank you

[Klord02]

@Morizb @Libraaer Hi ! I am trying to run the code for MSMD fusion and perform training on a relatively small part of the whole dataset. I would really appreciate if you could answer the following doubts that I have

1. @SxJyJay referenced https://github.com/SxJyJay/MDU_preprocess to create the preprocessed virtual points and the mentioned repo further contains a reference to https://github.com/tianweiy/MVP to set up. Do we have to carry out the Installation part as well or only the Getting Started part. I am asking this as I am working with limited memory and can't afford to carry out redundant steps.

2. Were you able to find the solution to the error AssertionError: Error: You are trying to evaluate on the test set but you do not have the annotations!

[SxJyJay]

@Morizb @Libraaer Hi ! I am trying to run the code for MSMD fusion and perform training on a relatively small part of the whole dataset. I would really appreciate if you could answer the following doubts that I have

1. @SxJyJay referenced https://github.com/SxJyJay/MDU_preprocess to create the preprocessed virtual points and the mentioned repo further contains a reference to https://github.com/tianweiy/MVP to set up. Do we have to carry out the Installation part as well or only the Getting Started part. I am asking this as I am working with limited memory and can't afford to carry out redundant steps.
2. Were you able to find the solution to the error AssertionError: Error: You are trying to evaluate on the test set but you do not have the annotations!

1. You are right. Installation of MVP environment is required for generating multi-depth seeds.

2. Nuscenes only provide validation annotations, because test annotations are private for the official leaderboard evaluation.

[Klord02]

@SxJyJay Thanks for the prompt reply.

As far as the 2nd point is concerned, it won't be a problem while carrying out the training part and dealing with the training data, right ?

Also, this means that the the links that you had provided in Step 2 of Getting Started themselves were only of the training data. Is my inference correct ?