Same validation and test configuration

[holtvogt]

Why is the test configuration in, e.g. kitti-3d-car and kitti-3d-3class using the same pickle file, split and pipeline as the validation configuration? Shouldn't the test configuration be using the test pickle file and test split?

• Validation in kitti-3d-car: https://github.com/open-mmlab/mmdetection3d/blob/c8347b7ed933d70fcfbfb73a3541046b8c8e8f5e/configs/base/datasets/kitti-3d-car.py#L118

• Test in kitti-3d-car: https://github.com/open-mmlab/mmdetection3d/blob/c8347b7ed933d70fcfbfb73a3541046b8c8e8f5e/configs/base/datasets/kitti-3d-car.py#L129

[ZCMax]

If you want to run the model on test dataset, you need to change kitti_infos_val.pkl to kitti_infos_test.pkl, then just follow the instructions in Testing and make a submission parf of https://mmdetection3d.readthedocs.io/en/latest/datasets/kitti_det.html.

[holtvogt]

Thanks! Won't I need to change the split parameter to 'testing' as well? Or what's the meaning behind this parameter? Does it affect the data preparation?

[ZCMax]

yes, you need to change it to testing as well, split means split of input data. KITTI dataset is divided into testing and training part after data preparation, you can find the data directory in our doc.

[holtvogt]

By doing so, I receive this after using the testing command:

Command:

python tools/test.py <config.py> <checkpoint.pth> --eval mAP

Error:

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 31/31, 19.3 task/s, elapsed: 2s, ETA:     0s
Converting prediction to KITTI format
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 31/31, 9838.3 task/s, elapsed: 0s, ETA:     0s
Result is saved to /tmp/tmphn8c_znk/results.pkl.
Traceback (most recent call last):
  File "tools/test.py", line 260, in <module>
    main()
  File "tools/test.py", line 256, in main
    print(dataset.evaluate(outputs, **eval_kwargs))
  File "/home/ws/x/Dokumente/Repositories/mmdetection3d/mmdet3d/datasets/itiv_dataset.py", line 201, in evaluate
    gt_annos = [info['annos'] for info in self.data_infos]
  File "/home/ws/x/Dokumente/Repositories/mmdetection3d/mmdet3d/datasets/itiv_dataset.py", line 201, in <listcomp>
    gt_annos = [info['annos'] for info in self.data_infos]
KeyError: 'annos'

I receive this with KITTI as well

[Joaovsky]

I have the exact same problem... Did you solved it?

[ZCMax]

By doing so, I receive this after using the testing command:

Command:

python tools/test.py <config.py> <checkpoint.pth> --eval mAP

Error:

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 31/31, 19.3 task/s, elapsed: 2s, ETA:     0s
Converting prediction to KITTI format
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 31/31, 9838.3 task/s, elapsed: 0s, ETA:     0s
Result is saved to /tmp/tmphn8c_znk/results.pkl.
Traceback (most recent call last):
  File "tools/test.py", line 260, in <module>
    main()
  File "tools/test.py", line 256, in main
    print(dataset.evaluate(outputs, **eval_kwargs))
  File "/home/ws/x/Dokumente/Repositories/mmdetection3d/mmdet3d/datasets/itiv_dataset.py", line 201, in evaluate
    gt_annos = [info['annos'] for info in self.data_infos]
  File "/home/ws/x/Dokumente/Repositories/mmdetection3d/mmdet3d/datasets/itiv_dataset.py", line 201, in <listcomp>
    gt_annos = [info['annos'] for info in self.data_infos]
KeyError: 'annos'

I receive this with KITTI as well

For test dataset, we do not have annotations, you need to submit the generated results to test server.

[Joaovsky]

I did that, though I obtained inside results/kitti-3class the folder kitti_resultspts_bbox, a kitti_resultspts_bbox.pkl and a results_eval.pkl. Also, inside that folder I have 7518 .txt files (from 000000.txt to 007517.txt) that are all empty... And both .pkl files have empty arrays and tensors

[holtvogt]

So, when using a custom, LiDAR-based dataset, MMDetection3D has no such testing functionality already implemented?

[Joaovsky]

Basically, they have unannotated data (images, point cloud). The problem is I need the annotations for the testing dataset and they don't have it. You can eitheir ask for submission to the website or do a training split and work with that new split as the test set.