IA-SSD
IA-SSD copied to clipboard
yifanzhang713
The running results are higher than those in the paper
Here are my results：
Car [email protected], 0.70, 0.70: bbox AP:96.4483, 90.1338, 89.4304 bev AP:90.4018, 88.7234, 86.8532 3d AP:89.2549, 79.5031, 78.3933 aos AP:96.41, 90.06, 89.26 Car [email protected], 0.70, 0.70: bbox AP:98.0873, 95.2228, 92.6929 bev AP:94.9569, 89.7264, 88.7445 3d AP:91.5120, 83.3311, 80.3146 aos AP:98.06, 95.13, 92.50 Car [email protected], 0.50, 0.50: bbox AP:96.4483, 90.1338, 89.4304 bev AP:96.5648, 90.2485, 89.7516 3d AP:96.5033, 90.2212, 89.6900 aos AP:96.41, 90.06, 89.26 Car [email protected], 0.50, 0.50: bbox AP:98.0873, 95.2228, 92.6929 bev AP:98.1673, 95.6244, 95.0024 3d AP:98.1293, 95.5496, 94.8515 aos AP:98.06, 95.13, 92.50 Pedestrian [email protected], 0.50, 0.50: bbox AP:72.6155, 70.4817, 67.0930 bev AP:67.1150, 61.2996, 56.7766 3d AP:63.1800, 58.5233, 52.3548 aos AP:68.11, 65.19, 61.52 Pedestrian [email protected], 0.50, 0.50: bbox AP:74.2353, 70.4244, 66.7236 bev AP:66.8335, 61.8515, 56.3665 3d AP:62.7567, 57.2351, 51.7856 aos AP:69.12, 64.52, 60.56 Pedestrian [email protected], 0.25, 0.25: bbox AP:72.6155, 70.4817, 67.0930 bev AP:81.8792, 78.3813, 73.3151 3d AP:81.6713, 78.2371, 73.1219 aos AP:68.11, 65.19, 61.52 Pedestrian [email protected], 0.25, 0.25: bbox AP:74.2353, 70.4244, 66.7236 bev AP:83.0304, 79.7448, 75.3895 3d AP:82.8435, 79.5333, 75.1033 aos AP:69.12, 64.52, 60.56 Cyclist [email protected], 0.50, 0.50: bbox AP:95.2029, 81.5999, 76.7175 bev AP:93.4303, 74.6626, 71.6172 3d AP:92.6345, 73.2495, 70.1991 aos AP:95.11, 81.35, 76.46 Cyclist [email protected], 0.50, 0.50: bbox AP:96.5793, 82.3327, 78.2697 bev AP:94.9052, 76.9932, 72.6225 3d AP:93.8756, 74.3714, 69.8898 aos AP:96.48, 82.08, 77.98 Cyclist [email protected], 0.25, 0.25: bbox AP:95.2029, 81.5999, 76.7175 bev AP:94.0906, 79.0643, 74.1753 3d AP:94.0766, 79.0527, 74.1413 aos AP:95.11, 81.35, 76.46 Cyclist [email protected], 0.25, 0.25: bbox AP:96.5793, 82.3327, 78.2697 bev AP:95.5894, 79.8844, 75.5143 3d AP:95.5789, 79.8644, 75.3781 aos AP:96.48, 82.08, 77.98
I understand that this is the result on KITTI data set: 83.3311 57.2351 74.3714. The result given in the paper is the result on the test set: 80.13 39.03 61.94
Can you tell me why there is such a big gap
I also meet this question. ar [email protected], 0.70, 0.70: bbox AP:90.7912, 90.1056, 89.3981 bev AP:90.1486, 88.1241, 86.3262 3d AP:89.0129, 79.3359, 78.1549 aos AP:90.78, 90.03, 89.25 Car [email protected], 0.70, 0.70: bbox AP:96.3784, 94.7560, 92.6249 bev AP:93.1338, 89.3610, 88.4013 3d AP:91.0773, 82.6979, 80.0324 aos AP:96.36, 94.66, 92.45 Car [email protected], 0.50, 0.50: bbox AP:90.7912, 90.1056, 89.3981 bev AP:90.7539, 90.1709, 89.6626 3d AP:90.7505, 90.1284, 89.5595 aos AP:90.78, 90.03, 89.25 Car [email protected], 0.50, 0.50: bbox AP:96.3784, 94.7560, 92.6249 bev AP:96.3739, 95.4765, 94.8889 3d AP:96.3537, 95.3647, 94.6830 aos AP:96.36, 94.66, 92.45 Pedestrian [email protected], 0.50, 0.50: bbox AP:73.5146, 70.7161, 67.5029 bev AP:62.6443, 57.1808, 51.5264 3d AP:58.2297, 51.3290, 47.5660 aos AP:67.48, 63.95, 60.63 Pedestrian [email protected], 0.50, 0.50: bbox AP:75.3936, 71.3427, 67.2052 bev AP:62.4366, 55.9178, 51.1720 3d AP:57.4104, 50.7089, 45.7775 aos AP:68.40, 63.70, 59.50 Pedestrian [email protected], 0.25, 0.25: bbox AP:73.5146, 70.7161, 67.5029 bev AP:81.9312, 77.1569, 72.9095 3d AP:81.9268, 77.1215, 72.8990 aos AP:67.48, 63.95, 60.63 Pedestrian [email protected], 0.25, 0.25: bbox AP:75.3936, 71.3427, 67.2052 bev AP:82.7359, 79.1794, 75.0512 3d AP:82.7374, 78.4838, 74.9875 aos AP:68.40, 63.70, 59.50 Cyclist [email protected], 0.50, 0.50: bbox AP:89.3332, 77.3331, 74.9401 bev AP:85.8449, 71.8508, 69.5462 3d AP:84.6466, 70.1752, 65.2901 aos AP:89.21, 76.95, 74.48 Cyclist [email protected], 0.50, 0.50: bbox AP:94.3156, 78.7866, 75.7217 bev AP:88.7303, 72.3587, 69.1804 3d AP:87.4132, 70.8158, 66.6779 aos AP:94.15, 78.36, 75.23 Cyclist [email protected], 0.25, 0.25: bbox AP:89.3332, 77.3331, 74.9401 bev AP:88.6376, 75.2272, 72.7598 3d AP:88.6376, 75.2272, 72.7598 aos AP:89.21, 76.95, 74.48 Cyclist [email protected], 0.25, 0.25: bbox AP:94.3156, 78.7866, 75.7217 bev AP:93.6530, 77.4996, 73.3536 3d AP:93.6530, 77.4996, 73.3536 aos AP:94.15, 78.36, 75.23
it's higher than those in paper
Hi, @hhhmrcscs @hht1996ok , very thanks for your intresting in IA-SSD.
First, your results are all on the validation split rather than test set, we have published the detailed resut in Tab. 3. we use R11 rather than R40 in valid split, so the 3d AP:89.2549, 79.5031, 78.3933; 3d AP:63.1800, 58.5233, 52.3548; 3d AP:84.6466, 70.1752, 65.2901 from @hhhmrcscs can be counterpart with it.
As for performance in Tab 2, all these results are from the KITTI benchmark, please see more info in http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d. Note that these model here are different with that for valid split.
Happy coding!
Hi, @hhhmrcscs @hht1996ok , very thanks for your intresting in IA-SSD.
First, your results are all on the validation split rather than test set, we have published the detailed resut in Tab. 3. we use R11 rather than R40 in valid split, so the 3d AP:89.2549, 79.5031, 78.3933; 3d AP:63.1800, 58.5233, 52.3548; 3d AP:84.6466, 70.1752, 65.2901 from @hhhmrcscs can be counterpart with it.
As for performance in Tab 2, all these results are from the KITTI benchmark, please see more info in http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d. Note that these model here are different with that for valid split.
Happy coding!
@yifanzhang713 Hi, I retrained IA-SSD and didn't change the settings. But the result I get for Pedestrian(54.5598) is much lower than in the paper.Is there any possible reason?
Car [email protected], 0.70, 0.70:
3d AP:89.2519, 79.3480, 78.2444
Pedestrian [email protected], 0.50, 0.50:
3d AP:57.4604, 54.5598, 49.8657
Cyclist [email protected], 0.50, 0.50:
3d AP:87.1069, 70.5114, 66.0638
my environment
Ubuntu 20.04
Python 3.7
CUDA11.3
Pytorch 1.10.1
OpenPCDet v0.5.2
spconv-cu113 2.1.21
Hi, @hhhmrcscs @hht1996ok , very thanks for your intresting in IA-SSD. First, your results are all on the validation split rather than test set, we have published the detailed resut in Tab. 3. we use R11 rather than R40 in valid split, so the 3d AP:89.2549, 79.5031, 78.3933; 3d AP:63.1800, 58.5233, 52.3548; 3d AP:84.6466, 70.1752, 65.2901 from @hhhmrcscs can be counterpart with it. As for performance in Tab 2, all these results are from the KITTI benchmark, please see more info in http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d. Note that these model here are different with that for valid split. Happy coding!
@yifanzhang713 Hi, I retrained IA-SSD and didn't change the settings. But the result I get for
Pedestrian(54.5598)is much lower than in the paper.Is there any possible reason?Car [email protected], 0.70, 0.70: 3d AP:89.2519, 79.3480, 78.2444 Pedestrian [email protected], 0.50, 0.50: 3d AP:57.4604, 54.5598, 49.8657 Cyclist [email protected], 0.50, 0.50: 3d AP:87.1069, 70.5114, 66.0638my environment
Ubuntu 20.04 Python 3.7 CUDA11.3 Pytorch 1.10.1 OpenPCDet v0.5.2 spconv-cu113 2.1.21
I am sorry for the late reply. The sample size and scene scale of pedestrians and cyclists in KITTI are too small compared to the car class, so the results inevitably fluctuate. This may be one of the factors that most single-stage detectors only focus on the car class, as far as we know, IASSD is the only single-stage point-based detector that simultaneously focuses on multi-class detection. So I suggest you can try to train a few more times on KITTI, or turn to test on a large dataset like waymo.
Hi, @hhhmrcscs @hht1996ok , very thanks for your intresting in IA-SSD. First, your results are all on the validation split rather than test set, we have published the detailed resut in Tab. 3. we use R11 rather than R40 in valid split, so the 3d AP:89.2549, 79.5031, 78.3933; 3d AP:63.1800, 58.5233, 52.3548; 3d AP:84.6466, 70.1752, 65.2901 from @hhhmrcscs can be counterpart with it. As for performance in Tab 2, all these results are from the KITTI benchmark, please see more info in http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d. Note that these model here are different with that for valid split. Happy coding!
@yifanzhang713 Hi, I retrained IA-SSD and didn't change the settings. But the result I get for
Pedestrian(54.5598)is much lower than in the paper.Is there any possible reason?Car [email protected], 0.70, 0.70: 3d AP:89.2519, 79.3480, 78.2444 Pedestrian [email protected], 0.50, 0.50: 3d AP:57.4604, 54.5598, 49.8657 Cyclist [email protected], 0.50, 0.50: 3d AP:87.1069, 70.5114, 66.0638my environment
Ubuntu 20.04 Python 3.7 CUDA11.3 Pytorch 1.10.1 OpenPCDet v0.5.2 spconv-cu113 2.1.21I am sorry for the late reply. The sample size and scene scale of pedestrians and cyclists in KITTI are too small compared to the car class, so the results inevitably fluctuate. This may be one of the factors that most single-stage detectors only focus on the car class, as far as we know, IASSD is the only single-stage point-based detector that simultaneously focuses on multi-class detection. So I suggest you can try to train a few more times on KITTI, or turn to test on a large dataset like waymo.
But why we can not reach the performerance in the paper as 58.91, the parameter is same as paper.......，and everytime the inferance result is inconsistent.
