yifanzhang713
Question about the batch size setting when testing the model
It seems that the batch size will affect the permformance when testing the model using the same pretrained weight file. Here's my result:
- testing with batch_size=8
Car [email protected], 0.70, 0.70:
bbox AP:96.2771, 90.1569, 89.5080
bev AP:90.4018, 88.8616, 86.9882
3d AP:89.3976, 79.5647, 78.4486
aos AP:96.25, 90.10, 89.35
Car [email protected], 0.70, 0.70:
bbox AP:97.8890, 95.3317, 92.7613
bev AP:94.7081, 91.3551, 88.8493
3d AP:91.7842, 83.3762, 80.3495
aos AP:97.86, 95.24, 92.57
Car [email protected], 0.50, 0.50:
bbox AP:96.2771, 90.1569, 89.5080
bev AP:96.3634, 90.2276, 89.7638
3d AP:96.3225, 90.2001, 89.6938
aos AP:96.25, 90.10, 89.35
Car [email protected], 0.50, 0.50:
bbox AP:97.8890, 95.3317, 92.7613
bev AP:97.9439, 95.6098, 95.0311
3d AP:97.9191, 95.5267, 94.8690
aos AP:97.86, 95.24, 92.57
Pedestrian [email protected], 0.50, 0.50:
bbox AP:72.8288, 70.9435, 67.4487
bev AP:66.6826, 61.2784, 57.2613
3d AP:60.7116, 58.2224, 52.1716
aos AP:68.53, 65.94, 62.35
Pedestrian [email protected], 0.50, 0.50:
bbox AP:74.6393, 70.8025, 67.1719
bev AP:66.3106, 61.6625, 56.5282
3d AP:61.5620, 56.8584, 51.7222
aos AP:69.55, 65.18, 61.33
Pedestrian [email protected], 0.25, 0.25:
bbox AP:72.8288, 70.9435, 67.4487
bev AP:81.5755, 78.6000, 73.2873
3d AP:81.5028, 78.5548, 73.2312
aos AP:68.53, 65.94, 62.35
Pedestrian [email protected], 0.25, 0.25:
bbox AP:74.6393, 70.8025, 67.1719
bev AP:82.6367, 79.7249, 75.3315
3d AP:82.5666, 79.6892, 75.1957
aos AP:69.55, 65.18, 61.33
Cyclist [email protected], 0.50, 0.50:
bbox AP:95.4895, 78.1226, 76.4692
bev AP:93.0930, 74.5530, 72.0119
3d AP:84.9917, 71.1507, 68.6455
aos AP:95.39, 77.80, 76.02
Cyclist [email protected], 0.50, 0.50:
bbox AP:96.5594, 81.0552, 78.2527
bev AP:94.3008, 75.5013, 72.4293
3d AP:89.7271, 71.4147, 68.1409
aos AP:96.45, 80.69, 77.77
Cyclist [email protected], 0.25, 0.25:
bbox AP:95.4895, 78.1226, 76.4692
bev AP:94.3544, 79.1233, 74.0973
3d AP:94.3544, 79.1233, 74.0973
aos AP:95.39, 77.80, 76.02
Cyclist [email protected], 0.25, 0.25:
bbox AP:96.5594, 81.0552, 78.2527
bev AP:95.5326, 79.4494, 75.4784
3d AP:95.5326, 79.4494, 75.4784
aos AP:96.45, 80.69, 77.77
- testing with batch_size=1
Car [email protected], 0.70, 0.70:
bbox AP:96.4403, 90.1502, 89.4654
bev AP:90.4357, 88.8551, 87.2133
3d AP:89.4362, 79.5503, 78.4923
aos AP:96.39, 90.05, 89.27
Car [email protected], 0.70, 0.70:
bbox AP:98.1592, 95.2860, 92.7122
bev AP:95.0694, 91.5086, 88.9651
3d AP:91.9366, 83.4404, 80.4498
aos AP:98.11, 95.15, 92.49
Car [email protected], 0.50, 0.50:
bbox AP:96.4403, 90.1502, 89.4654
bev AP:96.5742, 90.2334, 89.7347
3d AP:96.5268, 90.2056, 89.6426
aos AP:96.39, 90.05, 89.27
Car [email protected], 0.50, 0.50:
bbox AP:98.1592, 95.2860, 92.7122
bev AP:98.2310, 95.6126, 95.0153
3d AP:98.2032, 95.5374, 94.8227
aos AP:98.11, 95.15, 92.49
Pedestrian [email protected], 0.50, 0.50:
bbox AP:72.4337, 70.8737, 67.2378
bev AP:67.0958, 61.2871, 57.6957
3d AP:63.0052, 58.1972, 51.7695
aos AP:67.72, 65.71, 61.87
Pedestrian [email protected], 0.50, 0.50:
bbox AP:74.4579, 71.7198, 67.1743
bev AP:66.6856, 61.9405, 56.6848
3d AP:62.4795, 56.7675, 51.4291
aos AP:68.90, 65.67, 60.93
Pedestrian [email protected], 0.25, 0.25:
bbox AP:72.4337, 70.8737, 67.2378
bev AP:82.2410, 79.3855, 73.3600
3d AP:82.1994, 79.1053, 73.2722
aos AP:67.72, 65.71, 61.87
Pedestrian [email protected], 0.25, 0.25:
bbox AP:74.4579, 71.7198, 67.1743
bev AP:84.0556, 80.6710, 75.3752
3d AP:83.9976, 80.5698, 75.2735
aos AP:68.90, 65.67, 60.93
Cyclist [email protected], 0.50, 0.50:
bbox AP:95.0359, 77.4640, 76.2851
bev AP:85.9516, 72.4271, 70.1423
3d AP:84.7977, 69.2345, 64.6304
aos AP:94.92, 77.19, 75.87
Cyclist [email protected], 0.50, 0.50:
bbox AP:96.3046, 80.8322, 77.9135
bev AP:90.6485, 72.6650, 69.5093
3d AP:89.1079, 69.1793, 64.9591
aos AP:96.18, 80.47, 77.49
Cyclist [email protected], 0.25, 0.25:
bbox AP:95.0359, 77.4640, 76.2851
bev AP:93.6113, 74.7370, 73.4226
3d AP:93.6113, 74.7370, 73.4226
aos AP:94.92, 77.19, 75.87
Cyclist [email protected], 0.25, 0.25:
bbox AP:96.3046, 80.8322, 77.9135
bev AP:94.9939, 77.7644, 74.7668
3d AP:94.9939, 77.7644, 74.7668
aos AP:96.18, 80.47, 77.49
Specifically, the AP_0.5 of Cyclist under the batch_size=1 setting is about 1.9 lower than that under the batch_size=8 setting, and the AP_0.25 of Cyclist under the batch_size=1 setting is nearly 5 lower than that under the batch_size=8 setting.
So why would this happen? Can you give me some insights to handle this problem? Thanks!
@Monster2333333 Hi, Could you share your pretrained model and environment configuration?
I train with default settings, but the result I get for Pedestrian(54.5598) is much lower than in the paper.
Car [email protected], 0.70, 0.70:
3d AP:89.2519, 79.3480, 78.2444
Pedestrian [email protected], 0.50, 0.50:
3d AP:57.4604, 54.5598, 49.8657
Cyclist [email protected], 0.50, 0.50:
3d AP:87.1069, 70.5114, 66.0638
my environment
Ubuntu 20.04
Python 3.7
CUDA11.3
Pytorch 1.10.1
OpenPCDet v0.5.2
spconv-cu113 2.1.21
@kellen5l Hi, I just use the pretrained file provided by this repo. Maybe you can try to set the batch_size=8 when testing your model. BTW, my environment
Ubuntu 16.04.7 LTS
Python 3.8.12
CUDA 10.2
Pytorch 1.8.0
OpenPCDet v0.5.2
spconv 1.2.1
I hope this will help you.
@Monster2333333 thanks, I do use batch_ Size = 8. I was wondering if it was the problem of spconv version.
BTW, Could I ask you a question?
In IA-SSD.yaml line:44, AGGREGATION_MLPS[3]=[256], but it does not match to the network structure. Is there any possible reason? @yifanzhang713
IA-SSD.yaml
...
line:44 AGGREGATION_MLPS: [[64], [128], [256], [256], [], [512]]
...
network structure
...
(3): PointnetSAModuleMSG_WithSampling(
(groupers): ModuleList()
(mlps): ModuleList()
)
(4): Vote_layer(
...
@kellen5l It seems that the IA-SSD doesn't really use the spconv lib, I don't think it is the problem of spconv version.
In IA-SSD.yaml line 33 & 34 & 38, the RADIUS_LIST, NSAMPLE_LIST, MLPS are all empty for (3): PointnetSAModuleMSG_WithSampling layer. According to the definition of PointnetSAModuleMSG_WithSampling in pointnet2_modules.py, it will just gather the features using sampled index when there is no grouper.
Maybe this layer is the Centroid Aggregation module proposed in the paper, I guess.
Do you have any idea?
@Monster2333333 thanks a lot, but sorry I have no idea. I may study this problem this weekend. I've been busy looking for the problem of poor accuracy of Pedestrian(54.5598) class recently.
I switched the environment to the same as the official, but the Pedestrian(56.2933) class still has some performance gaps.
Car [email protected], 0.70, 0.70:
3d AP:88.8191, 79.0768, 77.9675
Pedestrian [email protected], 0.50, 0.50:
3d AP:60.4749, 56.2933, 51.8100
Cyclist [email protected], 0.50, 0.50:
3d AP:87.8407, 71.2274, 65.6902
在使用相同的预训练权重文件测试模型时，批量大小似乎会影响性能。 这是我的结果：
- 使用 batch_size=8 进行测试
Car [email protected], 0.70, 0.70: bbox AP:96.2771, 90.1569, 89.5080 bev AP:90.4018, 88.8616, 86.9882 3d AP:89.3976, 79.5647, 78.4486 aos AP:96.25, 90.10, 89.35 Car [email protected], 0.70, 0.70: bbox AP:97.8890, 95.3317, 92.7613 bev AP:94.7081, 91.3551, 88.8493 3d AP:91.7842, 83.3762, 80.3495 aos AP:97.86, 95.24, 92.57 Car [email protected], 0.50, 0.50: bbox AP:96.2771, 90.1569, 89.5080 bev AP:96.3634, 90.2276, 89.7638 3d AP:96.3225, 90.2001, 89.6938 aos AP:96.25, 90.10, 89.35 Car [email protected], 0.50, 0.50: bbox AP:97.8890, 95.3317, 92.7613 bev AP:97.9439, 95.6098, 95.0311 3d AP:97.9191, 95.5267, 94.8690 aos AP:97.86, 95.24, 92.57 Pedestrian [email protected], 0.50, 0.50: bbox AP:72.8288, 70.9435, 67.4487 bev AP:66.6826, 61.2784, 57.2613 3d AP:60.7116, 58.2224, 52.1716 aos AP:68.53, 65.94, 62.35 Pedestrian [email protected], 0.50, 0.50: bbox AP:74.6393, 70.8025, 67.1719 bev AP:66.3106, 61.6625, 56.5282 3d AP:61.5620, 56.8584, 51.7222 aos AP:69.55, 65.18, 61.33 Pedestrian [email protected], 0.25, 0.25: bbox AP:72.8288, 70.9435, 67.4487 bev AP:81.5755, 78.6000, 73.2873 3d AP:81.5028, 78.5548, 73.2312 aos AP:68.53, 65.94, 62.35 Pedestrian [email protected], 0.25, 0.25: bbox AP:74.6393, 70.8025, 67.1719 bev AP:82.6367, 79.7249, 75.3315 3d AP:82.5666, 79.6892, 75.1957 aos AP:69.55, 65.18, 61.33 Cyclist [email protected], 0.50, 0.50: bbox AP:95.4895, 78.1226, 76.4692 bev AP:93.0930, 74.5530, 72.0119 3d AP:84.9917, 71.1507, 68.6455 aos AP:95.39, 77.80, 76.02 Cyclist [email protected], 0.50, 0.50: bbox AP:96.5594, 81.0552, 78.2527 bev AP:94.3008, 75.5013, 72.4293 3d AP:89.7271, 71.4147, 68.1409 aos AP:96.45, 80.69, 77.77 Cyclist [email protected], 0.25, 0.25: bbox AP:95.4895, 78.1226, 76.4692 bev AP:94.3544, 79.1233, 74.0973 3d AP:94.3544, 79.1233, 74.0973 aos AP:95.39, 77.80, 76.02 Cyclist [email protected], 0.25, 0.25: bbox AP:96.5594, 81.0552, 78.2527 bev AP:95.5326, 79.4494, 75.4784 3d AP:95.5326, 79.4494, 75.4784 aos AP:96.45, 80.69, 77.77
- 使用 batch_size=1 进行测试
Car [email protected], 0.70, 0.70: bbox AP:96.4403, 90.1502, 89.4654 bev AP:90.4357, 88.8551, 87.2133 3d AP:89.4362, 79.5503, 78.4923 aos AP:96.39, 90.05, 89.27 Car [email protected], 0.70, 0.70: bbox AP:98.1592, 95.2860, 92.7122 bev AP:95.0694, 91.5086, 88.9651 3d AP:91.9366, 83.4404, 80.4498 aos AP:98.11, 95.15, 92.49 Car [email protected], 0.50, 0.50: bbox AP:96.4403, 90.1502, 89.4654 bev AP:96.5742, 90.2334, 89.7347 3d AP:96.5268, 90.2056, 89.6426 aos AP:96.39, 90.05, 89.27 Car [email protected], 0.50, 0.50: bbox AP:98.1592, 95.2860, 92.7122 bev AP:98.2310, 95.6126, 95.0153 3d AP:98.2032, 95.5374, 94.8227 aos AP:98.11, 95.15, 92.49 Pedestrian [email protected], 0.50, 0.50: bbox AP:72.4337, 70.8737, 67.2378 bev AP:67.0958, 61.2871, 57.6957 3d AP:63.0052, 58.1972, 51.7695 aos AP:67.72, 65.71, 61.87 Pedestrian [email protected], 0.50, 0.50: bbox AP:74.4579, 71.7198, 67.1743 bev AP:66.6856, 61.9405, 56.6848 3d AP:62.4795, 56.7675, 51.4291 aos AP:68.90, 65.67, 60.93 Pedestrian [email protected], 0.25, 0.25: bbox AP:72.4337, 70.8737, 67.2378 bev AP:82.2410, 79.3855, 73.3600 3d AP:82.1994, 79.1053, 73.2722 aos AP:67.72, 65.71, 61.87 Pedestrian [email protected], 0.25, 0.25: bbox AP:74.4579, 71.7198, 67.1743 bev AP:84.0556, 80.6710, 75.3752 3d AP:83.9976, 80.5698, 75.2735 aos AP:68.90, 65.67, 60.93 Cyclist [email protected], 0.50, 0.50: bbox AP:95.0359, 77.4640, 76.2851 bev AP:85.9516, 72.4271, 70.1423 3d AP:84.7977, 69.2345, 64.6304 aos AP:94.92, 77.19, 75.87 Cyclist [email protected], 0.50, 0.50: bbox AP:96.3046, 80.8322, 77.9135 bev AP:90.6485, 72.6650, 69.5093 3d AP:89.1079, 69.1793, 64.9591 aos AP:96.18, 80.47, 77.49 Cyclist [email protected], 0.25, 0.25: bbox AP:95.0359, 77.4640, 76.2851 bev AP:93.6113, 74.7370, 73.4226 3d AP:93.6113, 74.7370, 73.4226 aos AP:94.92, 77.19, 75.87 Cyclist [email protected], 0.25, 0.25: bbox AP:96.3046, 80.8322, 77.9135 bev AP:94.9939, 77.7644, 74.7668 3d AP:94.9939, 77.7644, 74.7668 aos AP:96.18, 80.47, 77.49具体来说，batch_size=1 设置下 Cyclist 的 AP_0.5 比 batch_size=8 设置下低 1.9 左右，batch_size=1 设置下 Cyclist 的 AP_0.25 比 batch_size= 下低近 5 8设置。
那么为什么会发生这种情况呢？你能给我一些见解来解决这个问题吗？谢谢！
As far as I know, Batch size does affect the performance of the network, which is a problem common to almost all networks. You can check the specific reasons on the Internet
@Monster2333333 Hi, I guess the (4):Vote_layer is corresponding to the Contextual Centroid Prediction in paper, the (5): PointnetSAModuleMSG_WithSampling is corresponding to the Centroid Aggregation in paper. Am I right?
But I didn't find the code of centroid-aware sampling mentioned in paper. Do you have any idea?
...
(4): Vote_layer(
(mlp_modules): Sequential(
(0): Conv1d(256, 128, kernel_size=(1,), stride=(1,), bias=False)
(1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU()
)
(ctr_reg): Conv1d(128, 3, kernel_size=(1,), stride=(1,))
)
(5): PointnetSAModuleMSG_WithSampling(
(groupers): ModuleList(
...
@hhhmrcscs With the same weight file, all modules including the BNs should have same params, so the testing batch size should not affect the performance.
Can you give me a more specific explanation? Thanks!
@kellen5l I think you are right.
The implementation of centroid-aware sampling is in IASSD_head.py, I guess. The sa_centerness_mask corresponds to the mask used in the centroid-aware sampling module.
def get_sa_ins_layer_loss(self, tb_dict=None):
sa_ins_labels = self.forward_ret_dict['sa_ins_labels']
sa_ins_preds = self.forward_ret_dict['sa_ins_preds']
sa_centerness_mask = self.generate_sa_center_ness_mask()
sa_ins_loss, ignore = 0, 0
for i in range(len(sa_ins_labels)): # valid when i =1, 2
if len(sa_ins_preds[i]) != 0:
try:
point_cls_preds = sa_ins_preds[i][...,1:].view(-1, self.num_class)
except:
point_cls_preds = sa_ins_preds[i][...,1:].view(-1, 1)
else:
ignore += 1
continue
point_cls_labels = sa_ins_labels[i].view(-1)
positives = (point_cls_labels > 0)
negative_cls_weights = (point_cls_labels == 0) * 1.0
cls_weights = (negative_cls_weights + 1.0 * positives).float()
pos_normalizer = positives.sum(dim=0).float()
cls_weights /= torch.clamp(pos_normalizer, min=1.0)
one_hot_targets = point_cls_preds.new_zeros(*list(point_cls_labels.shape), self.num_class + 1)
one_hot_targets.scatter_(-1, (point_cls_labels * (point_cls_labels >= 0).long()).unsqueeze(dim=-1).long(), 1.0)
one_hot_targets = one_hot_targets[..., 1:]
if ('ctr' in self.model_cfg.LOSS_CONFIG.SAMPLE_METHOD_LIST[i+1][0]):
centerness_mask = sa_centerness_mask[i]
one_hot_targets = one_hot_targets * centerness_mask.unsqueeze(-1).repeat(1, one_hot_targets.shape[1])
point_loss_ins = self.ins_loss_func(point_cls_preds, one_hot_targets, weights=cls_weights).mean(dim=-1).sum()
loss_weights_dict = self.model_cfg.LOSS_CONFIG.LOSS_WEIGHTS
point_loss_ins = point_loss_ins * loss_weights_dict.get('ins_aware_weight',[1]*len(sa_ins_labels))[i]
sa_ins_loss += point_loss_ins
if tb_dict is None:
tb_dict = {}
tb_dict.update({
'sa%s_loss_ins' % str(i): point_loss_ins.item(),
'sa%s_pos_num' % str(i): pos_normalizer.item()
})
sa_ins_loss = sa_ins_loss / (len(sa_ins_labels) - ignore)
tb_dict.update({
'sa_loss_ins': sa_ins_loss.item(),
})
return sa_ins_loss, tb_dict
sorry, I read it wrong, I didn't notice it was a test, I think you're right
@Monster2333333 thanks a lot. BTW, Could you reproduced the result of recall rate for foreground points in Table.1? I was wondering if it was the problem with my reproduced code
4096 points
in paper D-FPS 98.3% 100% 97.2%
mine D-FPS 93.4% 99.6% 98.3%
@Monster2333333 thanks a lot. BTW, Could you reproduced the result of recall rate for foreground points in Table.1? I was wondering if it was the problem with my reproduced code
4096 points in paper D-FPS 98.3% 100% 97.2% mine D-FPS 93.4% 99.6% 98.3%
Could you please give me the code you copied, I also want to try