VOC 2012 TEST results

[shaibagon]

Hi, I took the liberty of submitting the resnet pre-trained model results on VOC2012 TEST to the evaluation server. These are the results on the TEST set:

Competition | Category | Score Object Segmentation (comp6) | background | 94.55750 Object Segmentation (comp6) | aeroplane | 92.93460 Object Segmentation (comp6) | bicycle | 40.53500 Object Segmentation (comp6) | bird | 92.66040 Object Segmentation (comp6) | boat | 66.97070 Object Segmentation (comp6) | bottle | 76.63850 Object Segmentation (comp6) | bus | 94.08540 Object Segmentation (comp6) | car | 89.05940 Object Segmentation (comp6) | cat | 91.52270 Object Segmentation (comp6) | chair | 37.15340 Object Segmentation (comp6) | cow | 89.01260 Object Segmentation (comp6) | diningtable | 67.03180 Object Segmentation (comp6) | dog | 86.18840 Object Segmentation (comp6) | horse | 90.35510 Object Segmentation (comp6) | motorbike | 85.19530 Object Segmentation (comp6) | person | 85.28630 Object Segmentation (comp6) | pottedplant | 67.78140 Object Segmentation (comp6) | sheep | 86.65630 Object Segmentation (comp6) | sofa | 56.42840 Object Segmentation (comp6) | train | 85.75310 Object Segmentation (comp6) | tvmonitor | 76.54340   | Object Segmentation (comp6) | 78.68330

Note that I changed the evaluation to include the entire image at scale 513px (fixing #51)

These results are SIGNIFICANTLY lower (~10%) than reported results on deeplab v3 + aspp (88.5%)

[zhijiew]

Hi, Could you share your training settings in detail? To reproduce the result in pascal voc leader board, you need to follow these steps:

1. Pretrain on COCO
2. Train with pascal voc aug dataset, use one-tenth of the learning rate in the previous step
3. Train with pascal voc dataset, use the training set and validation together and use one-tenth of the learning rate in the previous step
4. inference with multi scale and flip

[shaibagon]

@zhijiew I did not train the model. I used the pretrained weights @jfzhang95 kindly provided for resnet backbone.

As for inference: I did not do any "tricks": single scale (@513px resolution) no flipping.

[zhijiew]

OK, I see, the pretrained weights provided in this report is for validation set, not for test set, you need use different training methods if you want to get better result in test set, you can try to follow my steps mentioned above. More details can be found here : )

[ymli39]

hi @zhijiew I tried to use the same network backbone and pre-trained the network on COCO, but when I start to finetune, I tried learning rate of 0.001, 0.0005, 0.0001 for 50 epochs, the performance is around 78 but never get improved. Did I do anything wrong?

[zhijiew]

@ymli39 Could you give more details about your training settings? Like data set split, batch size, input size, crop size and inference strategy (flip or multi scale etc.)?

[ymli39]

hi @zhijiew , I used same data setting code as provided (train set + trainaug), batchsize of 24, input size and crop size of 513. I did not do any flip or multiscale during training's validation. I trained the model without coco pretrain first, the validation accuracy is 78.5, then I trained coco using provided coco code, after finish coco pretrain, I start to train pascal based on coco pretrain, I did 50 epochs, but the iou did not improve at all.

[zhijiew]

@ymli39 You training steps are incomplete, as I said, the whole training process to reproduce the best performance on test set including 3 stages:

1. Train the model on coco.
2. Fine tune the model on pascal voc aug data set, which should include 11355 samples, here is the file list.
3. Fine tune the model original pascal voc data set, which should include 2912 samples (including the original training set and validation set), here is the file list.

And finally, inference the test input images with flip and multi scale.

[ymli39]

thanks! @zhijiew , for trainaug, do I save model for every epoch or I just save the best model on validation (I found that if I save the best model, the performance might not improve so the model will not be saved)? Right now I run trainaug set for 50 epochs, would it be too much?

[Barry-Zhou]

@ymli39 I see you get 78.5 in validation dataset. I used code that jfzhang95 provided and settings are default, lr is 0.007,lr-scheduler(poly),loss-type(ce),data(train set + trainaug), batchsize of 24, input size and crop size of 513. I did 50 epoch,but the validation accuracy is 75.5,do you have other operations?

[Vipermdl]

uracy i

Hi, My validation set result is 0.7506687327823487, similar to your situation, have you solved it?

[z-kkk]

Hi,I am a new at deep learning research, can you share your test code? Thank you very much! @shaibagon

[Alex6D]

@ymli39 I see you get 78.5 in validation dataset. I used code that jfzhang95 provided and settings are default, lr is 0.007,lr-scheduler(poly),loss-type(ce),data(train set + trainaug), batchsize of 24, input size and crop size of 513. I did 50 epoch,but the validation accuracy is 75.5,do you have other operations? @Barry-Zhou I have the same question with you, have you solved it?

[ymli39]

@Barry-Zhou @Alex6D Sorry about late response, I stopped working this project half year ago. I do not have any other operations. I also tried the coco pretrain model, there is no luck I can get test set score above 80% (for me coco pretrain does not improve at all). If you want to improve the validation score, (it might be just overfitting, without actually improving the test score) you could do a hard negative mining on specific classes.

[dayekuaipao]

So should I resize the picture to 513*513,then crop;or crop the picture directly?

[clownrat6]

Why is my validation mIoU just ≈0.6? Input shape: (256, 256) batch size: 4 lr: 0.001 epoch: 30

[xLuge]

Hi, Could you share your training settings in detail? To reproduce the result in pascal voc leader board, you need to follow these steps:

1. Pretrain on COCO
2. Train with pascal voc aug dataset, use one-tenth of the learning rate in the previous step
3. Train with pascal voc dataset, use the training set and validation together and use one-tenth of the learning rate in the previous step
4. inference with multi scale and flip

should we freeze the syncbn when we finetune the model trained after coco?

[HongkLin]

Hi, Could you share your training settings in detail? To reproduce the result in pascal voc leader board, you need to follow these steps:

1. Pretrain on COCO
2. Train with pascal voc aug dataset, use one-tenth of the learning rate in the previous step
3. Train with pascal voc dataset, use the training set and validation together and use one-tenth of the learning rate in the previous step
4. inference with multi scale and flip

I want to know more detail about inference with multi scale and flip, it is used in the inference of the 3 point or all about the 4 point?