HRNet-Semantic-Segmentation The performance gap

I have dowonload the pretrained model 'hrnet_w48_cityscapes_cls19_1024x2048_ohem_trainvalset.pth' and evaluate it on cityscapes test dataset, the results are as bellow, https://www.cityscapes-dataset.com/anonymous-results/?id=500275d541b67dd462afa9235b0fbe188e2fdd304e26d8f239a52a8bbfa2fb0d while the mIoU (76.6%) is largely behind the proposed results (81.6%).

I don't know why, is there something wrong?

Oct 09 '19 02:10 MandyMo

Because this model is trained based on pytorch-0.4.1, you should also run the test based on pytorch-0.4.1. The BN is different between pytorch-0.4.1 and pytorch-1.1, which results in worse performance.

Oct 09 '19 08:10 sunke123

Thank you.

Oct 09 '19 08:10 MandyMo

I have tested the model with pytorch0.4.1 and pytorch 1.1. I have random drawn several images as bellow, the left image presents the result for pytorch 0.4.1, and the right image presents the result for pytorch1.1.

It's strange that those the output from pytorch0.4.1 is identical with the counterpart produced by pythorch1.1.

Any details that I have missed ?

Oct 09 '19 16:10 MandyMo

Could you provide your testing settings and test the model on the val firstly?

Oct 10 '19 03:10 sunke123

I didn't test the model on the val firstly, and the test code I used as bellow:

Oct 11 '19 07:10 MandyMo

Yes. Due to the limitation of submission, you can test the model on the val. Then, we can check the problem together.

Oct 13 '19 03:10 sunke123

@MandyMo I have tested the model based on Pyotrch-0.4.1. You should get MIoU 91.97 on val.

Oct 13 '19 04:10 sunke123

Thank you, I will evaluate it on val part.

Oct 14 '19 03:10 MandyMo

I am sorry to trouble you again! I have evaluated the performance of the model on val set with Pytorch-0.4.1 (windows), while I can't reach the 91.97 iou. So I pick serveral image from the val set, can you offer me your evaluated results on the following five images.

frankfurt_000000_000294_gtFine_labelIds.png

frankfurt_000000_000576_gtFine_labelIds.png

frankfurt_000000_001016_gtFine_labelIds.png

frankfurt_000000_001236_gtFine_labelIds.png

frankfurt_000000_001751_gtFine_labelIds.png

Oct 21 '19 09:10 MandyMo

@MandyMo hello, I'm tring the HRNet code as with you. But when running "python tools/train.py --cfg experiments/cityscapes/seg_hrnet_w48_train_512x1024_sgd_lr1e-2_wd5e-4_bs_12_epoch484.yaml", I encounter the problem "ninja: build stopped: subcommand failed.". How about you? Are you encounter the same problem? If No, can you tell me you some suggestions? my pytorch is 1.1.0 and cuda 10.0. In addition, how do you install the packet "ninja",? the way:pip install ninja? if OK, I really hope to contact with you. my Email:1072319209qq.com. Thanks a lot!

Nov 05 '19 05:11 huangfuts

@MandyMo Thanks very much for your help! the problem, "ninja: build stopped: subcommand failed.", has been sovled. Wish you all the best~~~

Nov 06 '19 08:11 huangfuts

@MandyMo: have you re-produced the results?

Dec 04 '19 08:12 welleast

@MandyMo: have you re-produced the results?

No, I didn't reproduce the proposed results.

Dec 04 '19 09:12 MandyMo

@MandyMo did you use the same settings? what are your results?

Dec 11 '19 03:12 welleast

@huangfuts could you tell how you solved it my mail id is [email protected]

Jan 24 '21 05:01 ajithvcoder

I suspect this is an error with how the model is loading, you need to explicitly map the model weights to your device, otherwise whatever weights are not mapped will be randomly initiated.

See here for further info: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html (this is my guess for now, I am working through these issues)

Jul 01 '21 02:07 StuvX

HRNet-Semantic-Segmentation HRNet-Semantic-Segmentation copied to clipboard

The performance gap

HRNet-Semantic-Segmentation
HRNet-Semantic-Segmentation copied to clipboard