ADVIO segmentation problem

[ZhouXiner]

Hi, I try to segment all the sequences in ADVIO dataset. I successfully segment some cases like 10, 13, 23 and the rmse is great. But I notice that the segmentation model is not so workable in highlighted corrior like in case 17. And I also find that the distortion influences the results badly. In case 12, the segmentation results is better than the reults that I dont use the dirtortion coeffs. So do you have any good solutions to the two problems, like some better distortion coeffs or models. Thanks a lot！ And here is my results. https://1drv.ms/u/s!AgIBb31yefjshg6ObUe0bhNB6OJR?e=Ej8Vol

[sudarshan-s-harithas]

Hi, The lighting conditions do affect the model predictions, they have been trained with synthetic images of Synthia and real images from ScaNet dataset, these datasets had appropriate lighting situations. And can you elaborate about the distortion coefficients that you have mentioned ?

[ZhouXiner]

Hi, thanks for reply! Before I use the segmenation model, I will first undirtort the images extracted from the raw frame.mvo(ADVIO provide). In case 12, I use the ditortion coefficients ADVIO dataset privode: https://github.com/AaltoVision/ADVIO/blob/master/calibration/iphone-02.yaml which is: intrinsics: cols: 1 rows: 4 data: [1077.2, 1079.3,362.145, 636.3873] distortion: type: radial-tangential
parameters: cols: 1 rows: 4 data: [0.0478, 0.0339, -0.00033, -0.00091] Also, I try to segment only with the intrinsics, the results is not so reasonable too. So I wonder maybe you have the better distortion coefficients.

[karnikram]

Hi @ZhouXiner,

These are interesting observations. We did not try undistorting the images before inference. Neither do we use the intrinsics in the inference script as we are not interested in the depth predictions from the network. We simply use the mask predictions from the raw images as they were good enough for our VIO system.

For better distortion coefficients you could maybe contact the ADVIO authors for access to their calibration sequences and try recalibrating yourself.