Problem for porting VINS-Mobile to Android Platform

[vonzy]

Hello, I am trying porting VINS to Android recently, and I have finished porting VINS-Mobile to Linux cmake version, which was tested successfully with EuRoc_V1_01_easy and EuRoc_MH_01_easy dateset. Several days ago, I collected imu and camera data on Smartisan M1L android mobile phone and have a test with the program, I found the result will get diverged immediately and the cost of nonlinear optimization is about 400-500. So, I wonder what factors may cause this problem, and ask you for help. In the presentation of professor Shen, he mentioned that sensor synchronization and camera-IMU rotation are very important. In Android mobile phone, the timestamp of images may be inaccurate because the image is obtained from a callback function, no timestamp can be obtained when the callback was called, and the timestamp can only be obtained in the function. Also, camera-IMU rotation is [0, -1, 0; -1, 0, 0; 0, 0, -1] in theory, but whether it can be used is not sure. Another question is the configuration of imu parameters, I don't know how to set those parameters. The imu of Smartisan M1L is BMI160. But it seems the parameters are not very important because when I use the parameters of iphone for EuRoc dataset, it also works. All in all, I don't know what to do to solve the problem now, could you please give me some advice? Thank you very much!!!

[PeiliangLi]

1. How did you calibrate your camera and which resolution image have you used?
2. Please log the imu raw data here.

[vonzy]

1. I have calibrated the camera with checkerboard and matlab toolbox, and the resolution is 640x480. the calibration result is

Focal Length:          fc = [ 570.55125   571.44376 ] +/- [ 18.45453   18.81954 ]
Principal point:       cc = [ 294.36555   219.09984 ] +/- [ 15.38305   13.54507 ]
Skew:             alpha_c = [ 0.00000 ] +/- [ 0.00000  ]   => angle of pixel axes = 90.00000 +/- 0.00000 degrees
Distortion:            kc = [ 0.11629   -0.08567   -0.01099   -0.01631  0.00000 ] +/- [ 0.04120   0.07725   0.00874   0.00979  0.00000 ]
Pixel error:          err = [ 0.82083   0.77598 ]

2. Here is part of imu raw data:

223498264914780,0.2734527587890625,0.43896484375,-0.1546630859375,10.327178955078125,2.08624267578125,3.0623626708984375
223498269980698,0.269195556640625,0.4187164306640625,-0.1280364990234375,10.333160400390625,2.1832122802734375,3.174896240234375
223498275016099,0.2702484130859375,0.3974151611328125,-0.1013946533203125,10.378646850585938,2.26580810546875,3.3161468505859375
223498280082017,0.2830352783203125,0.3761138916015625,-0.0715789794921875,10.389419555664063,2.2933502197265625,3.3484649658203125
223498285117417,0.3011474609375,0.348419189453125,-0.0438690185546875,10.358306884765625,2.3196868896484375,3.3580474853515625
223498290183335,0.3181915283203125,0.3143310546875,-0.0225677490234375,10.291259765625,2.363983154296875,3.391571044921875
223498295218735,0.335235595703125,0.2834320068359375,0.00299072265625,10.184722900390625,2.40228271484375,3.3329010009765625
223498300284653,0.34588623046875,0.2621307373046875,0.027496337890625,10.072189331054688,2.444183349609375,3.223968505859375
223498305320054,0.34375,0.2621307373046875,0.04986572265625,10.00994873046875,2.465728759765625,3.1246185302734375
223498310385972,0.3363037109375,0.2855682373046875,0.0690460205078125,10.031494140625,2.4729156494140625,3.05877685546875
223498315421372,0.313934326171875,0.3388214111328125,0.081817626953125,10.039871215820313,2.4549560546875,3.038421630859375
223498320487290,0.2819671630859375,0.4091339111328125,0.08502197265625,10.0997314453125,2.47650146484375,3.0096893310546875
223498325553208,0.2510833740234375,0.4783782958984375,0.0796966552734375,10.129653930664063,2.487274169921875,3.081512451171875
223498330619126,0.2191162109375,0.5465545654296875,0.07330322265625,10.069793701171875,2.54473876953125,3.1988372802734375
223498335654526,0.187164306640625,0.5987548828125,0.0669097900390625,9.989593505859375,2.5998077392578125,3.34967041015625
223498340720444,0.1605377197265625,0.6232452392578125,0.0647735595703125,9.914169311523438,2.6381072998046875,3.4382476806640625
223498345755845,0.140289306640625,0.6157989501953125,0.0669097900390625,9.800445556640625,2.6536712646484375,3.5687408447265625
223498350821763,0.1232452392578125,0.5785064697265625,0.075439453125,9.683135986328125,2.64410400390625,3.7387237548828125
223498355887681,0.116851806640625,0.515655517578125,0.0828857421875,9.525115966796875,2.5543212890625,3.8380889892578125
223498360923081,0.11578369140625,0.43682861328125,0.09674072265625,9.405410766601563,2.4609375,3.8440704345703125
223498365988999,0.121124267578125,0.35693359375,0.109527587890625,9.309646606445313,2.2251129150390625,3.8644256591796875
223498371024400,0.1371002197265625,0.280242919921875,0.1244354248046875,9.187530517578125,1.964141845703125,3.835693359375
223498376090317,0.163726806640625,0.2067413330078125,0.1382904052734375,9.082183837890625,1.7295074462890625,3.68963623046875
223498381125718,0.1946258544921875,0.142822265625,0.1500091552734375,8.986419677734375,1.5128326416015625,3.545989990234375
223498386191636,0.2255096435546875,0.0895538330078125,0.16278076171875,8.974456787109375,1.3967132568359375,3.413116455078125
223498391227036,0.2606658935546875,0.061859130859375,0.173431396484375,8.970855712890625,1.314117431640625,3.1832733154296875
223498396292954,0.2968902587890625,0.07037353515625,0.1808929443359375,9.00677490234375,1.2626495361328125,2.9941253662109375
223498401328355,0.325653076171875,0.11724853515625,0.1883544921875,9.067825317382813,1.2063751220703125,2.78582763671875
223498406394273,0.3480224609375,0.1939544677734375,0.1904754638671875,9.132461547851563,1.1848297119140625,2.62542724609375
223498411429673,0.3607940673828125,0.2908935546875,0.18621826171875,9.142044067382813,1.20159912109375,2.5535888671875
223498416495591,0.368255615234375,0.4102020263671875,0.177703857421875,9.162399291992188,1.2434844970703125,2.4913482666015625
223498421561509,0.368255615234375,0.5369720458984375,0.173431396484375,9.156417846679688,1.298553466796875,2.5188751220703125
223498426596909,0.35546875,0.6605377197265625,0.1702423095703125,9.090576171875,1.3847503662109375,2.679290771484375
223498431662827,0.3416290283203125,0.769195556640625,0.16705322265625,9.017547607421875,1.49847412109375,2.7942047119140625
223498436698228,0.3160552978515625,0.860809326171875,0.165985107421875,8.924179077148438,1.6277618408203125,2.9725799560546875
223498441764146,0.2841033935546875,0.92791748046875,0.1723785400390625,8.817626953125,1.6876220703125,3.1976318359375
223498446799546,0.253204345703125,0.9769134521484375,0.1766357421875,8.665603637695313,1.688812255859375,3.444244384765625
223498451865464,0.2244415283203125,1.0024871826171875,0.1830291748046875,8.527938842773438,1.636138916015625,3.6393585205078125
223498456900864,0.1903533935546875,0.99822998046875,0.1936798095703125,8.332809448242188,1.5966339111328125,3.842864990234375
223498461966782,0.16265869140625,0.9779815673828125,0.2064666748046875,8.120925903320313,1.5104522705078125,4.0332183837890625
223498467002183,0.1360321044921875,0.9332427978515625,0.220306396484375,7.9485321044921875,1.4422149658203125,4.19122314453125
223498472068101,0.10833740234375,0.868255615234375,0.2352294921875,7.81207275390625,1.3775634765625,4.2881927490234375
223498477103501,0.085968017578125,0.790496826171875,0.255462646484375,7.703125,1.298553466796875,4.3612213134765625
223498482169419,0.0678558349609375,0.7063446044921875,0.2735748291015625,7.666015625,1.2518768310546875,4.4031219482421875
223498487204819,0.0540008544921875,0.6221923828125,0.2970123291015625,7.654052734375,1.1668701171875,4.442626953125
223498492270737,0.047607421875,0.53802490234375,0.3172454833984375,7.6480712890625,1.10821533203125,4.41748046875
223498497336655,0.041229248046875,0.4560089111328125,0.333221435546875,7.6732025146484375,0.992095947265625,4.42706298828125
223498502372056,0.0401611328125,0.3739776611328125,0.3428192138671875,7.7725677490234375,0.853240966796875,4.3396759033203125
223498507437974,0.0444183349609375,0.29833984375,0.3534698486328125,7.8527679443359375,0.7562713623046875,4.2834014892578125
223498512473374,0.0529327392578125,0.221649169921875,0.35986328125,7.9269866943359375,0.648529052734375,4.190032958984375
223498517539292,0.062530517578125,0.153472900390625,0.3651885986328125,7.9521331787109375,0.56951904296875,4.1086273193359375

[vonzy]

the format of imu data: timestamp,wx,wy,wz,ax,ay,az

[PeiliangLi]

One possible reason is timestamp problem, the callback time may much later than the real image timestamp. The camera calibration result seems not very accurate since the principle point is far away from the image center. Can you tell the imu frame of your device, so I can verify the rotation matrix?

[vonzy]

imu frame? what do you mean?

[wuyuzhang]

@vonzy I'm also trying to move the open-sourced VINS to android platform. How is it going? Have you figured out the drift when rotating problem? Could you please share the no-ros project code on linux to me please and we can share the progress in the future together.

[vonzy]

@wuyuzhang 1.I am trying to improve VO part now, and big rotation problem is still hard for VO/VIO to work out. @FangGet is now working for the porting. 2.Okay, but how can I send the project to you? through e-mail? or IM?

[wuyuzhang]

@vonzy Thanks a lot. My email is: [email protected] Let's keep in touch :)

[vonzy]

@wuyuzhang I have send the project to your e-mail address, please check if you receive the e-mail.

[TouqeerAhmad]

@vonzy would you be able to share your VINS-android or VINS-ubuntu with cmake ??

[bileki]

@vonzy @wuyuzhang Could you send me the project by email? I would like to help.