SVO Crashes after fast movement of the camera

[ginowu]

This issue can be easily reproduced when fast moving the camera. If not moving fast, this issue will every time pop up after several minutes operation of SVO. I am running SVO on an ARM board, with global shutter and wide FOV camera, the camera works under 60FPS, calibrated with PinHole model.

Everything works fine except this crash, if you need more information, please let me know, thanks!

Detailed backtrace logs as below:

[ INFO] [1459743582.568163366]: Init: KLT tracked 181 features [ INFO] [1459743582.568622265]: Init: KLT 50.8176px average disparity. [ INFO] [1459743582.775399697]: Init: Homography RANSAC 108 inliers. 2-View BA: Error before/after = 0.002261 / 0.002261 2-View BA: Wrong edges = 0 [ INFO] [1459743582.787112021]: Init: Selected second frame, triangulated initial map. [ WARN] [1459743611.638249973]: Lost 42 features! [ WARN] [1459743611.648372631]: Relocalizing frame [ INFO] [1459743611.662497000]: Relocalization successful. [ WARN] [1459743611.737652525]: Lost 41 features! [ INFO] [1459743611.763768587]: Relocalization successful. [ WARN] [1459743613.762549586]: Lost 42 features! [ WARN] [1459743613.773775509]: Relocalizing frame [ INFO] [1459743613.788123106]: Relocalization successful. [ WARN] [1459743613.863412742]: Lost 43 features! [ INFO] [1459743613.889019801]: Relocalization successful. [ WARN] [1459743614.063808865]: Lost 43 features! [ WARN] [1459743614.093338782]: Lost 41 features! [ INFO] [1459743614.145711997]: Relocalization successful. [ WARN] [1459743614.211366366]: Lost 44 features! [ INFO] [1459743614.244494664]: Relocalization successful. [ WARN] [1459743614.281957636]: Lost 44 features! [ WARN] [1459743614.311359638]: Lost 41 features! [ INFO] [1459743614.342045266]: Relocalization successful. [ WARN] [1459743614.592520420]: Lost 41 features! [ INFO] [1459743614.615404603]: Relocalization successful. [ WARN] [1459743615.267691521]: Lost 41 features! [ WARN] [1459743615.279631340]: Relocalizing frame [ INFO] [1459743615.295451985]: Relocalization successful. [ WARN] [1459743615.649694262]: Lost 41 features! [ INFO] [1459743615.679092253]: Relocalization successful. [ WARN] [1459743615.752373223]: Lost 41 features! [ WARN] [1459743615.777413207]: Lost 41 features! [ INFO] [1459743615.797496966]: Relocalization successful. [ WARN] [1459743627.076553125]: Lost 41 features! [ WARN] [1459743627.081644819]: Relocalizing frame [ INFO] [1459743627.102625490]: Relocalization successful. [ WARN] [1459743627.168522250]: Lost 41 features! [ WARN] [1459743627.197364468]: Lost 42 features! [ WARN] [1459743627.226960945]: Lost 41 features! [ WARN] [1459743627.262436553]: Lost 42 features! [ WARN] [1459743627.291777878]: Lost 46 features! [ WARN] [1459743627.319172254]: Lost 48 features! [ WARN] [1459743627.345300919]: Lost 44 features! [ INFO] [1459743627.371584739]: Relocalization successful. [ WARN] [1459743627.386140247]: Lost 41 features! [ WARN] [1459743627.418334811]: Lost 47 features! [ INFO] [1459743627.451215197]: Relocalization successful. [ WARN] [1459743627.471029843]: Lost 42 features! [ WARN] [1459743627.496638413]: Lost 41 features! [ INFO] [1459743627.527399456]: Relocalization successful. [ WARN] [1459743627.618765621]: Lost 41 features! [ INFO] [1459743627.662957449]: Relocalization successful. [ WARN] [1459743636.413749661]: Lost 41 features! [ WARN] [1459743636.420277479]: Relocalizing frame [ INFO] [1459743636.453482913]: Relocalization successful. [ WARN] [1459743665.863879871]: Lost 41 features! [ WARN] [1459743665.871568765]: Relocalizing frame [ INFO] [1459743665.897666233]: Relocalization successful. [ WARN] [1459743666.448371880]: Lost 41 features! [ WARN] [1459743666.508982834]: Lost 43 features! [ INFO] [1459743666.583197177]: Relocalization successful. [ WARN] [1459743676.873246847]: Tracking less than 50 features! [ WARN] [1459743676.881718541]: Relocalizing frame [ WARN] [1459743676.903617895]: Not enough matched features. [ INFO] [1459743677.186957291]: Relocalization successful. [ WARN] [1459743677.657077807]: Lost 41 features!

Program received signal SIGSEGV, Segmentation fault. svo::SparseImgAlign::computeResiduals (this=0xbeffe168, T_cur_from_ref=..., linearize_system=, compute_weight_scale=) at /root/catkin_ws/src/rpg_svo/svo/src/sparse_img_align.cpp:209 209 const float intensity_cur = w_cur_tl_cur_img_ptr[0] + w_cur_tr_cur_img_ptr[1] + w_cur_bl_cur_img_ptr[stride] + w_cur_br_cur_img_ptr[stride+1]; (gdb) bt #0 svo::SparseImgAlign::computeResiduals (this=0xbeffe168,

T_cur_from_ref=..., linearize_system=<optimized out>, 
compute_weight_scale=<optimized out>)
at /root/catkin_ws/src/rpg_svo/svo/src/sparse_img_align.cpp:209

#1 0xb6f2d18a in vk::NLLSSolver<6, Sophus::SE3>::optimizeGaussNewton (this=this@entry=0xbeffe168, model=...)

at /root/catkin_ws/src/rpg_vikit/vikit_common/include/vikit/nlls_solver_impl.hpp:45

#2 0xb6f2b960 in optimize (model=..., this=0xbeffe168)

at /root/catkin_ws/src/rpg_vikit/vikit_common/include/vikit/nlls_solver_impl.hpp:19

#3 svo::SparseImgAlign::run (this=this@entry=0xbeffe168,

ref_frame=..., cur_frame=...)
at /root/catkin_ws/src/rpg_svo/svo/src/sparse_img_align.cpp:68

#4 0xb6f0f702 in svo::FrameHandlerMono::processFrame (

this=0x517e0)

---Type to continue, or q to quit--- at /root/catkin_ws/src/rpg_svo/svo/src/frame_handler_mono.cpp:139 #5 0xb6f0ea98 in svo::FrameHandlerMono::relocalizeFrame (

this=0x517e0, T_cur_ref=..., ref_keyframe=...)
at /root/catkin_ws/src/rpg_svo/svo/src/frame_handler_mono.cpp:255

#6 0xb6f10a5e in svo::FrameHandlerMono::addImage (this=0x517e0,

img=..., timestamp=-0.076034542044246173)
at /root/catkin_ws/src/rpg_svo/svo/src/frame_handler_mono.cpp:82

#7 0x000125b6 in svo::VoNode::imgCb(boost::shared_ptr<sensor_msgs::Image_<std::allocator > const> const&) () #8 0xb0574eca in image_transport::RawSubscriber::internalCallback(boost::shared_ptr<sensor_msgs::Image_<std::allocator > const> const&, boost::function<void (boost::shared_ptr<sensor_msgs::Image_<std::allocator > const> const&)> const&) ()

from /opt/ros/indigo/lib//libimage_transport_plugins.so #9 0xb6f93000 in boost::detail::function::void_function_obj_invok---Type to continue, or q to quit---

er1<boost::function<void (boost::shared_ptr<sensor_msgs::Image_<std::allocator > const> const&)>, void, boost::shared_ptr<sensor_msgs::Image_<std::allocator > const> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<sensor_msgs::Image_<std::allocator > const>) () from /opt/ros/indigo/lib/libimage_transport.so #10 0xb05779bc in ros::SubscriptionCallbackHelperTboost::shared_ptr<sensor_msgs::Image_<std::allocator const> const&, void>::call(ros::SubscriptionCallbackHelperCallParams&) ()

from /opt/ros/indigo/lib//libimage_transport_plugins.so #11 0xb6cd321c in ros::SubscriptionQueue::call() ()

from /opt/ros/indigo/lib/libroscpp.so #12 0xb6ca5fc2 in ros::CallbackQueue::callOneCB(ros::CallbackQueue::TLS*) () from /opt/ros/indigo/lib/libroscpp.so #13 0xb6ca68e8 in ros::CallbackQueue::callAvailable(ros::WallDuration) () from /opt/ros/indigo/lib/libroscpp.so #14 0xb6cc5dc6 in ros::spinOnce() ()

from /opt/ros/indigo/lib/libroscpp.so ---Type to continue, or q to quit--- #15 0x000118f6 in main ()

(gdb) l 204 uint8_t* cur_img_ptr = (uint8_t_) cur_img.data + (v_cur_i+y-patch_halfsize_)stride + (u_cur_i-patch_halfsize); 205 206 for(int x=0; x<patch_size_; ++x, ++pixel_counter, ++cur_img_ptr, ++ref_patch_cache_ptr) 207 { 208 // compute residual 209 const float intensity_cur = w_cur_tlcur_img_ptr[0] + w_cur_trcur_img_ptr[1] + w_cur_blcur_img_ptr[stride] + w_cur_brcur_img_ptr[stride+1]; 210 const float res = intensity_cur - (ref_patch_cache_ptr); 211 212 // used to compute scale for robust cost 213 if(compute_weight_scale) (gdb) p $pc $1 = (void ()(void)) 0xb6f2b0be <svo::SparseImgAlign::computeResiduals(Sophus::SE3 const&, bool, bool)+678> (gdb) $2 = (void (*)(void)) 0xb6f2b0be <svo::SparseImgAlign::computeResiduals(Sophus::SE3 const&, bool, bool)+678> (gdb) p cur_img_ptr $3 = (uint8_t *) 0x803ed865 <error: Cannot access memory at address 0x803ed865> (gdb) p cur_img_ptr[0] Cannot access memory at address 0x803ed865 (gdb) p patch_size_ $4 = 4 (gdb) p x $5 = 0 (gdb) p cur_img.data $6 = (uchar *) 0x3ed8e0 "\253w@m\201\247\304\274\315\322\355\374\374\375\370\372\375\376\375", '\376' <repeats 20 times>, "\375\327\206DZe\241\343\256\227\300\375\376\376\373\365\357\372\375\353\375", '\376' <repeats 20 times>, "\240k\s\231\300\305\214p\206\270\345\375\375\375\371\372\357\335\375", '\376' <repeats 20 times>, "\202p\274\275\266\263\300\257fiu\204\243\320\366\374\375\361\361", '\376' <repeats 21 times>, "QT\300\317\315\314\273\236]Y]hs\201\224\302\360\376\361", '\376' <repeats 21 times>... (gdb) p stride $7 = 40 (gdb) p v_cur_i $8 = (gdb) p y $9 = 0 (gdb) p patch_halfsize_[svo-1] killing on exit Quit (gdb) p patch_halfsize_ $10 = 2 (gdb) [svo-1] escalating to SIGTERM shutting down processing monitor... ... shutting down processing monitor complete done

[fangthu]

@ginowu hi I think I also met with this problem. The reason maybe this :

When the camera is initialed or relocation, the algorithm capture the features in the picture. When the camera move fast, many features are no longer in the picture any more. So It lost many features. If the number of lost features is over than 40, the algorithm need to be relocated.

Of course, you can change the parameter of 'quality_max_drop_fts' to smaller value. But in my opinion, this algorithm is not suitable when the scenario in picture changes fast.

[zzaurak]

@ginowu have you found a solution yet? same problem here, i'm using raspberry pi 3 and raspicam. works kind of fine, but the occasional segfaults always come after fast movement of the camera... gdb shows the same output, seems to have something to do with this line... /root/catkin_ws/src/rpg_svo/svo/src/sparse_img_align.cpp:209 209 const float intensity_cur = w_cur_tl_cur_img_ptr[0] + w_cur_tr_cur_img_ptr[1] + w_cur_bl_cur_img_ptr[stride] + w_cur_br_cur_img_ptr[stride+1]; has anybody else a recomendation how to trace this problem down?

[maocong90]

@ginowu @fangthu @zzaurak have you found a solution yet? hi,I think I solved it! The reason is that the numbers are out of bounds. (gdb) p $pc $1 = (void ()(void)) 0xb6f2b0be <svo::SparseImgAlign::computeResiduals(Sophus::SE3 const&, bool, bool)+678> (gdb) p stride $4=160 (gdb) p scale $5=0.25 (gdb) p depth $6=1.081818115504404 (gdb) p u_cur $7=2.01064364e+14 —— it's wrong (gdb) p v_cur $8=1.36040793e+13 —— it's wrong (gdb) p u_cur_i $8=2147483647 —— it's wrong in 32-bit ，because of ' u_cur_i+border'

[maocong90]

@ginowu I also met with this problem. I am running SVO on pi 3 or pi 2, with Logitech camera or sj7000 camera, but FPS less than 12 . In ad dition, I am using usb_cam to get the image stream. So I am eager to know how to improve the low fps. Can you tell me more about the ARM board?

[maocong90]

@zzaurak How to implement svo with raspicam in pi2(or pi3) ?