Double-free error (segfault) aligning deep and RGB with two cameras

[davidpob99]

Required Info
Camera Model	D435
Firmware Version	5.15.1
Operating System & Version	Linux 20.04.6 LTS
Kernel Version (Linux Only)	5.10.104
Platform	NVIDIA Jetson Xavier AGX
SDK Version	2.54.2
Language	C++
Segment	others

Issue Description

We are using two D435 cameras, each on its own thread. We have a producer-consumer architecture: the cameras are the producers (2) and other (2) threads consume the frames, saved in a double buffer (cv::Mat).

Everything were working as expected before trying to align the deep and RGB images with realsense, as follows (like the rs-align example):

void getFrame(cv::Mat *mat) {
    rs2::frameset data;
    rs2::align align_to_color(RS2_STREAM_COLOR);

    while (true) {
	try {
	    if(pipe.poll_for_frames(&data))
		break;
	} catch (rs2::error e) {
	    logger->log->warn(e.what());
	}
    }
    data = align_to_color.process(data);
    
    rs2::frame color_frame = data.get_color_frame();
    frameToMat(color_frame).copyTo(mat);
    auto rsDepthFrame = data.get_depth_frame();
}

At some point, and it differs from execution to execution (sometimes 5 minutes, 20 minutes, 2 hours, ...) it raises a segmentation fault. Using AddressSanitizer we found that it is because a double free problem in the line where we do the aligning:

==23695==ERROR: AddressSanitizer: attempting double-free on 0xfffe2151e800 in thread T70:  
    #0 0xffffa0932f70 in operator delete(void*) ../../../../src/libsanitizer/asan/asan_new_delete.cc:165
    #1 0xffff828705f0 in void std::vector >, std::allocator > > >::_M_realloc_insert<
std::shared_ptr > const&>(__gnu_cxx::__normal_iterator >*, std::vector
lazy >, std::allocator > > > >, std::shared_ptr > const&) (/usr/local/lib/librealsens
e2.so.2.54+0x4c35f0) 
    #2 0xffff82a191c8 in librealsense::extrinsics_graph::register_extrinsics(librealsense::stream_interface const&, librealsense::stream_interface const&, rs2_extrinsics) (/usr/local/lib/
librealsense2.so.2.54+0x66c1c8) 
    #3 0xffff82a4019c in rs2_register_extrinsics (/usr/local/lib/librealsense2.so.2.54+0x69319c)        
    #4 0xffff828dc07c in librealsense::align::create_aligned_profile(rs2::video_stream_profile&, rs2::video_stream_profile&) (/usr/local/lib/librealsense2.so.2.54+0x52f07c)
    #5 0xffff828dc478 in librealsense::align::allocate_aligned_frame(rs2::frame_source const&, rs2::video_frame const&, rs2::video_frame const&) (/usr/local/lib/librealsense2.so.2.54+0x52
f478)                   
    #6 0xffff828dc8c8 in librealsense::align::process_frame(rs2::frame_source const&, rs2::frame const&) (/usr/local/lib/librealsense2.so.2.54+0x52f8c8)
    #7 0xffff828f4ba4 in rs2::frame_processor_callback::on
_frame(rs2_frame*, rs2_source*) (/usr/local/lib/librealsense2.so.2.54+0x547ba4)
    #8 0xffff828f049c in librealsense::processing_block::invoke(librealsense::frame_holder) (/usr/local/lib/librealsense2.so.2.54+0x54349c)
    #9 0xffff82a48d80 in rs2_process_frame (/usr/local/lib/librealsense2.so.2.54+0x69bd80)
    #10 0xaaaabf9edd34 in rs2::processing_block::invoke(rs2::frame) const /usr/local/include/librealsense2/hpp/rs_processing.hpp:291
    #11 0xaaaabf9edd34 in rs2::filter::process(rs2::frame) const /usr/local/include/librealsense2/hpp/rs_processing.hpp:372
    #12 0xaaaabf9e9ccc in rs2::align::process(rs2::frameset) /usr/local/include/librealsense2/hpp/rs_processing.hpp:732                         
    #13 0xaaaabf9e9ccc in RealsenseCamera::getFrame(Frame&) /home/tt/yield-spray/src/producer/RealsenseCamera.cpp:135                         
    #14 0xaaaabf9e4034 in Producer::run() /home/tt/yield-spray/src/producer/Producer.cpp:65                
    #15 0xffff5edf7f98  (/usr/lib/aarch64-linux-gnu/libstdc++.so.6+0xccf98)
    #16 0xffff5ec30620 in start_thread /build/glibc-hV2QTJ/glibc-2.31/nptl/pthread_create.c:477           
    #17 0xffff5eb87498  (/usr/lib/aarch64-linux-gnu/libc.so.6+0xd1498)  

When we use only one camera (and then only one producer and one consumer) the problem does not happen. For that reason we thing it may be a bug in a shared structure.

Has anyone had the same issues, or can give me any hint?

Thank you!

[MartyG-RealSense]

Hi @davidpob99 I would recommend not using threads if possible as it makes a program more susceptible to segmentation faults that otherwise would not occur in a non-threaded program. If you need the cameras to be processed efficiently then the methods used in the official C++ SDK example program for multiple cameras called rs-multicam may work as well for you. It automatically detects all attached cameras and places each one on a separate pipeline.

https://github.com/IntelRealSense/librealsense/blob/master/examples/multicam/rs-multicam.cpp# #12556

If your project requires threads though then the multi-threading with alignment C++ code shared by a RealSense user at https://github.com/IntelRealSense/librealsense/issues/6865#issuecomment-661024268 may be a helpful reference.

[MartyG-RealSense]

Hi @davidpob99 Do you require further assistance with this case, please? Thanks!

[davidpob99]

Hi, thank you for your quick response.

The use of threads is mandatory in our program, so the solution in rs-multicam doesn't fit.

Also, I've tried the method suggested in https://github.com/IntelRealSense/librealsense/issues/6865#issuecomment-661024268 but the problem (double free error) persists.

[MartyG-RealSense]

You could test whether aligning color to depth instead of depth to color makes a difference.

rs2::align align_to_color(RS2_STREAM_DEPTH);

You could also test whether using wait_for_frames() instead of poll_for_frames() makes a positive difference. Whilst poll_for_frames() is the best choice for multiple camera applications, it is recommended when using poll_for_frames() that you should manually decide when to put the CPU to sleep and for how long, otherwise you can end up using 100% of a single CPU core. wait_for_frames() might provide better stability even though it is best suited to the single producer / single consumer model.

As your case is the only one in the history of this RealSense support forum that reports a double free error, there are no other references available to assist with debugging it, unfortunately.

[MartyG-RealSense]

Hi @davidpob99 Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.