zed.retrieve_objects unreliable hang in Python object detection script with non-realtime mode

[elias-work]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
• [X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

I have been experimenting with your new Python object detection demo (very nice!), and I am encountering I believe to be a race condition with zed.retrieve_objects that is exposed by using non-realtime-svo mode.

This script: https://github.com/stereolabs/zed-examples/blob/master/object detection/custom detector/python/pytorch_yolov5/detector.py

Steps to Reproduce

1. Set svo_real_time_mode=False here: https://github.com/stereolabs/zed-examples/blob/master/object%20detection/custom%20detector/python/pytorch_yolov5/detector.py#L142
2. Run the script on an svo file a few times. My estimate for the hang rate is between 30-70% so it is easy to trigger.

Expected Result

The script should always terminate regardless of real-time mode.

Actual Result

When in real-time mode (or other unknown factors are present), the script may hang at the last frame.

ZED Camera model

ZED Mini

Environment

OS: Ubuntu 20.04
Python: 3.8
CUDA: 11.1
ZED SDK: 3.6.3
GPU: 3080 Ti

Anything else?

• I have triggered this on very small files (3 MB) and slightly larger (30-50 MB) so I doubt input matters but I can upload a sample if it would be useful.
• My motivation for caring about real-time mode is for enforcing deterministic detections and tracking.

Here are a couple of gdb tracebacks from the hung thread.

Example 1

#0  0x00007fd73e0a8b2c in step_six(double**, int, int, int*, int*, int&) () at /usr/local/zed/lib/libsl_zed.so
#1  0x00007fd73e0a9492 in zed_tracking::Munkres(slutils::cv::Mat&, bool) () at /usr/local/zed/lib/libsl_zed.so
#2  0x00007fd73df58c89 in std::vector<BoxMeasurement*, std::allocator<BoxMeasurement*> > HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > >::associate<ASBoxAppearanceScore2D<BoxTrack<BoxStateEstimator<BoxMeasurement> > >, 2>(std::vector<BoxTrack<BoxStateEstimator<BoxMeasurement> >*, std::allocator<BoxTrack<BoxStateEstimator<BoxMeasurement> >*> >&, std::vector<BoxMeasurement*, std::allocator<BoxMeasurement*> >&, float) () at /usr/local/zed/lib/libsl_zed.so
#3  0x00007fd73df5dd37 in BoxAssociationBehavior<BoxTrack<BoxStateEstimator<BoxMeasurement> >, HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > > >::associate(std::vector<BoxTrack<BoxStateEstimator<BoxMeasurement> >*, std::allocator<BoxTrack<BoxStateEstimator<BoxMeasurement> >*> >&, std::vector<BoxMeasurement*, std::allocator<BoxMeasurement*> >&) () at /usr/local/zed/lib/libsl_zed.so
#4  0x00007fd73df5e566 in BoxTracker<BoxAssociationBehavior<BoxTrack<BoxStateEstimator<BoxMeasurement> >, HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > > >, BoxTrack<BoxStateEstimator<BoxMeasurement> >, 4>::associate_imp() () at /usr/local/zed/lib/libsl_zed.so
#5  0x00007fd73df5e834 in ObjectTrackersInterface<BoxTracker<BoxAssociationBehavior<BoxTrack<BoxStateEstimator<BoxMeasurement> >, HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > > >, BoxTrack<BoxStateEstimator<BoxMeasurement> >, 4>, BoxAssociationBehavior<BoxTrack<BoxStateEstimator<BoxMeasurement> >, HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > > >, BoxTrack<BoxStateEstimator<BoxMeasurement> > >::addMeasure_imp(ObjectTrackerCurrentFrameData<BoxMeasurement>*) () at /usr/local/zed/lib/libsl_zed.so
#6  0x00007fd73df5f8ca in std::vector<sl::Bbox2D, std::allocator<sl::Bbox2D> > ai::ObjectDetector::track<(OBJECT_DETECTION_SOURCE_TYPE)1>(sl::Mat&, sl::Mat&, sl::Timestamp, sl::Transform, sl::Transform, sl::Transform, std::vector<BBoxOut, std::allocator<BBoxOut> >&) () at /usr/local/zed/lib/libsl_zed.so
#7  0x00007fd73e0f04ee in sl::ObjectsDetectorHandler::computeObjectDetection(sl::Camera*) () at /usr/local/zed/lib/libsl_zed.so
#8  0x00007fd73e0d0548 in sl::Camera::retrieveObjects(sl::Objects&, sl::ObjectDetectionRuntimeParameters) () at /usr/local/zed/lib/libsl_zed.so
#9  0x00007fd746c54828 in __pyx_pf_5pyzed_2sl_6Camera_110retrieve_objects (__pyx_v_object_detection_parameters=<optimized out>, __pyx_v_py_objects=<optimized out>, __pyx_v_self=0x7fd70b0a9210)
    at /usr/include/c++/9/bits/stl_tree.h:129
#10 __pyx_pw_5pyzed_2sl_6Camera_111retrieve_objects(PyObject*, PyObject*, PyObject*) (__pyx_v_self=0x7fd70b0a9210, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at pyzed/sl.cpp:23572
#11 0x0000000000503ec9 in  ()
#12 0x000000000056b5e0 in _PyEval_EvalFrameDefault ()
#13 0x00000000005f6226 in _PyFunction_Vectorcall ()
#14 0x000000000056b3fe in _PyEval_EvalFrameDefault ()
#15 0x00000000005696da in _PyEval_EvalCodeWithName ()
#16 0x000000000068db17 in PyEval_EvalCode ()
#17 0x000000000067eeb1 in  ()
#18 0x000000000067ef2f in  ()
#19 0x000000000067efd1 in  ()
#20 0x000000000067f377 in PyRun_SimpleFileExFlags ()
#21 0x00000000006b7902 in Py_RunMain ()
#22 0x00000000006b7c8d in Py_BytesMain ()
#23 0x00007fd8e08410b3 in __libc_start_main (main=0x4eed30 <main>, argc=6, argv=0x7ffd5dcce498, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd5dcce488)
    at ../csu/libc-start.c:308
#24 0x00000000005fb12e in _start ()

Example 2

#0  0x00007fda4d9f758d in step_four(double**, int, int, int*, int*, double**, int&, int&, int&) () at /usr/local/zed/lib/libsl_zed.so
#1  0x00007fda4d9f86b4 in zed_tracking::Munkres(slutils::cv::Mat&, bool) () at /usr/local/zed/lib/libsl_zed.so
#2  0x00007fda4d8a7c89 in std::vector<BoxMeasurement*, std::allocator<BoxMeasurement*> > HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > >::associate<ASBoxAppearanceScore2D<BoxTrack<BoxStateEstimator<BoxMeasurement> > >, 2>(std::vector<BoxTrack<BoxStateEstimator<BoxMeasurement> >*, std::allocator<BoxTrack<BoxStateEstimator<BoxMeasurement> >*> >&, std::vector<BoxMeasurement*, std::allocator<BoxMeasurement*> >&, float) () at /usr/local/zed/lib/libsl_zed.so
#3  0x00007fda4d8acd37 in BoxAssociationBehavior<BoxTrack<BoxStateEstimator<BoxMeasurement> >, HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > > >::associate(std::vector<BoxTrack<BoxStateEstimator<BoxMeasurement> >*, std::allocator<BoxTrack<BoxStateEstimator<BoxMeasurement> >*> >&, std::vector<BoxMeasurement*, std::allocator<BoxMeasurement*> >&) () at /usr/local/zed/lib/libsl_zed.so
#4  0x00007fda4d8ad566 in BoxTracker<BoxAssociationBehavior<BoxTrack<BoxStateEstimator<BoxMeasurement> >, HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > > >, BoxTrack<BoxStateEstimator<BoxMeasurement> >, 4>::associate_imp() () at /usr/local/zed/lib/libsl_zed.so
#5  0x00007fda4d8ad834 in ObjectTrackersInterface<BoxTracker<BoxAssociationBehavior<BoxTrack<BoxStateEstimator<BoxMeasurement> >, HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > > >, BoxTrack<BoxStateEstimator<BoxMeasurement> >, 4>, BoxAssociationBehavior<BoxTrack<BoxStateEstimator<BoxMeasurement> >, HungarianAssociator<BoxTrack<BoxStateEstimator<BoxMeasurement> > > >, BoxTrack<BoxStateEstimator<BoxMeasurement> > >::addMeasure_imp(ObjectTrackerCurrentFrameData<BoxMeasurement>*) () at /usr/local/zed/lib/libsl_zed.so
#6  0x00007fda4d8ae8ca in std::vector<sl::Bbox2D, std::allocator<sl::Bbox2D> > ai::ObjectDetector::track<(OBJECT_DETECTION_SOURCE_TYPE)1>(sl::Mat&, sl::Mat&, sl::Timestamp, sl::Transform, sl::Transform, sl::Transform, std::vector<BBoxOut, std::allocator<BBoxOut> >&) () at /usr/local/zed/lib/libsl_zed.so
#7  0x00007fda4da3f4ee in sl::ObjectsDetectorHandler::computeObjectDetection(sl::Camera*) () at /usr/local/zed/lib/libsl_zed.so
#8  0x00007fda4da1f548 in sl::Camera::retrieveObjects(sl::Objects&, sl::ObjectDetectionRuntimeParameters) () at /usr/local/zed/lib/libsl_zed.so
#9  0x00007fda565a3828 in __pyx_pf_5pyzed_2sl_6Camera_110retrieve_objects (__pyx_v_object_detection_parameters=<optimized out>, __pyx_v_py_objects=<optimized out>, __pyx_v_self=0x7fd9ea9de7e0)
    at /usr/include/c++/9/bits/stl_tree.h:129
#10 __pyx_pw_5pyzed_2sl_6Camera_111retrieve_objects(PyObject*, PyObject*, PyObject*) (__pyx_v_self=0x7fd9ea9de7e0, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at pyzed/sl.cpp:23572
#11 0x0000000000503ec9 in  ()
#12 0x000000000056b5e0 in _PyEval_EvalFrameDefault ()
#13 0x00000000005f6226 in _PyFunction_Vectorcall ()
#14 0x000000000056b3fe in _PyEval_EvalFrameDefault ()
#15 0x00000000005696da in _PyEval_EvalCodeWithName ()
#16 0x000000000068db17 in PyEval_EvalCode ()
#17 0x000000000067eeb1 in  ()
#18 0x000000000067ef2f in  ()
#19 0x000000000067efd1 in  ()
#20 0x000000000067f377 in PyRun_SimpleFileExFlags ()
#21 0x00000000006b7902 in Py_RunMain ()
#22 0x00000000006b7c8d in Py_BytesMain ()
#23 0x00007fdbf01970b3 in __libc_start_main (main=0x4eed30 <main>, argc=8, argv=0x7ffde9df4db8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffde9df4da8)
    at ../csu/libc-start.c:308
#24 0x00000000005fb12e in _start ()

[SDuranMottMac]

Hey Elias, have you managed to sort it out? I am finding the same issue while running it without real-time mode

[elias-work]

Sorry, I didn't see this until now. My temporary solution was to terminate before reaching the last frame, which is a decent workaround. I don't know if this is still the same in the most recent updates.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[SDuranMottMac]

Hi Elias, thanks for the response! In our case, the retrieve_object method does not always get stuck at the end, perhaps because our videos are around 10-15 min long? how long are your videos?

[Myzhar]

The team is on this issue. A solution will be provided in one of the next SDK release

[SDuranMottMac]

Thanks Walter!

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[SDuranMottMac]

Hi all, there has been any progress on this issue? Thanks

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days