Segmentation fault in FastLineDetector

[jendabenda]

System information (version)

• OpenCV => 4.6.0
• Operating System / Platform => Ubuntu 22.04, docker guest
• Compiler => gcc 11.3.0

Detailed description

We use FastLineDetector in java:

Mat image = Imgcodecs.imdecode(new MatOfByte(bytes), Imgcodecs.IMREAD_GRAYSCALE);
FastLineDetector fld = Ximgproc.createFastLineDetector();
Mat lines = new Mat();

// detect lines
fld.detect(image, lines);

Inside cv::ximgproc::FastLineDetectorImpl::extractSegments, which is called by FastLineDetector.detect, segmentaion fault popped up. It was a rare case, most of the time the code works well.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f41589ce0bd, pid=550771, tid=553630
#
# JRE version: OpenJDK Runtime Environment (11.0.17+8) (build 11.0.17+8-post-Ubuntu-1ubuntu222.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.17+8-post-Ubuntu-1ubuntu222.04, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libopencv_ximgproc.so.406+0x940bd]  cv::ximgproc::FastLineDetectorImpl::extractSegments(std::vector<cv::Point_<int>, std::allocator<cv::Point_<int> > > const&, std::vector<SEGMENT, std::allocator<SEGMENT> >&)+0x81d
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to //core.550771)
#
# An error report file with more information is saved as:
# //hs_err_pid550771.log
[thread 553640 also had an error]
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

We managed to pinpoint location of error from information above. https://github.com/opencv/opencv_contrib/blob/db16caf6ceee76b43b94c846be276e92a43e9700/modules/ximgproc/src/fast_line_detector.cpp#L382

Unfortunately we don't have crash dump or data that caused the error available.

Steps to reproduce

We don't have necessary data available, we cannot reproduce the issue.

Additional details

We use custom build of opencv

RUN git clone --single-branch --branch 4.6.0 --depth=1 https://github.com/opencv/opencv.git && \
    git clone --single-branch --branch 4.6.0 --depth=1 https://github.com/opencv/opencv_contrib && \
    cd opencv && \
    mkdir build && \
    ANT_HOME=/usr/share/ant cmake \
        -DOPENCV_EXTRA_MODULES_PATH=../opencv_contrib/modules \
        -DBUILD_JAVA=ON \
        -DBUILD_HEADLESS=ON \
        -DBUILD_LIST=java,java_bindings_generator,core,imgcodecs,imgproc,calib3d,ximgproc \
        -S . -B build/ && \
    ANT_HOME=/usr/share/ant cmake --build build/ --parallel 6