Runtime issue in LAPACK with calibrate_camera

[treyfortmuller]

1. Operating system: NixOS
2. The way you installed OpenCV: package, official binary distribution, manual compilation, etc: Nix package manager
3. OpenCV version 4.9
4. rustc version (rustc --version) rustc 1.80.0 (051478957 2024-07-21)

Really appreciate the library, thanks for the work on this!

Experiencing a runtime issue on a call to calibrate_camera - I basically cloned the python example provided in the opencv docs as closely as possible. I'm using the sample chessboard images provided in the samples/ dir of opencv. Here's a snippet of this program:

   let image_size = match calibration_image_size {
        None => bail!("Calibration image size was missing for camera calibration"),
        Some(size) => size,
    };

    let calibration_termination = cv::TermCriteria {
        typ: cv::TermCriteria_EPS + cv::TermCriteria_MAX_ITER,
        max_count: 30,
        epsilon: std::f64::EPSILON,
    };

    for i in 0..image_points.len() {
        // There must be a 3D point corresponding to each 2D point
        let image_point_frame = image_points.get(i).unwrap();
        let object_point_frame = object_points.get(i).unwrap();

    // Data to be populated by the calibration procedure
    let mut camera_matrix = cv::Mat::default(); // Should be 3x3
    let mut dist_coeffs = cv::Mat::default(); // 4, 5, 8, 12, or 14 elements...?

    // Estimated camera extrinsics for each frame
    let mut rvecs = cv::Vector::<cv::Mat>::new();
    let mut tvecs = cv::Vector::<cv::Mat>::new();

    // This explodes...
    let rms = opencv::calib3d::calibrate_camera(
        &object_points,
        &image_points,
        image_size,
        &mut camera_matrix,
        &mut dist_coeffs,
        &mut rvecs,
        &mut tvecs,
        0,
        calibration_termination,
    );

    println!("RMS reprojection error: {:#?}", rms);

No issues at all with corner detection, or computing subpixel refinements for the corner positions. Then when we run calibrate_camera it returns a Result<f64, Error> which is Ok(NaN) and we see a bunch of errors printed to the console regarding DGESDD, which is a Singular Value Decomposition routine in LAPACK.

 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value

RMS reprojection error: Ok(
    NaN,
)

I see two of these printed to the console per calibration frame. I've tried

• version 0.88.8 and 0.92.3 of this lib
• turning on all default features, and only including the features/modules I need including calib3d and ccalib
• various overloads of calibrate_camera available in the API

Here's the docs for dgesdd in LAPACK, but these were not super illuminating for me.

Wondering if anyone's ever gotten camera calibration working through this lib, I appreciate the help!

[treyfortmuller]

@twistedfall any chance you've given these workflows a shot before? I was able to get some results (albeit very poor calibrations) using USE_LU and USE_QR flags to calibrate_camera. The recommended defaults use an SVD decomp which is the failing LAPACK call.

[twistedfall]

I'm not a big expert on using camera calibration from OpenCV (read, no experience whatsoever), but I'll try to take a look at the Python example and compare it with your port to see what could be different

[treyfortmuller]

Really appreciate you taking a look. My worry is the InputArray, OutputArray, and InputOutputArray types I chose were somehow the wrong analagous types. They're so heavily templated in C++ its hard to tell exactly what object is expected.

I'm fairly confidence I don't have a poorly posed optimization problem, but rather its an issue at the API layer given the whole "illegal value" error message.

[twistedfall]

I'm not sure what particular problem is in the example that you provided, I can't run it as is unfortunately. But I've created a more or less verbatim port of the example that the tutorial you referenced which works exactly like the Python code on my machine. Can you please take a look to see if it helps you track down the issue?

https://github.com/twistedfall/opencv-rust/blob/c385e7f0ad3fd93f1aaae8d11059a906acb21238/examples/camera_calibration.rs

[treyfortmuller]

Hello again! I gave your example a shot - here you're running the calibration only with the image points and object points from one image. We're meant to collect the image points and object points for all the images and run the calibration on the total set of collected points.

I've adapted your example slightly with that fix and I'm able to reproduce the issue I was running into. (apologies for adding the camino and anyhow dependencies here, thats just an artifact of my testing infra).

//! Port of code from the tutorial at: https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html

use anyhow::Result;
use camino::{Utf8Path, Utf8PathBuf};
use opencv::core::{
    Point2f, Point3f, Size, TermCriteria, TermCriteria_EPS, TermCriteria_MAX_ITER, Vector,
};
use opencv::prelude::*;
use opencv::{calib3d, highgui, imgcodecs, imgproc};

pub fn calibrate(
    image_dir: Utf8PathBuf,
    extension: String,
    rig_rows: u8,
    rig_cols: u8,
) -> Result<()> {
    let pattern_size = Size::new(rig_rows as i32, rig_cols as i32);

    // termination criteria
    let criteria = TermCriteria {
        typ: TermCriteria_EPS + TermCriteria_MAX_ITER,
        max_count: 30,
        epsilon: 0.001,
    };

    let objp_len = rig_cols * rig_rows;
    let objp = Vector::from_iter(
        (0..objp_len).map(|i| Point3f::new((i % rig_rows) as f32, (i / rig_rows) as f32, 0.)),
    );

    println!("obj point len: {:#?}", objp.len());

    let images = Utf8Path::read_dir_utf8(&image_dir)?
        .into_iter()
        .flatten()
        .filter(|entry| {
            entry
                .path()
                .extension()
                .map_or(false, |ext| ext == extension)
        });

    // Arrays to store object points and image points from all the images.
    let mut objpoints = Vector::<Vector<Point3f>>::new(); // 3d point in real world space
    let mut imgpoints = Vector::<Vector<Point2f>>::new(); // 2d points in image plane.
    let mut imgsize = Size::new(0, 0);

    for image in images {
        println!("image: {:?}", image.path());

        let mut img = imgcodecs::imread_def(image.path().as_ref())?;
        let mut gray = Mat::default();
        imgproc::cvt_color_def(&img, &mut gray, imgproc::COLOR_BGR2GRAY)?;

        let mut corners = Vector::<Point2f>::default();
        let ret = calib3d::find_chessboard_corners_def(&gray, pattern_size, &mut corners)?;

        println!("Chessboard found? {ret}");

        if ret {
            objpoints.push(objp.clone());

            imgproc::corner_sub_pix(
                &gray,
                &mut corners,
                Size::new(11, 11),
                Size::new(-1, -1),
                criteria,
            )?;

            // Draw and display the corners
            calib3d::draw_chessboard_corners(&mut img, pattern_size, &corners, ret)?;
            highgui::imshow("Source", &img)?;
            highgui::wait_key_def()?;

            imgpoints.push(corners);

            imgsize = gray.size()?;
        }
    }

    println!("len image points: {:?}", imgpoints.len());
    println!("len object points: {:?}", objpoints.len());

    // Calibration
    let mut mtx = Mat::default();
    let mut dist = Mat::default();
    let mut rvecs = Vector::<Mat>::new();
    let mut tvecs = Vector::<Mat>::new();
    let rms = calib3d::calibrate_camera_def(
        &objpoints, &imgpoints, imgsize, &mut mtx, &mut dist, &mut rvecs, &mut tvecs,
    )?;

    println!("RMS: {:.5}", rms);

    // // Using cv.undistort()
    // let mut dst_undistort = Mat::default();

    // calib3d::undistort_def(&img, &mut dst_undistort, &mtx, &dist)?;
    // highgui::imshow("Result using undistort", &dst_undistort)?;

    // // Using remapping
    // let mut mapx = Mat::default();
    // let mut mapy = Mat::default();
    // calib3d::init_undistort_rectify_map(
    //     &mtx,
    //     &dist,
    //     &no_array(),
    //     &no_array(),
    //     img.size()?,
    //     f32::opencv_type(),
    //     &mut mapx,
    //     &mut mapy,
    // )?;
    // let mut dst_remap = Mat::default();
    // imgproc::remap_def(&img, &mut dst_remap, &mapx, &mapy, imgproc::INTER_LINEAR)?;
    // highgui::imshow("Result using remap", &dst_undistort)?;

    // highgui::wait_key_def()?;
    highgui::destroy_all_windows()?;
    Ok(())
}

The output of this program is:

obj point len: 42
image: "./test-data/calib/left06.jpg"
Chessboard found? true
image: "./test-data/calib/left09.jpg"
Chessboard found? false
image: "./test-data/calib/left11.jpg"
Chessboard found? false
image: "./test-data/calib/left14.jpg"
Chessboard found? true
image: "./test-data/calib/left07.jpg"
Chessboard found? true
image: "./test-data/calib/left05.jpg"
Chessboard found? true
image: "./test-data/calib/left13.jpg"
Chessboard found? true
image: "./test-data/calib/left01.jpg"
Chessboard found? true
image: "./test-data/calib/left03.jpg"
Chessboard found? true
image: "./test-data/calib/left04.jpg"
Chessboard found? true
image: "./test-data/calib/left08.jpg"
Chessboard found? true
image: "./test-data/calib/left12.jpg"
Chessboard found? true
image: "./test-data/calib/left02.jpg"
Chessboard found? true
len image points: 11
len object points: 11
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
 ** On entry to DGESDD parameter number  8 had an illegal value
RMS: NaN

The intrinsics matrix and distortion coefficients are also NaN.

[twistedfall]

I'm running this function with the following parameters:

calibrate("/var/tmp/1", "jpg", 7, 6);

The /var/tmp/1 directory contains all files matching "left*.jpg" from the https://github.com/opencv/opencv/tree/4.x/samples/data

ls /var/tmp/1
left01.jpg  left02.jpg  left03.jpg  left04.jpg  left05.jpg  left06.jpg  left07.jpg  left08.jpg  left09.jpg  left11.jpg  left12.jpg  left13.jpg  left14.jpg  left.jpg

I'm getting the following output:

obj point len: 42
image: "/var/tmp/1/left.jpg"
[ INFO:[email protected]] global registry_parallel.impl.hpp:96 ParallelBackendRegistry core(parallel): Enabled backends(3, sorted by priority): ONETBB(1000); TBB(990); OPENMP(980)
Chessboard found? false
image: "/var/tmp/1/left05.jpg"
Chessboard found? true
[ INFO:[email protected]] global registry.impl.hpp:114 UIBackendRegistry UI: Enabled backends(3, sorted by priority): GTK(1000); GTK3(990); GTK2(980) + BUILTIN(QT5)
QSettings::value: Empty key passed
QSettings::value: Empty key passed
image: "/var/tmp/1/left11.jpg"
Chessboard found? false
image: "/var/tmp/1/left12.jpg"
Chessboard found? true
image: "/var/tmp/1/left07.jpg"
Chessboard found? true
image: "/var/tmp/1/left02.jpg"
Chessboard found? true
image: "/var/tmp/1/left01.jpg"
Chessboard found? true
image: "/var/tmp/1/left03.jpg"
Chessboard found? true
image: "/var/tmp/1/left04.jpg"
Chessboard found? true
image: "/var/tmp/1/left08.jpg"
Chessboard found? true
image: "/var/tmp/1/left14.jpg"
Chessboard found? true
image: "/var/tmp/1/left13.jpg"
Chessboard found? true
image: "/var/tmp/1/left06.jpg"
Chessboard found? true
image: "/var/tmp/1/left09.jpg"
Chessboard found? false
len image points: 11
len object points: 11
RMS: 0.15537

So I'm either reproducing it wrong, or it might have to do with the particular way OpenCV was built.

Edit: just checked, I'm actually building it with LAPACK=OFF so that would probably explain it.

[twistedfall]

After building it with LAPACK enabled the output somewhat changes, no chessboard is found for "left05.jpg" thus less image and object points, but still no error:

obj point len: 42
image: "/var/tmp/1/left.jpg"
[ INFO:[email protected]] global registry_parallel.impl.hpp:96 ParallelBackendRegistry core(parallel): Enabled backends(2, sorted by priority): TBB(1000); OPENMP(990)
[ INFO:[email protected]] global parallel_for.tbb.hpp:54 ParallelForBackend Initializing TBB parallel backend: TBB_INTERFACE_VERSION=12090
[ INFO:[email protected]] global parallel.cpp:77 createParallelForAPI core(parallel): using backend: TBB (priority=1000)
Chessboard found? false
image: "/var/tmp/1/left05.jpg"
Chessboard found? false
image: "/var/tmp/1/left11.jpg"
Chessboard found? false
image: "/var/tmp/1/left12.jpg"
Chessboard found? true
[ INFO:[email protected]] global registry.impl.hpp:114 UIBackendRegistry UI: Enabled backends(3, sorted by priority): GTK(1000); GTK3(990); GTK2(980) + BUILTIN(QT6)
QSettings::value: Empty key passed
init done
opengl support available
image: "/var/tmp/1/left07.jpg"
Chessboard found? true
image: "/var/tmp/1/left02.jpg"
Chessboard found? true
image: "/var/tmp/1/left01.jpg"
Chessboard found? true
image: "/var/tmp/1/left03.jpg"
Chessboard found? true
image: "/var/tmp/1/left04.jpg"
Chessboard found? true
image: "/var/tmp/1/left08.jpg"
Chessboard found? true
image: "/var/tmp/1/left14.jpg"
Chessboard found? true
image: "/var/tmp/1/left13.jpg"
Chessboard found? true
image: "/var/tmp/1/left06.jpg"
Chessboard found? true
image: "/var/tmp/1/left09.jpg"
Chessboard found? false
len image points: 10
len object points: 10
RMS: 0.15771

The difference might also be because of the different OpenCV version, this output is from 4.10.0, the one before is from 4.9.0

[twistedfall]

I'm going to close this issue, if you still have any updates please feel free to reopen!