cv2.omnidir.calibrate - Assertion error(-215) - Python

[FilipSJohansson]

System information (version)

• OpenCV => 4.5.5
• Python => 3.9
• Operating System / Platform => Windows 64 Bit
• Compiler => Visual Studio 2019

Detailed description

When trying to calibrate a wide FOV camera using opencv-contrib-python the omnidir.calibrate() results in Assertion error (-215): "OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv_contrib\modules\ccalib\src\omnidir.cpp:854: error: (-215:Assertion failed) !objectPoints.empty() && objectPoints.type() == CV_64FC3 in function 'cv::omnidir::internal::computeJacobian'"

Steps to reproduce

function inputs: inputsDump.txt

load inputs:

file = open('inputsDump.txt', 'rb')
objectpoints, imagepoints, image_size, K, xi, D, calibration_flag, subpixel_criteria = pickle.load(file)

ret, K, xi, D, rvecs, tvecs, idx = cv2.omnidir.calibrate(objectpoints, imagepoints,image_size , K, xi, D, calibration_flag, subpixel_criteria)

Complete code

import cv2
assert cv2.__version__[0] >= '3', 'The fisheye module requires opencv version >= 3.0.0'
import numpy as np
import os
import glob
import matplotlib.pyplot as plt

# Globals
image_path = '../image_path/'
#output_path = '../output_path/'
image_extension = 'jpg'
image_size = (4032  , 3024)
checkerboard_size = (10,15)
subpixel_criteria = (cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 300, 0.1)
fisheye_calibration_flags = cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC+cv2.fisheye.CALIB_CHECK_COND+cv2.fisheye.CALIB_FIX_SKEW
omnidir_calibration_flags = cv2.omnidir.CALIB_USE_GUESS + cv2.omnidir.CALIB_FIX_SKEW + cv2.omnidir.CALIB_FIX_CENTER + cv2.omnidir.CALIB_FIX_XI
draw_chessboard_marked_images = False

def getObjectAndImagePoints(imagePath, imageExtension, checkerboardSize, subpixelCriteria, drawChessboardMarkedImages):
    objp = np.zeros((1, checkerboardSize[0]*checkerboardSize[1], 3), np.float32)
    objp[0,:,:2] = np.mgrid[0:checkerboardSize[0], 0:checkerboardSize[1], ].T.reshape(-1, 2)
    _img_shape = None
    objpoints = [] # 3d point in real world space
    imgpoints = [] # 2d points in image plane.
 
    images = glob.glob(imagePath + '*.' + imageExtension)
    for fname in images:
        img = cv2.imread(fname)
        if _img_shape == None:
            _img_shape = img.shape[:2]
        else:
            assert _img_shape == img.shape[:2], "All images must share the same size."
        gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
        # Find the chess board corners
        ret, corners = cv2.findChessboardCorners(gray, checkerboardSize, cv2.CALIB_CB_ADAPTIVE_THRESH+cv2.CALIB_CB_FAST_CHECK+cv2.CALIB_CB_NORMALIZE_IMAGE)
        print('Nr of corners ' + str(corners.shape))
 
        # If found, add object points, image points (after refining them)
        if ret == True:
            corners = cv2.cornerSubPix(gray, corners, (11,11), (-1,-1), subpixelCriteria)
            corners = corners.reshape((150,2)) # This is necessary only for omnidirectional calibration
            imgpoints.append(corners)
            objpoints.append(objp)
 
            if drawChessboardMarkedImages:
                gray2 = gray
                cv2.drawChessboardCorners(gray2, checkerboardSize, corners, ret)
                cv2.imwrite(os.path.splitext(fname)[0] + '_' + 'CHESSCORNERS.png', gray2)
    return (objpoints, imgpoints)
 
 
def getOmnidirectionalCalibrationParams(objectpoints, imagepoints, calibration_flag, subpixel_criteria):
    K = np.zeros((3,3))
    D = np.zeros((1,4))
    xi = np.array([])
    idx = np.array([])
    no_images = len(imagepoints)
    no_points = len(imagepoints[0])
    
    #objectpoints = np.reshape(objectpoints, (no_images, 1, no_points, 3))
    objectpoints = np.array(objectpoints,dtype=np.float64).reshape(no_images, 1, no_points, 3)
    #imagepoints = np.reshape(imagepoints, (no_images, 1, no_points, 2))
    imagepoints = np.array(imagepoints,dtype=np.float64).reshape(no_images, 1, no_points, 2)
    print('shape of objectpoints ' + str(objectpoints.shape))
    print('shape of imagepoints ' + str(imagepoints.shape))
    
    print("size of objectpoints " + str(objectpoints.size))
    print("size of imagepoints " + str(imagepoints.size))
    ret, K, xi, D, rvecs, tvecs, idx = cv2.omnidir.calibrate(objectpoints, imagepoints,image_size , K, xi, D, calibration_flag, subpixel_criteria)
    
    return ret, K, xi, D, rvecs, tvecs, idx
 

# Find camera calibration parameters
objectpoints, imagepoints = getObjectAndImagePoints(image_path, image_extension, checkerboard_size, subpixel_criteria, draw_chessboard_marked_images)
         
ret, K, xi, D, rvecs, tvecs, idx = getOmnidirectionalCalibrationParams(objectpoints, imagepoints, omnidir_calibration_flags, subpixel_criteria)
 
print("Found " + str(len(imagepoints)) + " valid images for calibration")
print("K = np.array(" + str(K.tolist()) + ")")
print("D = np.array(" + str(D.tolist()) + ")")
print("xi = np.array(" + str(xi.tolist()) + ")")

Issue submission checklist

• [x] I report the issue, it's not a question
• [x] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• [x] I updated to the latest OpenCV version and the issue is still there
• [x] There is reproducer code and related data files: videos, images, onnx, etc

[alalek]

Please provide complete minimal reproducer including necessary input data (nobody has access to your disk C:).

"minimal" means that we need the "broken" call only. You could dump passed inputs of this call into a file and load them in reproducer code (e.g, using cv::FileStorage or Python pickle).

[FilipSJohansson]

I have now included a input dump aswell.

[alalek]

Thank you for update! I can reproduce and confirm the problem.

Unfortunately this module doesn't have any tests. Probably its code is permanently broken.

---

Data is passed properly:

# InputArrayOfArrays: empty()=false kind=0x00050000 flags=0x01050000 total(-1)=8 dims(-1)=1 size(-1)=8x1 type(0)=CV_64FC3 dims(0)=2 size(0)=150x1
print(cv.utils.dumpInputArrayOfArrays(objectpoints))
# InputArrayOfArrays: empty()=false kind=0x00050000 flags=0x01050000 total(-1)=8 dims(-1)=1 size(-1)=8x1 type(0)=CV_64FC2 dims(0)=2 size(0)=150x1
print(cv.utils.dumpInputArrayOfArrays(imagepoints))

---

Algorithm has NaNs here:

• reprojectError: https://github.com/opencv/opencv_contrib/blob/4.5.5/modules/ccalib/src/omnidir.cpp#L680
• and here: https://github.com/opencv/opencv_contrib/blob/4.5.5/modules/ccalib/src/omnidir.cpp#L712

which causes empty _idx and empty loop here.

Finally it fails !objectPoints.empty() check and triggers the observed error.

---

gamma is NaN here due to negative sqrt() argument.

[alalek]

Similar problems are here: #2621 #804

[FilipSJohansson]

I guess this problem is related to release 4.5.5, and not due to python/images?

[gred0216]

I found a workaround to the assertion issue somewhere else. By expanding the dimensions of the imagepoints and objpoints array, they can be passed into the calibrate functions with no error.

objpoints_expand = np.expand_dims(np.asarray(objpoints), -2)

Here are the dimensions and type of the array.

print(objpoints[0].shape) # (88, 3) print(objpoints_expand.shape) # (7, 88, 1, 3) print(oe.dtype) # dtype('float64')

Hope this would help someone who has the same issue.