camera calibration results do not match with python pendant

[maGitty]

Description

As visualized below on images, the results of the camera calibration don't seem to be correct. I closely followed some python code that I wrote for testing purpose where I obtained good results. However, there were some points from gocv that differed quite a lot from pythons opencv library. E.g. the corners (for comparison - imgpoints collected during extraction is a list of all corners Matrices) Mat in python was 2-dimensional, which makes sense, as every single point has a x and a y coordinate. For this, the gocv corners computed by FindChessboardCornersSB were only 1-dimensional. Other from that, I built the objpoints (3D coordinates of the corners in real world) according to my python pendant to get the same matrix.

Note: I'm currently working with dev branch, as CalibrateCamera is not available yet on the release branch.

Steps to Reproduce

Below are some example pictures I took with my webcam. Below the pictures is a hopefully MWE to verify my results. The first code block is used to capture a frame and dump it to disk, the second one for computing the corner positions and to acquire the needed parameters/matrices to undistort.

original converted to grayscale: found corners: one of the undistorted frames:

Frames were captured like following:

cam, _ := gocv.VideoCaptureDevice(0)

img := gocv.NewMat()
gray := gocv.NewMat()
corners := gocv.NewMat()
size := image.Point{X:9,Y:6}

window := gocv.NewWindow("win")

for ok := cam.Read(&img); ok; ok = cam.Read(&img) {
	gocv.CvtColor(img, &gray, gocv.ColorBGRToGray)
	if gocv.FindChessboardCornersSB(gray,size, &corners, 0) {
		now := time.Now()
		gocv.IMWrite("frames/" + strconv.FormatInt(now.Unix(), 10) + ".jpg", gray)
		gocv.DrawChessboardCorners(&gray, size, corners, true)
	}
	window.IMShow(img)

	ch := window.WaitKey(33)
	if ch == 27 {
		break
	}
}

And the following code was used to calibrate the camera:

size := image.Point{X:9,Y:6}
winSize := image.Point{X:5,Y:5}
zeroZone := image.Point{X:-1,Y:-1}
criteria := gocv.NewTermCriteria(gocv.MaxIter+gocv.EPS, 30, 0.1)
imgpoints := gocv.NewPoints2fVector()
corners := gocv.NewMat()

frames, err := ioutil.ReadDir("frames/")
if err != nil {
	log.Fatal(err)
}

/************************
 EXTRACT CORNER POSITIONS
 ************************/
foundCorners := 0
for _, framePath := range frames {
	gray := gocv.IMRead(config.ImagePath + framePath.Name(), gocv.IMReadAnyColor)
	if gray.Empty() {
		log.Fatalln("Failed to read image:", framePath)
	}
	win.IMShow(gray)
	ch := win.WaitKey(33)
	if ch == 27 {
		break
	}

	if found := gocv.FindChessboardCornersSB(gray, size, &corners, 0); found {
		foundCorners++
		gocv.DrawChessboardCorners(&gray, size, corners, true)
		gocv.CornerSubPix(gray, &corners, winSize, zeroZone, criteria)
		// 2D points in image
		imgpoints.Append(gocv.NewPoint2fVectorFromMat(corners))

		winCalib.IMShow(gray)
		ch := winCalib.WaitKey(33)
		if ch == 27 {
			break
		}
	} else {
		log.Println("Could not find corners in", framePath.Name())
	}
}

/***************************
 COMPUTE CORRECTION MATRICES
 ***************************/
// Build Matrix of 3D points in real world
scaler := 2.3 // size of one square, I think in cm?
objpointVec := gocv.NewPoint3fVector()
objpoints := gocv.NewPoints3fVector()
for i := 0; i < config.Chessboard.Size.X; i++ {
	for j := 0; j < config.Chessboard.Size.Y; j++ {
		point := gocv.NewPoint3f(scaler*float32(j+1), scaler*float32(j), 0)
		objpointVec.Append(point)
	}
}
for n := 0; n < foundCorners; n++ {
	objpoints.Append(objpointVec)
}

cmat := gocv.NewMat()
distCoeffs := gocv.Zeros(1,5,gocv.MatTypeCV64F)
rvecs := gocv.Zeros(7,1,gocv.MatTypeCV64FC3)
tvecs := gocv.Zeros(7,1,gocv.MatTypeCV64FC3)

gocv.CalibrateCamera(objpoints, imgpoints, size, &cmat, &distCoeffs, &rvecs, &tvecs, gocv.CalibFixIntrinsic)
imgSize := image.Point{
	X: 640,
	Y: 480,
}
newMtx, rect := gocv.GetOptimalNewCameraMatrixWithParams(cmat, distCoeffs, imgSize, 0, imgSize, false)

/****************************
 UNDISTORT FRAMES FOR TESTING
 ****************************/
for _, framePath := range frames {
	gray := gocv.IMRead(config.ImagePath + framePath.Name(), gocv.IMReadAnyColor)
	if gray.Empty() {
		log.Fatalln("Failed to read image:", framePath)
	}
	win.IMShow(gray)
	ch := win.WaitKey(33)
	if ch == 27 {
		break
	}

	undistorted := gocv.NewMat()
	gocv.Undistort(gray, &undistorted, cmat, distCoeffs, newMtx)
	gocv.IMWrite("calibrated/" + framePath.Name(), undistorted)
}

Your Environment

• Operating System and version: Fedora 32
• OpenCV version used: 4.5.1.48
• How did you install OpenCV? building from source
• GoCV version used: 0.28.0 on branch dev
• Go version: go1.16 linux/amd64