Probably a bug in aruco::getBoardObjectAndImagePoints

[PeterHomberg]

System information ()

• OpenCV => 4.2.5-Pre
• Operating System / Platform => Windows 10 64 Bit
• Compiler => Visual Studio 2019

Detailed description

In my code sample below everything runs fine until I'm calling the function aruco::getBoardObjectAndImagePoints(m_board, charucoCorners, charucoIds, objPoints, imgPoints); Then an exception is thrown . The exception originates from the line 1116 in aruco.cpp:

imgPnts.push_back(detectedCorners.getMat(i).ptr< Point2f >(0)[p]);

The exception is thrown in the function: Mat InputArray::getMat(int i) const at line 43 in matrix_wrap.cpp : CV_Assert( i < 0 ); it seems to me that you can't call getMat with an index >=0

Steps to reproduce

std::vector<cv::Point2f> charucoCorners;
std::vector<int> charucoIds;
cv::aruco::interpolateCornersCharuco(markerCorners, markerIds, image, m_board, charucoCorners, charucoIds, cameraMatrix, distCoeffs);
    // C++ code example

// if at least one charuco corner detected
if (charucoIds.size() > 0) {
			cv::Scalar color = cv::Scalar(255, 0, 0);
			cv::aruco::drawDetectedCornersCharuco(imageCopy, charucoCorners, charucoIds, color);
			// cv::aruco::estimatePoseCharucoBoard(charucoCorners, charucoIds, board, cameraMatrix, distCoeffs, rvec, tvec);
			bool valid = cv::aruco::estimatePoseCharucoBoard(charucoCorners, charucoIds, m_board, cameraMatrix, distCoeffs,     rvec, tvec);
			// if charuco pose is valid
			if (valid)
			{
				cv::aruco::drawAxis(imageCopy, cameraMatrix, distCoeffs, rvec, tvec, 40.0f);
				vector<cv::Point3f> objPoints;
				vector<cv::Point2f>imgPoints;
				aruco::getBoardObjectAndImagePoints(m_board, charucoCorners, charucoIds, objPoints, imgPoints);
				success = true;
			}
		}

[alalek]

Complete minimal reproducer is required (including input data or it's "zero" replacement if applicable)

[PeterHomberg]

Image File for pose estimation:

minimal reproducer code:

#include <opencv2/core.hpp> #include <opencv2/aruco/charuco.hpp> // header file for opencv Aruco module #include <opencv2/imgcodecs.hpp> #include <opencv2/highgui.hpp> #include <opencv2/calib3d.hpp> #include <opencv2/imgproc.hpp> #include #include #include using namespace cv; using namespace std;

int main(int argc, char** argv) {

Mat image;
image = imread("CharucoPoseT20001.jpg", IMREAD_COLOR);

Mat cameraMatrix = (Mat_<double>(3, 3) << 2.7430930170111942e+03, 0., 9.4282063272949267e+02, 0.,
    2.7408493422768938e+03, 6.2128986255109442e+02, 0., 0., 1.);
Mat distCoeffs = (Mat_<double>(1, 5) << -8.9613623867351555e-02, 1.0316024796119334e+00,
    5.8820803241661074e-04, -2.0243284370249858e-03,
    -8.0793072773267205e-01);
Mat rvec;
Mat tvec;
Ptr<aruco::CharucoBoard> m_board = aruco::CharucoBoard::create(8, 5, 35.0, 26.0, aruco::getPredefinedDictionary(aruco::DICT_6X6_50));
Ptr<aruco::DetectorParameters> params = aruco::DetectorParameters::create();
vector<int> markerIds;
vector<vector<Point2f> > markerCorners;
//Mat imageCopy;
//image.copyTo(imageCopy);
aruco::detectMarkers(image, m_board->dictionary, markerCorners, markerIds, params);

if (markerIds.size() > 0) {
	//cv::aruco::drawDetectedMarkers(imageCopy, markerCorners, markerIds);
	std::vector<cv::Point2f> charucoCorners;
	std::vector<int> charucoIds;
	cv::aruco::interpolateCornersCharuco(markerCorners, markerIds, image, m_board, charucoCorners, charucoIds, cameraMatrix, distCoeffs);
	// if at least one charuco corner detected
	if (charucoIds.size() > 0) {
		//cv::Scalar color = cv::Scalar(255, 0, 0);
		//cv::aruco::drawDetectedCornersCharuco(imageCopy, charucoCorners, charucoIds, color);
		// cv::aruco::estimatePoseCharucoBoard(charucoCorners, charucoIds, board, cameraMatrix, distCoeffs, rvec, tvec);
		bool valid = cv::aruco::estimatePoseCharucoBoard(charucoCorners, charucoIds, m_board, cameraMatrix, distCoeffs, rvec, tvec);
		// if charuco pose is valid
		if (valid)
		{
			//cv::aruco::drawAxis(imageCopy, cameraMatrix, distCoeffs, rvec, tvec, 40.0f);
			vector<cv::Point3f> objPoints;
			vector<cv::Point2f>imgPoints;
			aruco::getBoardObjectAndImagePoints(m_board, charucoCorners, charucoIds, objPoints, imgPoints);
			
		}
	}
}

}

[thom9258]

Hey @PeterHomberg, im not one of the developers but i ran into the same problem, i have rewritten the function from and removed the opencv input and output array definitions, and modified the code to work like i think its supposed to, maybe the changed function is useful to you? please let me know after you have tried it as it might help me finding bugs too..

PS* if this is against any license or terms of use then it was not the intention of breaking it, and the code and everything belongs to the original owners/creators. I simply wanted to help :)

/*
Fix based on the original upload of the function from 26082015
written by thom0258 
date: 22042021
*/
void 
FIX_26082015_getBoardObjectAndImagePoints(const cv::Ptr<cv::aruco::Board> board, 
                                          std::vector<cv::Point2f> _corners,
                                          std::vector<int>         _ids, 
                                          std::vector<cv::Point3f> *objPoints, 
                                          std::vector<cv::Point2f> *imgPoints) 
{
    CV_Assert(board->ids.size() == board->objPoints.size());
    CV_Assert(_ids.size() == _corners.size());
    int nDetectedMarkers = (int)_ids.size();
    // look for detected markers that belong to the board and get their information
    for(int i = 0; i < nDetectedMarkers; i++) {
        int currentId = _ids[i];
        for(unsigned int j = 0; j < board->ids.size(); j++) {
            if(currentId == board->ids[j]) {
                for(int p = 0; p < 4; p++) {
                    objPoints->push_back(board->objPoints[j][p]);
                    imgPoints->push_back(_corners[p]);
                }
            }
        }
    }
}

[PeterHomberg]

Hello thom9258,

Thank you very much for your replay and the suggestion. I found that this is a similar approach as in the function estimatePoseCharucoBoard and I will definitely use it.

regards

Peter

[thom9258]

@PeterHomberg Hey i just found out you closed the bug report,

I was thinking that maybe you should reopen it so that the bug might get fixed by the actual authors so no one else runs into the bug like we did :)

[PeterHomberg]

I re-opened this issue, so that somebody from the authors can fix this problem.

[AleksandrPanov]

checked, problem is reproduced

[AleksandrPanov]

Aruco module was moved from contrib to main repo, getBoardObjectAndImagePoints() has been replaced by matchImagePoints(). I have updated your reproducer to a new API. Now the problem is not reproduced. main.cpp.txt

[AleksandrPanov]