OpenCVForUnity ArUco Sample

ArUco Sample

Open qpaozixu opened this issue 3 years ago • 15 comments

I buy the OpenCVForUnity package today and I want to know how could I detect multiple marker to obtain camera pose？ Thanks for your help!

Dec 19 '20 14:12 qpaozixu

ArUcoExample estimates the pose of several markers detected in this part of the code. https://github.com/EnoxSoftware/OpenCVForUnity/blob/master/Assets/OpenCVForUnity/Examples/ContribModules/aruco/ArUcoExample/ArUcoExample.cs#L211-L217

Dec 20 '20 14:12 EnoxSoftware

Here is a good Unity ARUco marker detection template for the HoloLens 2:

https://github.com/doughtmw/ArUcoDetectionHoloLens-Unity

Dec 21 '20 12:12 A00107408

It's only estimatePose"Single"Markers. How could I make MarkerMap and get camera pose not just get camera pose with one mark. Could support these type below? 1.MarkerMap 2.MarkerMapPoseTracker

Jan 07 '21 04:01 qpaozixu

OpenCV does not seem to support types such as MarkerMap. https://answers.opencv.org/question/147358/aruco-module-does-it-have-markermap/

Jan 07 '21 15:01 EnoxSoftware

Actually openCV is support MarkerMap. I could run with c++ on win10. But I don't really know how c# work with openCV, so I buy this plugin to slove my problem.

Enox Software [email protected] 於 2021年1月7日週四下午11:16寫道：

OpenCV does not seem to support types such as MarkerMap.

https://answers.opencv.org/question/147358/aruco-module-does-it-have-markermap/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EnoxSoftware/OpenCVForUnity/issues/87#issuecomment-756178294, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK64BZUNLQVTMVQDJX2OFALSYXF3LANCNFSM4VCK77OQ .

Jan 08 '21 14:01 qpaozixu

erica liu [email protected] 於 2021年1月8日週五下午10:13寫道：

Actually openCV is support MarkerMap. I could run with c++ on win10. But I don't really know how c# work with openCV, so I buy this plugin to slove my problem.

Enox Software [email protected] 於 2021年1月7日週四下午11:16寫道：

OpenCV does not seem to support types such as MarkerMap.

https://answers.opencv.org/question/147358/aruco-module-does-it-have-markermap/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EnoxSoftware/OpenCVForUnity/issues/87#issuecomment-756178294, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK64BZUNLQVTMVQDJX2OFALSYXF3LANCNFSM4VCK77OQ .

#include <opencv2/core.hpp> #include <opencv2/highgui.hpp> #include <opencv2/imgproc.hpp> #include <opencv2/imgcodecs.hpp> #include <opencv2/videoio.hpp> #include <opencv2/calib3d.hpp>

#include "aruco.h" using namespace aruco;

#include <glm/glm.hpp>

//Aruco參數 cv::VideoCapture TheVideoCapturer; CameraParameters TheCameraParameters; MarkerMap TheMarkerMapConfig; MarkerDetector TheMarkerDetector; vector<Marker> TheMarkers; MarkerMapPoseTracker TheMSPoseTracker; cv::Mat TheInputImage, TheInputImageCopy; float TheMarkerSize;

Start(){

TheVideoCapturer.open(0, cv::CAP_DSHOW);
		
if (!TheVideoCapturer.isOpened())
{
	std::cout << "Unable to open the device" << std::endl;
	exit(EXIT_FAILURE);
}

//先吃一張來設定寬高 
{
	TheVideoCapturer >> TheInputImage;
	if (TheInputImage.empty())
	{
		std::cout << "Unable to get frames from the device" << std::endl;
		TheVideoCapturer.release();
		exit(EXIT_FAILURE);
	}
	else
	{
		camW = TheInputImage.size().width;
		camH = TheInputImage.size().height;
	}
}
		
//讀取相機參數
TheCameraParameters.readFromXMLFile("path/to/cam-parameters.yml"));
if (TheCameraParameters.isValid())
	TheCameraParameters.resize(TheInputImage.size());
			
//讀取Markermap
TheMarkerMapConfig.clear();
TheMarkerMapConfig.readFromFile("path/to/marker-map.yml");
TheMarkerDetector.setDictionary(TheMarkerMapConfig.getDictionary());

if (TheCameraParameters.isValid() && TheMarkerMapConfig.isExpressedInMeters()) {
	TheMSPoseTracker.setParams(TheCameraParameters, TheMarkerMapConfig);
	TheMarkerSize = (float)cv::norm(TheMarkerMapConfig[0][0] - TheMarkerMapConfig[0][1]);
}
		
//設定偵測器參數
TheMarkerDetector.setDetectionMode((DetectionMode)0, 0);
TheMarkerDetector.getParameters().setCornerRefinementMethod((aruco::CornerRefinementMethod) 1);
TheMarkerDetector.getParameters().detectEnclosedMarkers(0);
TheMarkerDetector.getParameters().ThresHold = 7;
TheMarkerDetector.getParameters().trackingMinDetections = 3;
TheMarkerDetector.getParameters().error_correction_rate = 6;

}; glm::mat4 GetProjectionMatrix() { glm::mat4 proj; cv::Mat camMatrix = TheCameraParameters.CameraMatrix; float alpha = camMatrix.at(0, 0); float beta = camMatrix.at(1, 1); float cx = camMatrix.at(0, 2); float cy = camMatrix.at(1, 2); float n = 0.1f; float f = 4.f; float w = camW; float h = camH;

cv::Mat projectionMatrix = cv::Mat::zeros(4, 4, CV_32F);
projectionMatrix.at<float>(0, 0) = 2 * alpha / w;
projectionMatrix.at<float>(0, 2) = (w - 2 * cx) / w;
projectionMatrix.at<float>(1, 1) = 2 * beta / h;
projectionMatrix.at<float>(1, 2) = (-h + 2 * cy) / h;
projectionMatrix.at<float>(2, 2) = -(f + n) / (f - n);
projectionMatrix.at<float>(2, 3) = (-2.0f*f*n) / (f - n);
projectionMatrix.at<float>(3, 2) = -1;

cv::Mat glProjMatrix = cv::Mat::zeros(4, 4, CV_32F);
cv::transpose(projectionMatrix, glProjMatrix);

for (unsigned int row = 0; row < 4; ++row)
	for (unsigned int col = 0; col < 4; ++col)
		proj[row][col] = glProjMatrix.at<float>(row, col);
return proj;

}; Update(){ TheVideoCapturer >> TheInputImage; TheInputImage.copyTo(TheInputImageCopy);

TheMarkers = TheMarkerDetector.detect(TheInputImageCopy, TheCameraParameters, TheMarkerSize);
for (auto idx : TheMarkerMapConfig.getIndices(TheMarkers))
{
	TheMarkers[idx].draw(TheInputImageCopy, cv::Scalar(0, 0, 255), 1);
}



if (TheMSPoseTracker.isValid()) {
	if (TheMSPoseTracker.estimatePose(TheMarkers)) {
		cv::Mat rvec, tvec;
		TheMSPoseTracker.getRvec().copyTo(rvec);
		TheMSPoseTracker.getTvec().copyTo(tvec);

		tvec.at<float>(0, 0) = tvec.at<float>(0, 0);
		tvec.at<float>(0, 1) = tvec.at<float>(0, 1);
		tvec.at<float>(0, 2) = tvec.at<float>(0, 2);

		cv::Mat rotation, viewMatrix = cv::Mat::zeros(4, 4, CV_32F);
		Rodrigues(rvec, rotation);

		for (unsigned int row = 0; row < 3; ++row)
		{
			for (unsigned int col = 0; col < 3; ++col)
			{
				viewMatrix.at<float>(row, col) = rotation.at<float>(row, col);
			}
			viewMatrix.at<float>(row, 3) = tvec.at<float>(0, row);
		}
		viewMatrix.at<float>(3, 3) = 1.0f;

		cv::Mat cvToGl = cv::Mat::zeros(4, 4, CV_32F);
		cvToGl.at<float>(0, 0) = 1.0f;
		cvToGl.at<float>(1, 1) = -1.0f; // Invert the y axis
		cvToGl.at<float>(2, 2) = -1.0f; // invert the z axis
		cvToGl.at<float>(3, 3) = 1.0f;
		viewMatrix = cvToGl * viewMatrix;

		//glm::mat4 transform;
		//try { transform = glm::translate(transform, glm::vec3(stof(fw->xPosAdj->value()), 0, 0)); }
		//catch (exception& ex) {}
		//try { transform = glm::translate(transform, glm::vec3(0, stof(fw->yPosAdj->value()), 0)); }
		//catch (exception& ex) {}
		//try { transform = glm::translate(transform, glm::vec3(0, 0, stof(fw->zPosAdj->value()))); }
		//catch (exception& ex) {}
		//
		//transform = glm::rotate(transform, (float)glm::radians(fw->zRotAdj->value()), glm::vec3(0, 0, 1));
		//transform = glm::rotate(transform, (float)glm::radians(fw->yRotAdj->value()), glm::vec3(0, 1, 0));
		//transform = glm::rotate(transform, (float)glm::radians(fw->xRotAdj->value()), glm::vec3(1, 0, 0));
		//
		////try catch
		//transform = glm::scale(transform, glm::vec3(stof(fw->xScaAdj->value()), stof(fw->yScaAdj->value()), stof(fw->zScaAdj->value())));
		//cv::Mat cvInv = cv::Mat::zeros(4, 4, CV_32F);
		//for (unsigned int row = 0; row < 4; ++row)
		//{
		//	for (unsigned int col = 0; col < 4; ++col)
		//	{
		//		cvInv.at<float>(row, col) = transform[col][row];
		//	}
		//}
		//viewMatrix = viewMatrix * cvInv;

		cv::Mat glViewMatrix = cv::Mat::zeros(4, 4, CV_32F);
		cv::transpose(viewMatrix, glViewMatrix);

		glm::mat4 glmViewMatrix;
		for (unsigned int row = 0; row < 4; ++row)
			for (unsigned int col = 0; col < 4; ++col)
				glmViewMatrix[row][col] = glViewMatrix.at<float>(row, col);

		cameraView = glmViewMatrix;
				
		glm::mat4 glmProjectionMatrix = GetProjectionMatrix();
		cameraProj = glmProjectionMatrix;

	}
}

}

Jan 08 '21 14:01 qpaozixu

The MarkerMap class is not included in OpenCV's internal Aruco. https://github.com/opencv/opencv_contrib/search?q=MarkerMap https://github.com/opencv/opencv_contrib/tree/master/modules/aruco/src

Jan 08 '21 16:01 EnoxSoftware

MarkerMap is in this folder

Enox Software [email protected] 於 2021年1月9日週六上午12:11寫道：

The MarkerMap class is not included in OpenCV's internal Aruco. https://github.com/opencv/opencv_contrib/search?q=MarkerMap https://github.com/opencv/opencv_contrib/tree/master/modules/aruco/src

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EnoxSoftware/OpenCVForUnity/issues/87#issuecomment-756843406, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK64BZTE6VBZ7H73O4JBAQDSY4VEPANCNFSM4VCK77OQ .

Jan 09 '21 02:01 qpaozixu

How could I get marker IDs and change they to string when I detect these markers?

Thanks for your help!

erica liu [email protected] 於 2021年1月9日週六上午10:08寫道：

MarkerMap is in this folder

Enox Software [email protected] 於 2021年1月9日週六上午12:11寫道：

The MarkerMap class is not included in OpenCV's internal Aruco. https://github.com/opencv/opencv_contrib/search?q=MarkerMap https://github.com/opencv/opencv_contrib/tree/master/modules/aruco/src

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EnoxSoftware/OpenCVForUnity/issues/87#issuecomment-756843406, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK64BZTE6VBZ7H73O4JBAQDSY4VEPANCNFSM4VCK77OQ .

Jan 09 '21 02:01 qpaozixu

Oh! I got it (marker ID)

erica liu [email protected] 於 2021年1月9日週六上午10:25寫道：

How could I get marker IDs and change they to string when I detect these markers?

Thanks for your help!

erica liu [email protected] 於 2021年1月9日週六上午10:08寫道：

MarkerMap is in this folder

Enox Software [email protected] 於 2021年1月9日週六上午12:11寫道：

The MarkerMap class is not included in OpenCV's internal Aruco. https://github.com/opencv/opencv_contrib/search?q=MarkerMap https://github.com/opencv/opencv_contrib/tree/master/modules/aruco/src

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EnoxSoftware/OpenCVForUnity/issues/87#issuecomment-756843406, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK64BZTE6VBZ7H73O4JBAQDSY4VEPANCNFSM4VCK77OQ .

Jan 09 '21 02:01 qpaozixu

How could I change marker dict not use resource GridBoard-mx5-my-d marker dict?

/use aruco_mip_36h12_dict to replace GridBoard-mx5/

erica liu [email protected] 於 2021年1月9日週六上午10:28寫道：

Oh! I got it (marker ID)

erica liu [email protected] 於 2021年1月9日週六上午10:25寫道：

How could I get marker IDs and change they to string when I detect these markers?

Thanks for your help!

erica liu [email protected] 於 2021年1月9日週六上午10:08寫道：

MarkerMap is in this folder

Enox Software [email protected] 於 2021年1月9日週六上午12:11寫道：

The MarkerMap class is not included in OpenCV's internal Aruco. https://github.com/opencv/opencv_contrib/search?q=MarkerMap https://github.com/opencv/opencv_contrib/tree/master/modules/aruco/src

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/EnoxSoftware/OpenCVForUnity/issues/87#issuecomment-756843406, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK64BZTE6VBZ7H73O4JBAQDSY4VEPANCNFSM4VCK77OQ .

Jan 09 '21 04:01 qpaozixu

when I import openCVForUnity in ios project it will be crash even though I don't use any thing in this plugin.

How could I do? It seem libiconv2.dylib is broken.

Jan 13 '21 13:01 qpaozixu

Oh I got it. I need to embed opencv2.framework.

erica liu [email protected] 於 2021年1月13日週三下午9:40寫道：

when I import openCVForUnity in ios project it will be crash even though I don't use any thing in this plugin.

How could I do? It seem libiconv2.dylib is broken.

Jan 13 '21 14:01 qpaozixu

When I build project to ios it's failed but it fine in version 2019.2. How could I fix it?

failed version : unity 2019.4.17

Jan 13 '21 15:01 qpaozixu

Could you tell me the environment you tested? OpenCVForUnity version : Unity Version : Xcode version :

Jan 16 '21 09:01 EnoxSoftware

OpenCVForUnity OpenCVForUnity copied to clipboard

ArUco Sample

OpenCVForUnity
OpenCVForUnity copied to clipboard