Meshroom
Meshroom copied to clipboard
alicevision
[question] correspond the xy coordinates in the pointcloud to image
Describe the problem Hello. I'm interacting with meshroom via scripts. The problem I'm trying to solve is seemingly simple: in the UI, you provide the ability to overlay an image that was used during SFM with its corresponding points in the pointcloud. I need to do exactly this, but in a script. So far, I've used SFMTransform - which as far as I can tell applies the corresponding transforms to the base sfm.abc pointcloud to reconstruct a new one. What I'm unable to do though is correspond the 2D projection of these points (without depth) with the image itself. For a bit more context, I have polygons in the image that I need to apply to the 2D matrix in order to isolate particular points from the image's perspective.
To summarize: given an image and the pointcloud, I need to be able to correspond the xy coordinates in the pointcloud with their xy coordinates in the image - exactly what "sync with image selection + image overlay" do in the UI (see the attached image below).
Any help would be appreciated tremendously. Thank you.
Do you want to re-project the 3d points in the image, like e.g. for computing the reprojection error (distance between a feature detected in the image and the projection of its corresponding 3d point in the image)? If so, I don't think you'll be able to do it only from the existing nodes. SFMTransform can bring the point cloud in the camera reference system but then you still need to project the points in the image (i.e. apply the matrix K) for which there is no node. You can either write python code to do that, or use the underlying alicevision library to solve the problem in probably 5-6 lines of code. It will be similar to this: https://github.com/alicevision/AliceVision/blob/0f7c953929756961e79b77e9ae2cf54c8df3d2d1/src/aliceVision/sfm/sfmStatistics.cpp#L36-L52 but instead of doing it for all the views you just do it for a specific view.
I'll poke around at the code sample you sent over and see what I can cook up, but I have another question based on your answer. So, there are two nodes that I'm using which output pointcloud files - the first is incrementalSfm which is what's used in the standard pipeline and the second is SFMTransform. I have little knowledge on SFM terminology but I see files stored under the node which reconstructs the dense scene that contain matrices K, Rt (R & t?) and P. If I'm understanding correctly, SFMTransform applies the rotation matrix but not K which appears to be some kind of scaling factor for the x & y values. Is this understanding correct?
yes SFMTransform only applies rigid R t rototranslations. K is a property of the (each) camera.
I see, so my approach so far is as follows:
- incrementalSFM to get the general pointcloud
- single camera SFMTransform to scale/rotate/translate the pointcloud to a particular camera's view that corresponds to an image
- apply K on the resultant nx3 pointcloud matrix as follows: dot(pointcloud-matrix, K)
- shift points per the intrinsic principal point
- overlay image and visually inspect
It seems like there's something wrong with my process though, as the image and pointclouds still don't appear to line up properly (see the attached image for reference)
I really appreciate your help, if this is out of the scope for the issues section please let me know.
is that the original image or an undistorted one? What you do seems correct but you need to deal with distortion too. Either you apply distortion to the points when you project them or you undistort the original image according to the intrinsics.
Neither of these are distorted. That could be the issue. I see the distortion vector [1x3] in the camera's intrinsics. If I want to apply this to the pointcloud directly, are there any changes that need to be made to the vector (ie: is this meant to undistort the original image rather than distort the pointcloud). My current approach is to perform element-wise multiplication on the dot product between K and the pointcloud but that doesn't seem to be yielding reasonable results.
the distortion vector is used to distort the point cloud projected on the camera image plane (note: before applying K, here is the explanation of the projection equation http://robots.stanford.edu/cs223b04/JeanYvesCalib/htmls/parameters.html). This is a non linear operation as it operates on the radial distance of the point wrt the center of the image.
The code in alicevision to perform the distortion with 3 radial parameter is here https://github.com/alicevision/AliceVision/blob/156c6321d58ce290ef98ee9630c92111fc6f4313/src/aliceVision/camera/DistortionRadial.hpp#L231 (note: it takes as input a 2d point in normalized coordinates, i.e. a 3D point whose coordinates are divided by z-coordinate and only the inhomogeneous part (x/z, y/z) is considered). So overall it goes: 3D point --> normalized 2d point (division by z) --> apply distortion --> apply K
That worked perfectly! Thank you so much I appreciate the help tremendously. I have just one last thing to ask. The pointcloud in the GUI attached - is this just the raw output from the StructureFromMotion node plotted, or are there any operations happening on that point cloud before it is displayed?
no that's just the output of the SFM node, no postprocessing or other operations are applied to the pointcloud.
Hi @stevebottos, I tried everything and still having the same issue. It seems that getting a correct 2D projection from Meshroom outputs is challeging for everyone. Could you please share the working code here? Or is there anyone else or someone from Meshroom team that can solve this issue for us? Thanks.
