Problems with DrawAxis function

[tas-pats]

System Information

OpenCV version: opencv-python 4.7.0.72 and opencv-contrib-python 4.10.0.84 OS: Windows 11 Python version: 3.11.3

Detailed description

I am detecting the ChArUco boards using the OpenCV. Once I have detected and estimated its pose, I am trying to plot the axis using the drawFrameAxes function; however, I can see different axis plotted just by changing their lengths even and everything remains the same. The included photo has two axis of different length plotted:

Steps to reproduce

original image:

import os
import numpy as np
import matplotlib.pyplot as plt
import cv2

# camera params after calibration
mtx = np.array([[3341.0930071 ,    0.        , 2020.04757797],
   [   0.        , 3332.54238818, 1108.10670945],
   [   0.        ,    0.        ,    1.        ]])
   
dist = np.array([-0.10474984,  0.04565679,  0.00948721,  0.        ,  0.        ])

img_rgb = cv2.imread("some img")
img_rgb = cv2.cvtColor(img_rgb, cv2.COLOR_RGB2BGR)
img = cv2.cvtColor(img_rgb, cv2.COLOR_RGB2GRAY)

# board info
dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_100)
board = cv2.aruco.CharucoBoard((7, 5), 0.055, 0.041, dictionary)

params = cv2.aruco.DetectorParameters()
detector = cv2.aruco.ArucoDetector(dictionary, params)

# detect markers and charuco corners
marker_corners, marker_ids, rejectedCandidates = detector.detectMarkers(img)

ret, charucoCorners, charucoIds = cv2.aruco.interpolateCornersCharuco(marker_corners, marker_ids, img, board)

ret, rvec, tvec = cv2.aruco.estimatePoseCharucoBoard(charucoCorners, charucoIds, board, mtx, dist, None, None)

# start plotting
axis_size = 0.1
result_short = cv2.drawFrameAxes(np.copy(img_rgb), mtx, dist, rvec, tvec, axis_size)

axis_size = 5
result_long = cv2.drawFrameAxes(np.copy(img_rgb), mtx, dist, rvec, tvec, axis_size)

# plot
plt.subplot(1,2,1)
plt.imshow(result_short)
plt.subplot(1,2,2)
plt.imshow(result_long)

plt.show()

result_both = cv2.drawFrameAxes(np.copy(img_rgb), mtx, dist, rvec, tvec, 0.2)
result_both = cv2.drawFrameAxes(result_both, mtx, dist, rvec, tvec, 2)

### Issue submission checklist

- [X] I report the issue, it's not a question
- [X] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- [X] I updated to the latest OpenCV version and the issue is still there
- [X] There is reproducer code and related data files (videos, images, onnx, etc)

[A-Choudhari]

thats strange. I replicated it on my machine but everything seems to be fine. both lines seem to have the same direction. please let me know if I'm misunderstanding the issue or anything.

[tas-pats]

@A-Choudhari if you have included your generated image (which looks like you did since I didn't change opacity) you should see that your lines have the almost the same direction, but why there is a (small) gap between the two when parameters are the same and the only difference is their length? I would expect both vectors to be following exactly the same path, but one would be longer. If you wanted to see the green (y-axis) point in a different direction, try extending the axis_size to a larger value, such as 10+.

[A-Choudhari]

I tested it out with various axis sizes now and what I noticed is that as long as the ratio between the two axis sizes are less than or equal to 3, the difference in the direction of the vectors is not noticeable. Also, if the axis-size values are both less than 1, I wasn't able to pick up any difference from the eye, but logic-wise there probably is some disparity. I'll start looking into this and try to fix it. The issue must be due to rounding errors. Here's some additional figures that I tested out.

[Burnside999]

I seem to have had a similar problem before.

When dealing with this kind of perspective, if a line is too long in length, it may cross the end point, resulting in being mapped in the opposite direction.

Of course, this is just my personal guess.

[Burnside999]

The drawFrameAxes function creates four points (0, 0, 0), (length, 0, 0), (0, length, 0) and (0, 0, length) in 3D space and maps them to the 2D image according to the given parameters, which is accomplished by the projectPoints function.

However, due to camera distortion, it is normal behavior that points of different lengths do not necessarily end up on a vector. However when the length is too long, the possibility of the line transforming to other directions is indeed a problem worth considering, which could be a problem caused by the end point of the line crossing the end point of the perspective as I mentioned above.

I haven't done any experiments on this, and I'm not an expert in the field. This is a guess based on my previous experience of doing a little work related to perspective transformations.

[A-Choudhari]

Yes, I have been noticing the same thing. For now, the best fix would be to make any line that extends out of the frame of the picture to be changed into a new line with a shorter length that is till the end of the frame. This check will have to be done for all 3 lines. I have started basic implementation locally but still have a bit of work to do.

[asmorkalov]

I dumped intermediate values for the reproducer:

imagePoints before undistort: [0, 0, 0], [0.1, 0, 0], [0, 0.1, 0], [0, 0, 0.1]
imagePoints after undistort: [2284.44, 1151.9], [2075.78, 1190.44], [2261.71, 981.563], [2250.37, 1029.78]
imagePoints before undistort: [0, 0, 0], [5, 0, 0], [0, 5, 0], [0, 0, 5]
imagePoints after undistort: [2284.44, 1151.9], [-36915.2, 8972.03], [5994.52, 26044.7], [1788.2, -592.229]

Looks like the issue is not related to cv::line at all. The axis endpoint become from frame border inside of undistortion process. undistortPoints is not well defined outside the frame border as far as the distortion model itself. It's the issue reason, but not OpenCV bug. The only option I see is to check if undistorted points fit the frame and throw/warning assertion othervise. Need to discuss it with the team.

[s-trinh]

Ok, I did not read correctly the issue.

---

off-topic

Unless I have missed something, but undistortPoints should not be involved in the process when calling drawFrameAxes?

Because you go from 3D object frame → 3D camera frame using (rvec,tvec) → 2D image frame by projecting onto the image plane using the camera intrinsic matrix → then add the distortion to the image coordinates

---

~~Straight lines are drawned for the frame axis and not arrowed tip I think? If so this looks weird to me that two red/green lines are drawned. Because even if the tip coordinates is completely wrong, we should still see one straight line?~~

[asmorkalov]

1. undistortPoints / distortPoints points shift points to proper location. It's needed definitely, especially if camera has significant distortion.
2. You are right, the line should be distorted too and should not be straight, but it's not implemented.

[asmorkalov]

Discussed the issue on the Core Team meeting. Decided to not distort axes and emit warning.