image_pipeline

image_pipeline copied to clipboard

Both cameracalibrator and camerachecker fail for me (running with a 16x11 checkerboard) on the current branch noetic. It used to work with the package installed through apt with the same configuration.

When running the camerachecker I get an error

Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/camera_checker.py", line 77, in run
    self.function(m)
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/camera_checker.py", line 191, in handle_stereo
    L = self.image_corners(lgray)
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/camera_checker.py", line 148, in image_corners
    (ok, corners, b) = self.mc.get_corners(im)
ValueError: too many values to unpack (expected 3)

Using cameracalibrator, I get a similar error when pressing on "calibrate".

Traceback (most recent call last):
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/camera_calibrator.py", line 268, in on_mouse
    self.c.do_calibration()
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/calibrator.py", line 1375, in do_calibration
    self.cal_fromcorners(self.good_corners)
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/calibrator.py", line 1089, in cal_fromcorners
    lcorners = [(lco, lid, b) for (lco, rco, lid, rid, b) in good]
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/calibrator.py", line 1089, in <listcomp>
    lcorners = [(lco, lid, b) for (lco, rco, lid, rid, b) in good]
ValueError: not enough values to unpack (expected 5, got 3)

Also, using cameracalibrator the detected checkerboard points are no longer shown on the image (but it is detecting them - the bars on the right hand side increase and the calibrate button activates after a while).

I'm on commit df8713d3b1460f2fcdd37503e63374245d0943c1 (branch noetic), on Ubuntu 20.04 with python 3.8.

Update: Noticed that commit had "WIP" in its commit message. It works when I check out the commit before that (98364014ae3d4aa6ca92a0c7f6e8541ad0354675).

Reopened because this seems to be an unknown issue after all (see discussion in #551). It seems that the addition of charuco detection does not work yet and, worse, it broke normal checkerboard calibration as well. There are various issues with the current head of noetic (df8713d). Sadly, I don't have my fixes any more, but here's how far I remember getting:

• The first issue is what's mentioned above is caused by get_corners now returning four values, not 5, namely: (ok, corners, ids, b)
• Second issue happens because self.good_corners should now contain tuples with five entries, but it still contains tuples with only three entries. The left and right ids are missing. https://github.com/ros-perception/image_pipeline/blob/df8713d3b1460f2fcdd37503e63374245d0943c1/camera_calibration/src/camera_calibration/calibrator.py#L1354 should read self.good_corners.append( (lcorners, rcorners, lids, rids, lboard) ), I believe.
• Next issue is that downsample_and_detect returns (scrib, corners, downsampled_corners, ids, board, (x_scale, y_scale)), so lines 1303 and 1304 are wrong - downsamples_corners and ids are switched: https://github.com/ros-perception/image_pipeline/blob/df8713d3b1460f2fcdd37503e63374245d0943c1/camera_calibration/src/camera_calibration/calibrator.py#L1303
• After fixing all of this (and possibly more which I can't remember), I got the NotImplemented Error from https://github.com/ros-perception/image_pipeline/blob/df8713d3b1460f2fcdd37503e63374245d0943c1/camera_calibration/src/camera_calibration/calibrator.py#L1105 at which point I stopped due to lack of time and lack of knowledge.

I could make all this into a PR, but I would rather @JStech take a look, I really don't know this code well enough and will likely break something else.

Final note: The commad git diff noetic:camera_calibration/src/camera_calibration/calibrator.py melodic:camera_calibration/src/camera_calibration/calibrator.py returns almost no difference between the files - so I would be extremely surprised if the current melodic branch wasn't broken as well...?

It sounds like we might want to revert the new calibration board PR. What do you think @JStech ?

Fine by me. I'll test it more thoroughly and try again.

Thanks for the thorough issue report. Just to confirm, when this happened you were doing a stereo calibration with a regular checkerboard target, not a ChArUco board, right? I can't figure out where that NotImplemented error could have come from.

I've fixed the problems you identified, and I looked for other places those functions are used to try to get anything else. I don't have a stereo rig to test with, though, so I'm not certain I got everything. Would you mind testing my branch?

Oh, sorry, I guess I wasn't clear on this: I actually tried both. We started out with a regular checkerboard and got the issues mentioned in the first post above. Then we tried with a Charuco board and got the other issues mentioned in the longer post.

I don't know what you mean with the NotImplemented error - it's in Line 1105:: https://github.com/JStech/image_pipeline/blob/8027bd147aca78ae487daa464a9ab54764995ec0/camera_calibration/src/camera_calibration/calibrator.py#L1105 It only happens on Stereo Calibration.

I will test you branch, thanks a lot for the effort! I might not get around to it before Tuesday though, don't have the setup with me at the moment.

OK--it sounded like you were only using a regular checkerboard, but then still hit that NotImplemented error. That was my confusion.

Thanks for your help--no worries about when you get to it.

I tested your branch:

• Stereo calibration with the checkerboard works again, as expected.
• Monocular Charuco calibration works as well.

There is still an issue though: After (charuco) calibration, when receiving new images, the application crashes when trying to calculate the linear error:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/camera_calibrator.py", line 107, in run
    self.function(m)
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/camera_calibrator.py", line 188, in handle_monocular
    drawable = self.c.handle_msg(msg)
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/calibrator.py", line 945, in handle_msg
    linear_error = self.linear_error(undistorted, board)
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/calibrator.py", line 908, in linear_error
    (x1, y1) = corners[(cc * r) + 0, 0]
IndexError: index 15 is out of bounds for axis 0 with size 15

This only happens when the markers are in view.

For debugging purposes, here are the values of cr, r, corners and cc just before that line:

corners [[[332.09833 398.10028]]
 [[342.33527 458.2361 ]]
 [[384.2665  379.48782]]
 [[396.85782 439.77615]]
 [[436.93237 360.8934 ]]
 [[451.86768 420.67276]]
 [[490.23227 342.07867]]
 [[507.50687 401.49393]]
 [[526.1002  465.74286]]
 [[544.4055  322.71494]]
 [[563.9809  381.8097 ]]
 [[585.1819  445.7221 ]]
 [[598.9458  303.03622]]
 [[621.09045 361.82098]]
 [[645.07874 425.4847 ]]]
cc 5
cr 7
r 3

I also (once) got the error:

**** Calibrating ****
Traceback (most recent call last):
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/camera_calibrator.py", line 268, in on_mouse
    self.c.do_calibration()
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/calibrator.py", line 990, in do_calibration
    self.cal_fromcorners(self.good_corners)
  File "/home/user/Projects/ROS/catkin_ws_noetic/src/image_pipeline/camera_calibration/src/camera_calibration/calibrator.py", line 773, in cal_fromcorners
    reproj_err, self.intrinsics, self.distortion, rvecs, tvecs = cv2.aruco.calibrateCameraCharuco(
cv2.error: OpenCV(4.2.0) /io/opencv/modules/calib3d/src/calibration.cpp:1211: error: (-215:Assertion failed) fabs(sc) > DBL_EPSILON in function 'cvFindExtrinsicCameraParams2'

Probably unrelated: I get a "runtime warning" when pressing 'calibrate', don't know if it's relevant:

src/image_pipeline/camera_calibration/src/camera_calibration/calibrator.py:177: RuntimeWarning: invalid value encountered in float_scalars
  return math.acos(numpy.dot(ab,cb) / (numpy.linalg.norm(ab) * numpy.linalg.norm(cb)))

I tested with --pattern charuco --size 5x7 --charuco_marker_size 0.0172 --square 0.0279 --aruco_dict 5x5_250 and I used the pattern you showed here

Thanks for the info. I've fixed linear_error to handle partial board views.

I don't know what to do about the OpenCV error in the ChArUco calibration--it's not clear to me what that assertion means, and it's not happened to me. If you manage to reproduce it, I'd be happy to dig into it more.

I think we should revert the charuco PR, if @JStech doesn't have time to debug, I don't want to leave this in a broken state

I believe it's fixed, as of this change (I'm about to open the PRs). The other issues that @PfeifferMicha mentioned are not consistent--I've never seen them, and I believe Micha only saw them once each.

@PfeifferMicha Can you please confirm that noetic is fixed for this issue now that #561 is merged?

Closing due to inactivity after presumed fix was merged