[Intrinsic camera calibration] Extremely high delay of up to 4s on the camera image screen of the calibration software

[zymouse]

Please make sure of the following things

• [x] I have read the documentation.
• [x] I'm sure there are no duplicate issues or discussions.
• [x] I'm sure this issue is not fixed in the latest version.

Version

decf391

Describe the bug

is a great tool, and in combination with apriltag corner detection it also verifies that the internal calibration is great.

During use, it was found that the latency of the camera image was found to be very high.

ultimate

Thank you very much, these calibration tools are great and do a much better job of calibrating the sensors!

[knzo25]

Hi, thanks for using our tools.

The delay is probably due to the detector being too slow for your image size. What are the dimensions of your image?

Each detector additionally has parameters, which if tuned, will make the latency lower. The ROS tool among others will resize your image, detect in the smaller image, re-scale, and then refine the detections. I don't really trust that, so I did not implement that on purpose.

@aohsato The latency when we calibrate C2 cameras is also almost unbearable. Do you have a preference on how to proceed? My current ideas are as follows:

• Implement the resize strategy as optional, and perform some experiments to check if this approach actually incurs into a bigger reprojection error. Provide config files for C1 and C2 cameras separately
• Tune the detector parameters to try to reduce the latency for big images. This will probably reduce the detection range, but we won't calibrate with boards over 30m (we don't have the space)
• Perform detections with queues and multi threading. This will not reduce the latency, but will increase the number of images actually processed and the UI will look more fluid

[aohsato]

@knzo25 Thanks, I think that's a good point!

Your three ideas are great, but it might be good to use two different image sizes for some purposes.

• Resized small image: camera image visualization and corner detection for sampling keyframes.
• Original size image: corner detection and calibration execution with sampled keyframes.

(keyframe: collected frames actually used for calibration)

What do you think? Is the code change too big?

[knzo25]

Ok, I will do that. The changes are not to big, but also not that small. Let's discuss the priority internally.

@zymouse Please for a while until we implement the changes :pray:

[zymouse]

@knzo25 k Lower the frequency of image_raw, this improved when I lowered it to 10hz

[knzo25]

@zymouse Hi, thanks for the comment. We also replicated the issue last week. Usually we run everything at 10hz, and it runs smoothly for out 1920x1080 camera (it runs slow for higher res cameras), but when I ran it free run (~30Hz) it got laggy.

Since we have the hardware to replicate and check the fix, we should be able to address it soon-ish (by the end of the month, together with the next release)

[ParashRahman]

I'm experiencing slowness. What is the size parameter units?

[knzo25]

Could you try this branch? https://github.com/tier4/CalibrationTools/pull/152