Publish camera_info topic (from eeprom) in plumb_bob model instead rational_polynomial

[FPSychotic]

The camera_info topic is being published in rational_polinomial model which is not supported by ROS1. This make almost every VO/VIO/Slam software that take the calibration from the topic doesn't work, except rtabnmap because added specific support. ROS2 support it, but as almost every software is ported from ROS, still do not support rational_polynomial as ROS2 still support plumb_bob. Just there is not point in use rational_polynomial. Just make note there is files in plumb_bob model in the src folder that it looks were used previously to load the calibration from file instead from eeprom. Quite curious there is not files for 400p resolution, so what calibration was using the camera then? Other curious point is you are selling fisheye option, and planned oak-d with fisheyes, but the model supported for fisheyes in ROS is equidistant, how you have solved this for people like me that bought the fisheyes IR+rgb? No idea how to do it right (with the right calibration) in ROS with them. I think the right thing should can load in eeprom plumbob, equidistant for fisheyes and rational_polinomial from the calibration tool and can save them. Ideal should be load all of them and use them with an arg in the launch or recognising the module in fisheyes case. A temporal patch could be an arg in launch choosing load calibration from eeprom or from file and can calibrate in equidistant too

[saching13]

Hello @FPSychotic ,

The camera_info topic is being published in rational_polinomial model which is not supported by ROS1 It is supported from Jade. Here is the link to distortion_model file in ROS Melodic.

And I agree that most packages use plumb_bob and we will provide backward compatibility as I mentioned in discord previously. That said as I requested before can you share the VIO/SLAM software here that has issues so that we can test it internally when we make the changes?

Just make note there is files in plumb_bob model in the src folder that it looks were used previously to load the calibration from file instead from eeprom. Quite curious there is not files for 400p resolution, so what calibration was using the camera then?

Those files are more like dummy data that were added initially when we didn't have a way to convert the data stored on the device. Now that is available, I suggest not using these files. I will be removing the files soon.

Other curious point is you are selling fisheye option, and planned oak-d with fisheyes, but the model supported for fisheyes in ROS is equidistant, how you have solved this for people like me that bought the fisheyes IR+rgb? No idea how to do it right (with the right calibration) in ROS with them.

Quoting from REP 104.

Recently OpenCV has added support for an 8-parameter rational polynomial distortion model. This new model offers improved stability and accuracy for wide angle and fisheye lenses, which can severely distort the image. Tee rep continues to say that the complex models are not supported but that was as of 2010(ROS Daimondback). it is supported as of now.

From this, you can see that fisheye can use the same rational polynomial distortion model as long as the angle is approx less than 180 deg. when we go above that we will introduce the equidist. but it is not the priority as of now. Please share the packages that supports equidist only for the SLAM/VIO so that we can test it ? I had checked openVSLAM and they do but they don't even use the camera_info topic in SLAM.

[FPSychotic]

Some of the stacks I couldn't make work with OAK-D but that I could with a similar setup with d435. https://github.com/appliedAI-Initiative/orb_slam_2_ros https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_odometry https://github.com/uzh-rpg/rpg_svo_pro_open OpenVins as frontend in rtabmap but I think it is a issue with time stamps, you said me that was fixed ,but I seen another openned, I think is the same. https://github.com/introlab/rtabmap. https://github.com/UZ-SLAMLab/ORB_SLAM3 In orbslam3 I only texted with ROS, which is quite limited, didn't try make it work seriously, only a try , but didn't work, d435i worked. Still very basic support. Rtabmap map and VINS fusion works, but Vins fusion super slow, but I need more tests, didn't try with the last improvements. https://github.com/MIT-SPARK/Kimera-VIO. Kimera VIO can take the data from camera_info or from a file. I only tested from camera info , it doent work with the ros launch. i didn't try kimera standalone either. In the end as I said the only one that worked is RTABmap, and need use arg synch aprox and Vins Fusion (tons of delay, maybe nothing with OAK-D)

I know that modifications, that allow equidistant and rarional_polynomial but that is a feature for opencv and image_proc, in the same way that CUDA was added, but ROS doesn't support cuda, softwares does, and ROS in the wide expression doesn't support rational_polinomial, so it is not retrocompatibility,it is in fact compatibility use plumb_bob, any other thing is just kill the usability Would be much more easy find a package that use it. I hope the list be useful for you. Thanks by give support in Christmas times. Let you know that Issac ROS will add support to rational polynomial after my request, but didn't give term. Leave the fisheye support of sensors and future cameras to such supposed compatibility is a mistake, there is not such compatibility. The prove is I bought my fisheyes sensors and no way to use them, please use equidistant instead, at least there is 3 stacks that support it, rational_polinomial no one

On Tue, 28 Dec 2021, 17:42 Sachin Guruswamy, @.***> wrote:

Hello @FPSychotic https://github.com/FPSychotic ,

The camera_info topic is being published in rational_polinomial model which is not supported by ROS1 It is supported from Jade. Here http://docs.ros.org/en/melodic/api/sensor_msgs/html/distortion__models_8h.html is the link to distortion_model file in ROS Melodic.

And I agree that most packages use plumb_bob and we will provide backward compatibility as I mentioned in discord previously. That said as I requested before can you share the VIO/SLAM software here that has issues so that we can test it internally when we make the changes?

Just make note there is files in plumb_bob model in the src folder that it looks were used previously to load the calibration from file instead from eeprom. Quite curious there is not files for 400p resolution, so what calibration was using the camera then?

Those files are more like dummy data that were added initially when we didn't have a way to convert the data stored on the device. Now that is available, I suggest not using these files. I will be removing the files soon.

Other curious point is you are selling fisheye option, and planned oak-d with fisheyes, but the model supported for fisheyes in ROS is equidistant, how you have solved this for people like me that bought the fisheyes IR+rgb? No idea how to do it right (with the right calibration) in ROS with them.

Quoting from REP 104.

Recently OpenCV has added support for an 8-parameter rational polynomial distortion model. This new model offers improved stability and accuracy for wide angle and fisheye lenses, which can severely distort the image. Tee rep continues to say that the complex models are not supported but that was as of 2010(ROS Daimondback). it is supported as of now.

From this, you can see that fisheye can use the same rational polynomial distortion model as long as the angle is approx less than 180 deg. when we go above that we will introduce the equidistant. but it is not the priority as of now. Please share the packages that supports equidist only for the SLAM/VIO so that we can test it ? I had checked openVSLAM and they do but they don't even use the camera_info topic in SLAM.

— Reply to this email directly, view it on GitHub https://github.com/luxonis/depthai-ros/issues/54#issuecomment-1002213961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AITQOVFKD4ILJJ2ADTZYYODUTHZGZANCNFSM5K3RDMSA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[saching13]

Thanks for sharing. will look into those.

I know that modifications, that allow equidistant and rarional_polynomial but that is a feature for opencv and image_proc, in the same way that CUDA was added, but ROS doesn't support cuda, softwares does, and ROS in the wide expression doesn't support rational_polinomial, so it is not retrocompatibility,it is in fact compatibility use plumb_bob, any other thing is just kill the usability

ROS is a middle-ware it doesn't care what the model is. ROS REP suggests supporting these alternative models which other packages have not and the rational model is more accurate. if there is a better way to convert it back to plumb bob I would prefer to do that instead of changing what is stored to plumb bob on the device and create a deviation from the devices that are already in user's hand. From the company's perspective that is not a helpful solution when there is a better option of managing and converting to support plumb bob so that these packages can run with OAK without issues.

[FPSychotic]

Thanks for sharing. will look into those.

I know that modifications, that allow equidistant and rarional_polynomial but that is a feature for opencv and image_proc, in the same way that CUDA was added, but ROS doesn't support cuda, softwares does, and ROS in the wide expression doesn't support rational_polinomial, so it is not retrocompatibility,it is in fact compatibility use plumb_bob, any other thing is just kill the usability

ROS is a middle-ware it doesn't care what the model is. ROS REP suggests supporting these alternative models which other packages have not and the rational model is more accurate. if there is a better way to convert it back to plumb bob I would prefer to do that instead of changing what is stored to plumb bob on the device and create a deviation from the devices that are already in user's hand. From the company's perspective that is not a helpful solution when there is a better option of managing and converting to support plumb bob so that these packages can run with OAK without issues.

Yeah, I understand that perspective and I would like devs give support to that model, but they are not doing it, no even in ROS2. I understand the company don't want go plumb_bob, really I don't think it is ideal, because that I said add an arg to give the option to read calibration from topic or from file. That would give the chance to calibrate the file in the model I want, plumb, equidistant, or radial or even an alternative for rational (fisheye setup em as cameras can be swapped as it is my case). Can read from file from the launch looks a nice option for me. Basically this need be done in a way or in other , because previously OAK can became a stardard to devs support it, first need be can be used, and the real thing is, right now , and from at least from last June , I couldn't use the camera in ROS, and I cannot see how that will change or improve with a new model or with my upgraded sensors. I really thing it should be priority one for luxonis, new models are done to, improve depth and drone friendly, there is not drone friendly without ROS and sadly without plumb_bob and fisheye support (equidistant). I know I'm a headache but really I try to help and is my honest opinion

[saching13]

I think you are not understanding my message. I am not saying we are not going to support the plumb_bob. I said we would provide a backward compatibility to plumb_bob But the approach is different. Please take a look at the distortion model equations of plumb_bob and rational polynomial. From the looks of it with some math work I can compute the parameters of plumb_bob from the rational polynomilal.

Rational Polynomial model

Plumb Bob model

See you can divide the (1+k1r2+k2r4+k3r6)/(1+k4r2+k5r4+k6r6) numerator of this part by denominator and multiply with a factor to convert the 6 coeffs to 3 coeffs leading to Plumb Bob. Hope this clears up how we can achieve the plumb_bob without changing our eeprom.

and equidistant will be added in the future. we also already provide the rectified image. you can directly use that in other package without the need of feeding them the distortion model since it will already be corrected.

[y2d2]

Hi @saching13 ,

How can we enable the rectification to happen and be published? In the stereo parameters (https://github.com/luxonis/depthai-ros) I found:

i_publish_left_rect: false
i_publish_right_rect: false
i_publish_synced_rect_pair: false

I have set all 3 of them on true, however, I am not able to see the rectified topics when listing the active topics:

/oakd/imu/data
/oakd/left/camera_info
/oakd/left/image_raw
/oakd/left/image_raw/compressed
/oakd/left/image_raw/compressedDepth
/oakd/left/image_raw/theora
/oakd/right/camera_info
/oakd/right/image_raw
/oakd/right/image_raw/compressed
/oakd/right/image_raw/compressedDepth
/oakd/right/image_raw/theora

Also which parameter is it for the RGB, since it is not shown in the documentation?

thanks!

YD

[Serafadam]

Hi @y2d2, could you share more information according to the bug report template? Regarding rgb stream, currently it's not possible to get the rectified stream from the driver directly, although you can use image_proc ROS package to rectify the image (example here)

[y2d2]

@Serafadam thanks for the reply. Do you want me to open a new bug report or report it in this thread?

[Serafadam]

New bug report would be preferable