I want to write my own intrinsic parameter on device but I'm confused about rectification process

[demul]

I want to write my own intrinsic parameter on device. I know I can access the calibration parameter on device by the command "Intel.Realsense.CustomRW -r" and I can write my own intrinsic parameter by "Intel.Realsense.CustomRW -w -f myparam.xml"

But there is a confusing point! The device driver publish rectified frames. So if i calibrate my image by this rectified frames, i'll get distortion parameters of already rectified image. But it seems that the parameters shown by "Intel.Realsense.CustomRW -r" has the distortion parameters of raw(not-rectified) image.

So, how can I get raw(not-rectified) frames from the driver?

My machine : D455 Environment : Ubuntu Driver : https://github.com/intel/ros2_intel_realsense

[MartyG-RealSense]

Hi @demul The RealSense ROS2 wrapper that you linked to at https://github.com/intel/ros2_intel_realsense is obsolete and has not been updated for years. The current ROS2 wrapper can be found at https://github.com/IntelRealSense/realsense-ros/tree/ros2-beta

Although most RealSense stream formats are rectified, the Y16 format is unrectified, as described at https://github.com/IntelRealSense/realsense-ros/issues/1670 where a RealSense user shared the edit that they made to the wrapper's base_realsense_node.cpp file to publish unrectified Y16.

https://github.com/IntelRealSense/realsense-ros/tree/ros2-beta/realsense2_camera/src

[demul]

Hi @MartyG-RealSense

I tried https://github.com/IntelRealSense/realsense-ros/issues/1670 But It's ROS1

I tried https://github.com/IntelRealSense/realsense-ros/tree/ros2-beta/realsense2_camera/src But It doesn't work at all. even though i mofidy "base_realsense_node.cpp" directly (like https://github.com/IntelRealSense/realsense-ros/issues/1670).

I just want u to try to get Y16 infra image on ros2 driver on D455 and tell me how to do it.

[MartyG-RealSense]

I went back to the beginning of this case to re-read it for new insights. I note that you were using the CustomRW tool that is part of the custom calibration system and were writing your own XML calibration but were concerned about calibrating with rectified frames.

However, the calibration tool does not calibrate with a rectified format. It calibrates with unrectified Y16 infrared, as described on page 11 of Intel's custom calibration white-paper guide at the link below.

https://dev.intelrealsense.com/docs/d400-series-custom-calibration-white-paper

So an approach that you could perhaps use is:

1. Calibrate with the custom calibration tool knowing that it is using Y16 to calibrate.

2. Export its calibration table to xml

3. Edit the contents of the XML file to your custom intrinsics.

4. Load the edited xml file back in with CustomRW and write its calibration to the camera hardware.

[demul]

Thankss @MartyG-RealSense I can't use the intel custom calibration tool. I must get Y16 stream inside of my automatic calibration system. So i must get Y16 stream on API level(ros2). Would u kindly try to get Y16 infra image on ros2 driver on D455 and tell me how to do it? Please try it and teach me how to reproduce. (Docker is ok) Thank u

[MartyG-RealSense]

I will refer your question to my Intel RealSense development colleagues. Thanks very much for your patience!

[demul]

Thanks @MartyG-RealSense Is ther any progress on it???

[MartyG-RealSense]

Not yet. I will follow up the inquiry that I made to my colleagues. Thanks very much for your patience!

[MartyG-RealSense]

A member of the RealSense ROS development team provided the following advice about edits to base_realsense_node.cpp to try.

---

For a code update I suggest changing this line:

https://github.com/IntelRealSense/realsense-ros/blob/ddd6fe116859b960a59412538e91d085a368c871/realsense2_camera/src/base_realsense_node.cpp#L107

_format[RS2_STREAM_INFRARED] = RS2_FORMAT_Y16;

And if point cloud texture is needed, these lines should be updated too:

https://github.com/IntelRealSense/realsense-ros/blob/ddd6fe116859b960a59412538e91d085a368c871/realsense2_camera/src/base_realsense_node.cpp#L2180

https://github.com/IntelRealSense/realsense-ros/blob/ddd6fe116859b960a59412538e91d085a368c871/realsense2_camera/src/base_realsense_node.cpp#L2228

[demul]

@MartyG-RealSense The solution u suggest have no difference with "https://github.com/IntelRealSense/realsense-ros/issues/1670" u have already suggested.

Plz try and verify it works before u comment I am repeating to say "have a try and tell me" I don't want to waste my time anymore

[demul]

There is the least time-wasting solution u can give

1. Connect D445 on your pc
2. Try to get Y16 stream from the damn ros2 driver
3. Identify it do not work
4. Fix bugs
5. Make a bug fix branch
6. And give me the branch

[MartyG-RealSense]

I do genuinely understand your frustration. Using the edits suggested in https://github.com/IntelRealSense/librealsense/issues/10766#issuecomment-1225614058 to edit base_realsense_node.cpp yourself is the only solution that we can offer at the present time though, unfortunately. I do apologize.

[demul]

@MartyG-RealSense So r u saying that u don't have any idea to fix that damn software gliches at all? Are you kidding??? I cannot understand why does Intel software development team exist.

[demul]

"base_realsense_node.cpp yourself is the only solution that we can offer at the present time though" No. The solution you can give is

1. Connect D445 on your pc
2. Try to get Y16 stream from the damn ros2 driver
3. Identify it do not work
4. Fix bugs
5. Make a bug fix branch
6. And give me the branch

You're just short of responsibility and you're too lazy to do after-sales service.

[demul]

This is not a simple thing that a lazy employee reject a nervous customer's request. This means, Realsense is unusable for mass produce because it is not auto-calibratable in manufacturing pipeline. And i don't intend to keep this to myself.

[demul]

@MartyG-RealSense I managed to invent temporary solution.

1. set distortion parameter to 0, and left-right transformation to identity using "Intel.Realsense.CustomRW -w -f myparam.xml"
2. maybe i can get image that have no undistortion and no baseline alignment are applied. Only crop is applied.
3. calibrate with that "almost raw image".
4. Adjust cx and cy according to cropping method

So U can help me with that,

1. Inform me what 'Rectification Process' to exactly. Especially "how to crop".(how much pixels are cropped on top and bottom. so that i can adjust cx and cy correctly)
2. Please Identify if i set distortion parameter to 0 and set left-right transformation to identity, I can get "almost raw image".

[MartyG-RealSense]

Thank you for your patience whilst I discussed your case with my Intel RealSense colleagues. They provided the feedback below.

---

Y16 is unrectified calibration frames. It is supported in SDK APIs (for example, enable_stream in librealsense\include\librealsense2\hpp\rs_pipeline.hpp and demonstrated in the Realsense Viewer). The ROS2 wrapper does not support calibration.

For custom calibration, please use Y16 as described in the custom calibration white paper and use the sample app as reference.

https://dev.intelrealsense.com/docs/d400-series-custom-calibration-white-paper

Do not use the sample app directly as that is for example purposes only. Users are expected to write their own calibration app with the SDK API and their own calibration algorithm.

Many of our customers have successfully used this approach to calibrate their devices on production lines. CustomRW expects results produced from this approach. To avoid device malfunction, please do NOT overwrite the calibration on the device with results from rectified images.

If you have a Non Disclosure Agreement (NDA) with Intel then please open a ticket on Intel's Zendesk support channel.

[MartyG-RealSense]

Hi @demul Was the information provided in the comment above helpful to you, please? Thanks!

[demul]

Partially helpful. The comment "Y16 is unrectified calibration frames. It is supported in SDK APIs" is helpful. It's true. But the comment "The ROS2 wrapper does not support calibration." is not helpful. and It's false. I succeed to get Y16 frame from ros2 wrapper with so many modifications. Actually, the reason why the old ROS2 wrapper(before i modified) doesn't support calibration is that, you wrongly wrapped it.

To explain in more detail, There are the two major reasons. First, The profile with Y16 stream do not have intrinsic parameter. it looks like a bug. it's ok. Second, The architecture is counterlogical. The old ROS2 wrapper always require intrinsic parameter. Even though we do not request depth image. It's counterlogical. If we just request infra images or(and) a rgb image, It makes sense that we don't have to input any intrinsic parameter into ROS2 wrapper. but the wrapper requires intrinsic parameter and if we do not input it, it crashes.

If either problem is solved, you can get Y16 stream from ROS2 wrapper. Because i'm so short of time, i just solve only the second problem. and finally succed to get Y16 frame from ROS2 wrapper. But both problems must be addressed! The first one seems to be just a bug, but the second one is clearly a architectural glich.

Thanks for your supports. @MartyG-RealSense

[MartyG-RealSense]

You are very welcome, @demul - it's great to hear that you achieved a solution. Thanks very much for your detailed response, which I will pass to the RealSense ROS development team. Thanks again!

[MartyG-RealSense]

Hi @demul Do you require further assistance with this case, please? Thanks!

[demul]

I'll not close this issue. This issue is critical and must be tracked until officially solved.

[MartyG-RealSense]

Thanks very much for the update. What do you believe the next step should be in order to resolve this issue to your satisfaction, please?

[demul]

Partially helpful. The comment "Y16 is unrectified calibration frames. It is supported in SDK APIs" is helpful. It's true. But the comment "The ROS2 wrapper does not support calibration." is not helpful. and It's false. I succeed to get Y16 frame from ros2 wrapper with so many modifications. Actually, the reason why the old ROS2 wrapper(before i modified) doesn't support calibration is that, you wrongly wrapped it.

To explain in more detail, There are the two major reasons. First, The profile with Y16 stream do not have intrinsic parameter. it looks like a bug. it's ok. Second, The architecture is counterlogical. The old ROS2 wrapper always require intrinsic parameter. Even though we do not request depth image. It's counterlogical. If we just request infra images or(and) a rgb image, It makes sense that we don't have to input any intrinsic parameter into ROS2 wrapper. but the wrapper requires intrinsic parameter and if we do not input it, it crashes.

If either problem is solved, you can get Y16 stream from ROS2 wrapper. Because i'm so short of time, i just solve only the second problem. and finally succed to get Y16 frame from ROS2 wrapper. But both problems must be addressed! The first one seems to be just a bug, but the second one is clearly a architectural glich.

Thanks for your supports. @MartyG-RealSense

You already know the next steps. It is to solve these issues and update ros2 wrapper branch. I solved this issue individually, but as i told, it is just temporary solution. This issue needs official fixes. and i already guide you where/how/what to fix.

[MartyG-RealSense]

Thank you @demul - I will discuss the issue further with my Intel RealSense colleagues.

[MartyG-RealSense]

After discussion with my RealSense colleagues, an official Intel internal feature request for adding Y16 support to the ros2_beta ROS wrapper has been created. In the meantime, please continue to use your own solution.

This case should be kept open whilst the change request is active. Thanks!

[demul]

@MartyG-RealSense

That's sounds great! Thank u for your supports!

[demul]

@MartyG-RealSense Hi Marty. Long time no talk!

After discussion with my RealSense colleagues, an official Intel internal feature request for adding Y16 support to the ros2_beta ROS wrapper has been created. In the meantime, please continue to use your own solution.

This case should be kept open whilst the change request is active. Thanks!

I'm curious is there any progression on this feature. Any good news for me??

[MartyG-RealSense]

Hi @demul I check the status of this Y16 feature request every few months but there is no progress to report so far, unfortunately.

[SamerKhshiboun]

Hi @MartyG-RealSense , I think you can close this issue since it was fixed in https://github.com/IntelRealSense/realsense-ros/pull/2840 @demul follow the Readme in order to understand how to use these parameters. Let me know if you have any other question.

[MartyG-RealSense]

Okay, thanks very much @SamerKhshiboun for bringing it to my attention!