How to manage BGRA8 image encoding

[lucarossini-iit]

Hello guys and thanks for the amazing work! I am currently using the elevation mapping tool with a ZedX Mini camera that uses the BGRA8 encoding for the colored image. This is the config file I am currently using:

#### Plugins ########
plugin_config_file: '$(rospack find elevation_mapping_cupy)/config/core/plugin_config.yaml'

#### Channel Fusions ########
pointcloud_channel_fusions:
  rgb: 'color'                # 'color' fusion is used for the 'rgb' channel
  default: 'average'          # 'average' fusion is used for channels not listed here

image_channel_fusions:
  rgb: 'color'                # 'color' fusion is used for the 'rgb' channel
  default: 'exponential'      # 'exponential' fusion is used for channels not listed here
  feat_.*: 'exponential'      # 'exponential' fusion is also used for any channel starting with 'feat_' Regular expressions can be used for channel names

subscribers:
  front_cam:
    topic_name: '/zedx_left/zed_node/point_cloud/cloud_registered'
    data_type: pointcloud
  color_cam: # for color camera
    topic_name: '/zedx_left/zed_node/rgb/image_rect_color'
    camera_info_topic_name: '/zedx_left/zed_node/rgb/camera_info'
    data_type: image

publishers:
  elevation_map_raw:
    layers: ['elevation', 'traversability', 'variance']
    basic_layers: ['elevation']
    fps: 5.0
  elevation_map_recordable:
   layers: ['elevation', 'traversability']
   basic_layers: ['elevation', 'traversability']
   fps: 2.0
  elevation_map_filter:
    layers: ['min_filter', 'smooth', 'inpaint', 'elevation']
    basic_layers: ['min_filter']
    fps: 3.0

However, when I run the code, it returns me the following error:

Mismatch in the size of multichannel_image (4), channels (1). Please check the input.
Current Channels: rgb

Digging in you code, it should support the bgra8 image encoding but I don't know how to fix this problem. Probably the issue is in the rgbfield of the Channel Fusion section of the config file but I am not sure how the code will react by simply modifying the config file field.

What should I do to use the elevation mapping with bgra8 endoded images?

Thank you very much. Luca Rossini

[lucarossini-iit]

Hello guys. Is there any news related to this topic?

Thank you

[mktk1117]

Hi, can you try setting the channels in the subscriber?

subscribers: front_cam: topic_name: '/zedx_left/zed_node/point_cloud/cloud_registered' data_type: pointcloud color_cam: # for color camera topic_name: '/zedx_left/zed_node/rgb/image_rect_color' camera_info_topic_name: '/zedx_left/zed_node/rgb/camera_info' data_type: image channels: ["rgb", "a"]

Something like this?

[lucarossini-iit]

Hi @mktk1117 and thank you for your reply. So far, I avoided the problem converting the encoding of my camera from bgra to rgb. Yet, I see a strange behavior when I generate the elevation mapping in correspondence of differently colored areas. For instance, the elevation mapping is not generated on areas of the ground where we have some circular orange adhesive patches and also in correspondence of a wooden box with a light color, despite the point cloud detects both areas. The floor is colored with a dark gery-ish color and I am wondering if the contrast in color influences the generation of the elevation map, and how it can be eventually disabled. Maybe your previous answer will fix also this behavior.

I cannot attach pictures right now but I will do it in the next days.

Thank you. Luca

[mktk1117]

Hi Luca, Umm, that's weird... The elevation layer is only generated from point cloud data so the color should not affect as long as there's a point cloud. I did see some issue with realsense which does not see the bright area due to high contrast, but in this case the point cloud is missing.

[lucarossini-iit]

Hello there! I just took some pictures from the robot to give you a clearer idea of the problem. I moved the robot in front in front of the circular orange markers and in front of a light colored step and the elevation mapping is not returning any value in correspondence of these two areas.

Pointcloud and elevation mapping

Pointcloud

Elevation mapping

Pointcloud and elevation mapping

Pointcloud

Elevation mapping

Here you see that the elevation mapping is not constructed also in correspondence of the light colored tape on the floor

The funny thing is that I am able to detect the step and generate correctly the elevation map by covering the step with a darker layer:

Is it possible that the color encoding returns null values in correspondence of specific colors?

Thank you, Luca

[lucarossini-iit]

I just tried with the D435i Realsense (with rgb encoding) and it works fine.

[mktk1117]

That's super weird. You input the same point cloud to the elevation map right? And you're visualizing the elevation_map_raw with elevation layer for both height and color? Can you also visualize the point cloud without the color transformer?

[lucarossini-iit]

Yes, the point cloud is in input to the elevation map of course and the elevation layer is used both for the height and color.

I don't know if I can actually deactivate the color transformer but here's a screenshot with the pointcloud visualized with color transformer intensity.

Now I will try to iterate inside the pointcloud and see if there are points with a=0 that maybe can be ignored by the elevation_mapping tool even if you do not use the rgba fields for this purpose.

Thank you for your help :)

[lucarossini-iit]

I just made a simple script to remove all points with a < 255 and none of the points was removed and so the problem is not related to transparent points.

[mktk1117]

If you have a rosbag to try, could you share it?

[lucarossini-iit]

Hello @mktk1117 and sorry for the late reply, I was out of the office and somehow I missed the notification of your reply. Here's a bag file with the pointcloud taken from the ZedX Mini:

https://drive.google.com/file/d/1M5WfcTheIMeCIR9ijf6EP84UbcAxACy_/view?usp=sharing

This is a pointcloud of the of the camera facing the light colored wooden step. Please run rosrun tf static_transform_publisher 0 0 0 0 0 0 map zedx_left_left_camera_frame 30to visualize it on rViz in case you need it.

[mktk1117]

Thanks for the bag file. I found the reason. It was because, I was filtering out pointcloud by checking if there's anything with nan value in the points.

raw_points = raw_points[~cp.isnan(raw_points).any(axis=1)] After I modified to only check the xyz, the hole part was updated.

Please check #103 If it looks okay, I'll merge it soon.