gazebosim
Issues with depth cameras
Environment
-
OS Version: Ubuntu 20.04
-
Source or binary build? Gazebo Garden source build gz-rendering is on branch gz-rendering7 with the latest commit being: commit 451e34e46622ab7a8f6aa4d521ba0a1565b8fdb7 Update examples to use metal as default graphics API on mac (#937)
- Use metal as default on apple, fix compile warning, remove optix
- Add defaultGraphicsAPI utility function
Signed-off-by: Ian Chen [email protected]
-
If this is a GUI or sensor rendering bug, describe your GPU and rendering system. Otherwise delete this section. The GPU information I get when entering the sudo lshw -C display prompt is this: description: VGA compatible controller product: GP106GL [Quadro P2200] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:02:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: pm msi pciexpress vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 resources: irq:86 memory:91000000-91ffffff memory:70000000-7fffffff memory:80000000-81ffffff ioport:2000(size=128) memory:92000000-9207ffff
- Rendering plugin: [ogre2].
- [x] Sensor rendering error.
- [ ] GUI rendering error.
- EGL headless mode: Not sure what
- [ ] Running in EGL headless mode
- Generally, mention all circumstances that might affect rendering capabilities:
- [ ] running a [gz-ros2](https://github.com/gazebosim/ros_gz.git -b humble) and ros2-ros bridge to transfer the messages to ros
- Rendering system info:
- On Linux, provide the outputs of the following commands:
LANG=C lspci -nn | grep VGA # might require installing pciutils
- On Linux, provide the outputs of the following commands:
- Rendering plugin: [ogre2].
0000:02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2200] [10de:1c31] (rev a1)
echo "$DISPLAY"
:1
LANG=C glxinfo -B | grep -i '\(direct rendering\|opengl\|profile\)' # might require installing mesa-utils package
direct rendering: Yes OpenGL vendor string: NVIDIA Corporation OpenGL renderer string: Quadro P2200/PCIe/SSE2 OpenGL core profile version string: 4.6.0 NVIDIA 545.23.08 OpenGL core profile shading language version string: 4.60 NVIDIA OpenGL core profile context flags: (none) OpenGL core profile profile mask: core profile OpenGL version string: 4.6.0 NVIDIA 545.23.08 OpenGL shading language version string: 4.60 NVIDIA OpenGL context flags: (none) OpenGL profile mask: (none) OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 545.23.08 OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
ps aux | grep Xorg
gdm 1119 0.0 0.2 25384456 67268 tty1 Sl+ Feb13 0:00 /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/125/gdm/Xauthority -background none -noreset -keeptty -verbose 3 username 1879 1.6 0.4 25518904 139304 tty2 Sl+ Feb13 15:37 /usr/lib/xorg/Xorg vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3 username 52408 0.0 0.0 9348 2644 pts/19 S+ 09:36 0:00 grep --color=auto Xorg
sudo env LANG=C X -version # if you don't have root access, try to tell the version of Xorg e.g. via package manager
```
X.Org X Server 1.20.13 X Protocol Version 11, Revision 0 Build Operating System: linux Ubuntu Current Operating System: Linux ramses 5.15.0-94-generic #104TILDESIGN 20.04.1-Ubuntu SMP Tue Jan 16 13:34:09 UTC 2024 x86_64 Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-94-generic root=UUID=41b36428-5a36-44f3-af1d-b236b2e37d7d ro quiet splash vt.handoff=7 Build Date: 29 January 2024 12:44:21PM xorg-server 2:1.20.13-1ubuntu1~20.04.15 (For technical support please see http://www.ubuntu.com/support) Current version of pixman: 0.38.4 Before reporting problems, check http://wiki.x.org to make sure that you have the latest version.
Description
-
Expected behavior:
-
When I create a camera sensor in the model.sdf file like this:
<frame name="camera_stereo_reference_frame"> <pose>0.23 0.046 -0.12 0 1.5708 0</pose> </frame> <visual name="camera_visual_left"> <visibility_flags>1</visibility_flags> <pose relative_to="camera_stereo_reference_frame"> 0 0 0 0 0 0 </pose> <transparency>0.5</transparency> <geometry> <box> <size>0.025 0.025 0.025</size> </box> <!-- <sphere><radius>0.005</radius></sphere> --> </geometry> </visual> <sensor name="camera_sensor_left" type="camera"> <pose relative_to="camera_stereo_reference_frame"> 0 0 0 0 0 0 </pose> <always_on>1</always_on> <update_rate>45</update_rate> <topic>camera/image_left</topic> <camera name="left_camera"> <camera_info_topic>left_info</camera_info_topic> <image> <width>640</width> <height>480</height> <format>R8G8B8</format> </image> <lens> <type>stereographic</type> <scale_to_hfov>false</scale_to_hfov> <cutoff_angle>3.14</cutoff_angle> <env_texture_size>640</env_texture_size> <intrinsics> <fx>256.666691776 </fx> <fy>254.19732909</fy> <cx>353.781709595</cx> <cy>236.287076</cy> <s>0</s> </intrinsics> </lens> <distortion> <k1>0</k1> <k2>0</k2> <k3>0</k3> <p1>0</p1> <p2>0</p2> <center>0 0</center> </distortion> <noise> <type>gaussian</type> <mean>0.0</mean> <stddev>0.0</stddev> </noise> </camera> </sensor>
And I want to create a depth camera at the exact same position with the exact same view to compare my stereo algorithm (that I am performing on the above and a shifted but equal camera) like this:
<visual name="depth_camera_visual">
<pose relative_to="camera_reference_frame"> 0 0 0 0 0 0 </pose>
<visibility_flags>0</visibility_flags>
<geometry>
<box>
<size>0.03 0.03 0.03</size>
</box>
<sphere><radius>0.005</radius></sphere>
</geometry>
<transparency>0.5</transparency>
</visual>
<sensor name="GT" type="depth_camera">
<pose relative_to="camera_reference_frame"> 0 0 0 0 0 0 </pose>
<always_on>1</always_on>
<update_rate>35</update_rate>
<topic>camera/depth_GT</topic>
<camera_info_topic>depth_info</camera_info_topic>
<camera name="GT_depth_camera">
<camera_info_topic>depth_info</camera_info_topic>
<horizontal_fov>1.789572604</horizontal_fov>
<depth_camera>
<camera_info_topic>depth_info</camera_info_topic>
<clip>
<near>0.05</near>
<far>1</far>
</clip>
</depth_camera>
<image>
<width>640</width>
<height>480</height>
<format>L8</format>
</image>
<lens>
<type>stereographic</type>
<scale_to_hfov>false</scale_to_hfov>
<cutoff_angle>3.14</cutoff_angle>
<env_texture_size>640</env_texture_size>
<intrinsics>
<fx>256.666691776 </fx>
<fy>254.19732909</fy>
<cx>353.781709595</cx>
<cy>236.287076</cy>
</intrinsics>
</lens>
<clip>
<near>0.05</near>
<far>1000</far>
</clip>
<distortion>
<k1>0</k1>
<k2>0</k2>
<k3>0</k3>
<p1>0</p1>
<p2>0</p2>
<center>0 0</center>
</distortion>
<noise>
<type>gaussian</type>
<mean>0.0</mean>
<stddev>0.0</stddev>
</noise>
</camera>
</sensor>
I get a different coverage of the underground. **Note ** as the intrinsic parameters of the depth map did not have an influence on the view and only the horizontal_fov tag had an effect I calculated this:
FoV = 2 × arctan（w / 2f）
Which for a focal length fx = 256.666691776 pixels and an image resolution of 640x480 pixels (aka w = 640 pixels) yields a fov of 102.535° or 1.789572604 radians.
I am assuming that the depth cameras and normal cameras have the same model. They have the exact same pose.
- Actual behavior: The Images from the depth camera and the normal camera are covering different areas. I tried to analyze it using visual editors and I am quite sure it is not just a translational shift. There is a different FOV.
Steps to reproduce
- Put the camera sensor in a model.sdf file and load them into any gazebo map with texture to verify.
- Publish the sensor messages and look at them in rqt or any other tool
Output
I apologize for the low quality map. It is hard to see but the difference can be seen from the position of the rocks.
In case I missed anything check out my stackoverflow post about this
hmm looks like it could be a bug in depth camera FOV and/or the camera intrinsics published are incorrect.
https://github.com/gazebosim/gz-sensors/issues/363 also looks related
There is code in CameraSensor.cc that handles custom intrinsic parameters: https://github.com/gazebosim/gz-sensors/blob/27239b4ac9877d87273a18010a307172899dd7a6/src/CameraSensor.cc#L319, but I don't see this in DepthCameraSensor.cc. The same is true for RgbdCameraSensor. That might explain things.
https://github.com/gazebosim/gz-sensors/pull/432 should fix the issue. Please re-open if that's not the case.