[Simulation, Humble] Segmentation fault in `gzserver`

[goekce]

I followed the instructions in https://emanual.robotis.com/docs/en/platform/turtlebot3/manipulation/#software-setup for Humble. I had to install the following packages:

ros-humble-moveit-servo
ros-humble-gazebo-ros2-control
ros-humble-gazebo-plugins

When I launch the simulation using

ros2 launch turtlebot3_manipulation_bringup gazebo.launch.py

Then I get the following segmentation fault:

exit code -11, cmd 'gzserver .../turtlebot3_world.model -slibgazebo_ros_init.so -slibgazebo_ros_factory.so -slibgazebo_ros_force_system.so'

In the end the ros2_control nodes cannot be created and the simulation does not work.

The simulation uses the Gazebo plugin libgazebo_ros_diff_drive.so which comes with ros-humble-gazebo-plugins. Without ros-humble-gazebo-plugins the simulation starts and the arm can be moved but not the base, because libgazebo_ros_diff_drive.so is responsible for reading /cmd_vel messages.

My hunch ist that 62c83db026f91a3f6edcb847cbe505aeabeb2d9c could be responsible for the segmentation fault, maybe because ros2_control and libgazebo_ros_diff_drive.so are used together.

Can someone help me with this problem?

[goekce]

I noticed that

ros2 launch turtlebot3_manipulation_bringup base.launch.py start_rviz:=true use_fake_hardware:=true fake_sensor_commands:=true

can successfully instantiate the differential driver controller which listens on /cmd_vel, however adding Gazebo

ros2 launch turtlebot3_manipulation_bringup base.launch.py start_rviz:=true use_fake_hardware:=true fake_sensor_commands:=true use_sim:=true

fails.

I noticed that last commits came from @HPaper . Can you help with this issue @HPaper ?

[HPaper]

I noticed that

ros2 launch turtlebot3_manipulation_bringup base.launch.py start_rviz:=true use_fake_hardware:=true fake_sensor_commands:=true

can successfully instantiate the differential driver controller which listens on /cmd_vel, however adding Gazebo

ros2 launch turtlebot3_manipulation_bringup base.launch.py start_rviz:=true use_fake_hardware:=true fake_sensor_commands:=true use_sim:=true

fails.

I noticed that last commits came from @HPaper . Can you help with this issue @HPaper ?

base.launch is supposed to run the hardware_interface required to operate the actual hardware. If you want to run turtlebot3_manipulation in gazebo, run gazebo.launch.py ​​of turtlebot_manipulation_bringup.

[goekce]

When I launch the simulation using

ros2 launch turtlebot3_manipulation_bringup gazebo.launch.py

Then I get the following segmentation fault:

exit code -11, cmd 'gzserver .../turtlebot3_world.model -slibgazebo_ros_init.so -slibgazebo_ros_factory.so -slibgazebo_ros_force_system.so'

In the end the ros2_control nodes cannot be created and the simulation does not work @HPaper . Please see my first post for details.

BTW gazebo.launch.py uses base.launch.py under the hood:

https://github.com/ROBOTIS-GIT/turtlebot3_manipulation/blob/3baa265d0f4843955ed77716be6f88b71689d7a1/turtlebot3_manipulation_bringup/launch/gazebo.launch.py#L120-L127

this is the reason why I tried to debug using base.launch.py

[goekce]

I have new insights. I started and killed the simulation using CTRL-C about 10 times. The simulation worked in the third and in the tenth time. In all other times the gzserver died. If I don't use the libgazebo_ros_diff_drive.so, then gzserver never dies.

I remember something similar happening in turtlebot3_simulations, where gzserver used to crash similarly but not often like this. The solution was to restart the simulation and hope that it did not crash again.

My environment: ROS Docker image with desktop-full configuration on a cloud computer with remote desktop. I cloned the turtlebot3_manipulation packages according to the documentation. Even these packages are also available in apt, starting the simulation using the system-installed ROS packages does not work, probably because Gazebo cannot find some models.

I tried also on a local computer using the same Docker image. The simulation worked in about 5 of 10 tries. Maybe this is a race issue and the plugin must be loaded in a specific order.

How to move forward:

1. Live with the workaround: Restart the launch file until the simulation works and then keep Gazebo open.
2. Try without using the libgazebo_ros_diff_drive.so by using the differential drive controller from ros2_control
3. Some Gazebo users reported intermittent issues with gzserver. In one case using another physics engine helped. Maybe trying another physics engine could also help in our case

Do you have other ideas?

[sunghowoo]

@goekce From what I can see, your issue seems to be that the simulation fails to start intermittently when gzserver crashes while running gazebo.launch.py. I recently encountered a similar issue and debugged it on the humble-devel branch. The branch where the issue has been fixed is updated on humble-teleop. I have confirmed that when running $ ros2 launch turtlebot3_manipulation_bringup gazebo.launch.py, it successfully starts 10 out of 10 times.