Write commands not updated after some time

[avanmalleghem]

I'm using ros2_control with diff_drive_controller and a custom hardware_interface. Here is the write method of this custom interface.

hardware_interface::return_type MyRobot::write(
const rclcpp::Time & /*time*/,
const rclcpp::Duration & /*period*/)
{
RCLCPP_DEBUG_STREAM_THROTTLE(
  rclcpp::get_logger(
    "otto_hw_interface"), steady_clock_, 5000,
  "command left : " << hw_commands_[0] << " (rad/s) - command right : " << hw_commands_[1] <<
    " (rad/s)");

return hardware_interface::return_type::OK;
}

The issue is that after some time (randomly, sometimes more than 1hour) running correctly, I observe that hw_commands_ is not populated anymore.

Can you help me understand this behavior ?

Thanks in advance.

Environment (please complete the following information):

• ROS2 Humble
• Yocto Distro
• Jetson Nano
• diff_drive_controller : 2.23.0
• controller_manager : 2.28.0

[bmagyar]

By not populated you mean you get the same prints or that the prints stop?

Are you sure the controller's update function is called? You could check that the controller is still active and receiving commands

On Thu, 28 Sept 2023, 10:01 Antoine Van Malleghem, @.***> wrote:

I'm using ros2_control with diff_drive_controller and a custom hardware_interface. Here is the write method of this custom interface.

hardware_interface::return_type MyRobot::write(const rclcpp::Time & /time/,const rclcpp::Duration & /period/) {RCLCPP_DEBUG_STREAM_THROTTLE( rclcpp::get_logger( "otto_hw_interface"), steady_clock_, 5000, "command left : " << hw_commands_[0] << " (rad/s) - command right : " << hw_commands_[1] << " (rad/s)"); return hardware_interface::return_type::OK; }

The issue is that after some time (randomly, sometimes more than 1hour) running correctly, I observe that hw_commands_ is not populated anymore.

Can you help me understand this behavior ?

Thanks in advance.

Environment (please complete the following information):

• ROS2 Humble
• Yocto Distro
• Jetson Nano
• diff_drive_controller : 2.23.0
• controller_manager : 2.28.0

— Reply to this email directly, view it on GitHub https://github.com/ros-controls/ros2_control/issues/1122, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA24PYNJJHS77ET3U2QS7LTX4U4G3ANCNFSM6AAAAAA5KTVZN4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[avanmalleghem]

Hello,

I receive 0 during like 10 minutes and then I receive correct values again. I've observed that it is because I enter into the time out condition of the update method in the diff_drive_controller. I'm currently troubleshooting it and come back to you at each step.

But of course, if you have any idea based on your experience, I'm taking it !!!

Update : it seems that the issue is that last_command_msg->header.stamp isn't updated durint like 10 minutes (I guess last_command_msg as a whole is not received). However, I subscribed to the cmd_vel topic and it seems messages are flowing...

update 2 : it seems it is an issue in the publisher itself. This publisher suddenly stops publishing messages. More information here : https://robotics.stackexchange.com/questions/104389/node-suddenly-stops-publishing-cmd-vel

[bmagyar]

I suspect this is controller behaviour. If the diff drive controller times out, it will stop the robot, hence the zeroes.

On Thu, 28 Sept 2023, 15:46 Antoine Van Malleghem, @.***> wrote:

Hello,

I receive 0 during like 10 minutes and then I receive correct values again. I've observed that it is because I enter into the time out condition of the update method in the diff_drive_controller. I'm currently troubleshooting it and come back to you at each step.

But of course, if you have any idea based on your experience, I'm taking it !!!

— Reply to this email directly, view it on GitHub https://github.com/ros-controls/ros2_control/issues/1122#issuecomment-1739417058, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA24PYO2HZCWJ3DPSHZORPTX4WEUPANCNFSM6AAAAAA5KTVZN4 . You are receiving this because you commented.Message ID: @.***>

[avanmalleghem]

I went a little further... When I ros2 topic echo from my host on the cmd_vel topic (just in front of the diff_drive_controller), I see messages in it but I still see timeout in the diff_drive_controller (see my screen where I added logs in the diff_drive_controller code, you see the growing value of the variable "age_of_last_command"). If I ros2 topic echo from the jetson nano on the same cmd_vel topic, the diff_drive_controller starts working again and "age_of_last_command" variable is more or less 0 again...

[avanmalleghem]

Update : even if I ros2 topic echo another topic than the cmd_vel topic, the whole process is working again... So it seems there is some magic in this command.

[avanmalleghem]

Update : it looks like it is linked to FastDDS. When I switch to CycloneDDS, it works like a charm.

[bmagyar]

Oh dear... 🤦 So is it something where the dif drive doesn't get more messages?

[avanmalleghem]

@bmagyar nop, it happens whatever you do but after some times (it can be hours). The behavior is that, after some times, the diff_drive_controller doesn't consume input cmd_vel anymore when using FastDDS.