Disconnecting robot cable leaves driver in buggy state

[ametrix]

Summary

Disconnecting and reconnecting robot's network cable leaves driver in buggy state.

Here is the log:

[ WARN] [1632833181.573771179]: Failed to read from stream, reconnecting in 2 seconds... [ WARN] [1632833184.581813469]: Failed to read from stream, reconnecting in 4 seconds... [ WARN] [1632833189.605824902]: Failed to read from stream, reconnecting in 8 seconds... [ WARN] [1632833198.629815884]: Failed to read from stream, reconnecting in 16 seconds... [ WARN] [1632833215.653788639]: Failed to read from stream, reconnecting in 32 seconds... [ WARN] [1632833248.677814791]: Failed to read from stream, reconnecting in 64 seconds... [ WARN] [1632833277.549824994]: ur_robot_driver's RTDE interface registered a new maximum delay [100 ms] to perform a ::read() [ERROR] [1632833277.549853846]: Could not get fresh data package from robot [ INFO] [1632833277.568883761]: Connection to reverse interface dropped. [ INFO] [1632833277.569134354]: Robot connected to reverse interface. Ready to receive control commands. [ WARN] [1632833313.701815160]: Failed to read from stream, reconnecting in 64 seconds...

Introduction to the issue

Versions

• ROS Driver version: (commit hash 44858fb - Fix package name in calibration example)
• Affected Robot Software Version(s): 5.11
• Affected Robot Hardware Version(s): ur10e
• Robot Serial Number:
• UR+ product(s) installed:
• URCaps Software version(s): 1.0.5

Impact

Can't control robot correctly after this.

Issue details

We have looked at the code and in URProducer::tryGet() there is a istream_.connect() call when stream_.read(buf, sizeof(buf), read) fails. But it will connect socket after its state changed to something other than SocketState::Connected. The driver is slow to detect this change since it only handles end of file sent from the server:

TCPSocket::read()
...
 if (res == 0)
  {
    state_ = SocketState::Disconnected;
    return false;
  }
  else if (res < 0)
    return false;

And we see that EOF is not received immediately after disconnecting and connecting the cable. When socket is returning error( res < 0) the state also have to be set in disconnected state, we consider this as a bug. Also may be there should be keepalive setting with strict values, to be able to detect disconnect immediately.

There is an another problem after socket reconnects, that we don't receive anything from RTDE interface. Looking at the code we saw that there is a negotiation step for RTDE protocol, after which streaming from robot starts. We think that this is not implemented on cable disconnects.

Use Case and Setup

Project status at point of discovered

When we tried to disconnect the robot cable.

Steps to Reproduce

Turn robot set it to a running state with remote control mode selected. Start the Universal Robots ROS Driver. Disconnect the robot cable and connect it after couple of seconds.

Expected Behavior

Driver sockets to reconnect to the robot's.

Actual Behavior

Not all sockets reconnected after reconnect.

Workaround Suggestion

Restart the driver after cable is unplugged and plugged again.

[urrsk]

@ametrix Thanks for you report. I agree that this could be a nice feature support a more seamless behavior. Though, it might not always be a good solution to just start the robot move again after cable disconnection.

[fmauch]

@ametrix thanks for explicitly reporting this. We have that on our radar and hopefully will find the time to tackle this soon.

[github-actions[bot]]

This issue has not been updated for a long time. If no further updates are added, this will be closed automatically. Comment on the issue to prevent automatic closing.

[github-actions[bot]]

This issue has been closed due to inactivity. Feel free to comment or reopen if this is still relevant.

[fmauch]

This should still stay.

[github-actions[bot]]

This issue has not been updated for a long time. If no further updates are added, this will be closed automatically. Comment on the issue to prevent automatic closing.

[fmauch]

Still relevant