Headless mode is not working

[ametrix]

Summary

resend_robot_program service fails randomly after switching between remote to manual mode and back

Versions

• ROS Driver version: master
• Affected Robot Software Version(s): URSoftware 5.10.2.106319 (May 04 2021)
• Affected Robot Hardware Version(s): ur10e
• Robot Serial Number:
• UR+ product(s) installed:
• URCaps Software version(s):

Impact

The headless mode is useless with this behavior

Issue details

We are trying to use headless mode in our project. Until now we used the externalcontrol-1.0.5.urcap for controlling the robot and it worked well. We removed/disabled the external control program from polyscope and started ur_robot_driver with headless_mode:=true. Our software starts parallelly with the driver and waits for all services exposed by it. After that waits for robot mode and safety mode to be ok. Also it polls the robot to check that it is in remote control mode. After everything is ok we also check if program is running and we call the stop service and then the send_robot_program service is called. This procedure is repeated when any of robot mode, safety mode and remote mode changes. When the resend_robot_program is called at startup we get these errors: "Sending data through socket failed.", "Could not send program to robot" and "Could not resend robot program". As a workaround we put 10 sec sleep in our application's init function after waiting services to give ur_robot_driver more time to fully start. Then the service worked correctly. But we still get randomly these errors when we switch the robot from remote to manual mode and vice versa."

Use Case and Setup

Project status at point of discovered

When we tried to use headless_mode

Steps to Reproduce

• Start the driver with (use correct robot ip): roslaunch ur_robot_driver ur10e_bringup.launch robot_ip:=x.x.x.x headless_mode:=true
• Switch the robot from remote to manual mode and vice versa.
• Try to call resend_robot_program service

Expected Behavior

resend_robot_program service to execute successfully. Running program state visible in teach pendant.

Actual Behavior

resend_robot_program service fails with these errors: "Sending data through socket failed.", "Could not send program to robot" and "Could not resend robot program"

Workaround Suggestion

Workaround for startup sleep 10 sec

[fmauch]

By

Switch the robot from remote to manual mode and vice versa.

you mean switching between remote control mode and local mode on the teach pendant? Or do you mean moving the robot around using a teach pendant?

I cannot reproduce this with a URSim robot. Trying a real robot later...

[fmauch]

I could reproduce it using a real robot:

1. have robot in remote control mode, start the driver
2. set the robot to local control
3. set the robot to manual
4. move the robot around
5. set the robot to automatic
6. set the robot to remote_control mode
7. resend the program

---

Edit: This seems not to be enough to reproduce this.

[fmauch]

Though I could not find a reliable way to reproduce this, this definitely seems to happen sometimes. I have been playing around with sending the program while being in local (manual) mode and then sometimes this happened.

Observations so far:

• connections to both, the primary and secondary interface seem to close a while after this happened, not instantly

[ametrix]

I forgot to mention that it works flawlessly with URSim and also we didn't able to reproduce the issue with it. We have this problem only with the real robot. So I confirm your findings.

[ametrix]

We have updated our robot's firmware to version 5.11. The version of the Universal Robots ROS Driver which we use is the same (commit hash 44858fbb83a6731b8bfcec76458412049ce61448 - Fix package name in calibration example).

After some debugging with our hardware and searching in the internet I found this documentation about remote control e-series robots via TCP/IP:

In e-Series, there are remote and local modes. When the robot is in the local mode, it is not allowed to receive URScript commands and the connection to the ports 30001, 30002, and 30003 will be lost. If you need to monitor the robot state both in remote and local modes, you should use the ports 30004, 30011, 30012, and 30013.

https://www.universal-robots.com/articles/ur/interface-communication/remote-control-via-tcpip/

With our robot I have observed this situations.

1. First scenario.

• Robot is in RemoteControl mode.
• Then Universal Robots ROS Driver is started with headless_mode:=true.
• Calling resend_robot_program service is successful (success: True, message: "Successfully resent robot program") and the script is executed correctly in teach pendant.
• Then control mode is switched to LocalControl.
• After that we are going back to RemoteControl.
• Calling resend_robot_program fails with: "Sending data through socket failed." "Could not send program to robot".
• As a workaround I modified the code of the ur_driver.cpp to reconnect the secondary interface after this error occurs and everything worked fine.

2. Second scenario.

• Robot is in LocalControl mode.
• Then Universal Robots ROS Driver is started with headless_mode:=true.
• Then control mode is switched to RemoteControl.
• Calling resend_robot_program service is successful (success: True, message: "Successfully resent robot program"), but the script is not executed correctly in teach pendant. It seems the robot ignores messages sent to secondary_stream, because the connection to port 30002 was made in local mode. Reconnecting secondary_stream socket in RemoteControl mode fixes the problem.

From the above my conclusion is that secondary stream must be reconnected similarly to dashboard client after switch to RemoteControl mode.

[ametrix]

One additional note: When sockets are connected to 30001, 30002, and 30003 ports when robot is in LocalMode, they will work without errors (Read/Write), but robot will ignore any data sent to them. Also switch to RemoteControl and vice versa will not disconnect this sockets. This is my observation with our ur10e robot.

[fmauch]

Thanks for following up on this! One problem is that we can't really determine whether we are in remote_control mode without polling it though the dashboard interface.