ros2cli
ros2cli copied to clipboard
ros2
ros2cli daemon: cannot use Destroyable because destruction was requested
Bug report
Required Info:
- Operating System: Ubuntu 22.04
- Installation type: binary
- Version or commit hash: 0.19.0
- DDS implementation: cyclonedds
Description
From time to time, I get the following traceback when I use the ros2 cli instead of the usual output.
It persists until I run ros2 daemon stop. Unfortunately, the problem occurs seemingly arbitrarily, so I cannot give steps to reproduce (yet).
Traceback (most recent call last):
File "/opt/ros/rolling/bin/ros2", line 33, in <module>
sys.exit(load_entry_point('ros2cli==0.19.0', 'console_scripts', 'ros2')())
File "/opt/ros/rolling/lib/python3.10/site-packages/ros2cli/cli.py", line 89, in main
rc = extension.main(parser=parser, args=args)
File "/opt/ros/rolling/lib/python3.10/site-packages/ros2topic/command/topic.py", line 41, in main
return extension.main(args=args)
File "/opt/ros/rolling/lib/python3.10/site-packages/ros2topic/verb/echo.py", line 220, in main
qos_profile = self.choose_qos(node, args)
File "/opt/ros/rolling/lib/python3.10/site-packages/ros2topic/verb/echo.py", line 146, in choose_qos
pubs_info = node.get_publishers_info_by_topic(args.topic_name)
File "/usr/lib/python3.10/xmlrpc/client.py", line 1122, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python3.10/xmlrpc/client.py", line 1464, in __request
response = self.__transport.request(
File "/usr/lib/python3.10/xmlrpc/client.py", line 1166, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.10/xmlrpc/client.py", line 1182, in single_request
return self.parse_response(resp)
File "/usr/lib/python3.10/xmlrpc/client.py", line 1354, in parse_response
return u.close()
File "/usr/lib/python3.10/xmlrpc/client.py", line 668, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'rclpy._rclpy_pybind11.InvalidHandle'>:cannot use Destroyable because destruction was requested">
I am also getting this issue with the same behaviour. What additional information is required @fujitatomoya ?
@austin-InDro
thanks for the ping, can you reproduce this issue?
can you try to start the daemon with debug flag to see if any messages come up?
# ros2 daemon start --debug
The daemon has been started
root@tomoyafujita:~/ros2_ws/colcon_ws# Interface kind: 2, info: [('43.135.146.3', 'enp4s0', True)]
Interface kind: 10, info: [('fe80::5:73ff:fea0:6', 'enp4s0', True)]
Addresses by interfaces: {2: {'enp4s0': '43.135.146.89'}, 10: {'enp4s0': '2607:fd28:130:e15:0:dddd:2:15e'}}
Serving XML-RPC on http://127.0.0.1:11511/ros2cli/
...<snip>
I get the following, doesn't seem to be any errors, but I can pipe it to a log and wait for the error to appear as well.
ros2 daemon start --debug
1659016481.930559 [0] python3: config: //CycloneDDS/Domain/General: 'NetworkInterfaceAddress': deprecated element (file:///home/austin/Documents/cyclonedds.xml line 4)
The daemon has been started
(ros) austin@austin-greisman:~$ Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
Serving XML-RPC on http://127.0.0.1:11511/ros2cli/
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Okay so I was able to get the error while the debug was running:
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True), ('192.168.42.1', 'enx4ce1734ea1fa', False)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111', 'enx4ce1734ea1fa': '192.168.42.143'}}
Daemon node was reset
get_node_names_and_namespaces()
Interface kind: 2, info: [('192.168.42.1', 'enx4ce1734ea1fa', True), ('10.34.34.1', 'wlp116s0', False)]
Addresses by interfaces: {2: {'enx4ce1734ea1fa': '192.168.42.143', 'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('192.168.42.1', 'enx4ce1734ea1fa', True), ('10.34.34.1', 'wlp116s0', False)]
Addresses by interfaces: {2: {'enx4ce1734ea1fa': '192.168.42.143', 'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('192.168.42.1', 'enx4ce1734ea1fa', True), ('10.34.34.1', 'wlp116s0', False)]
Addresses by interfaces: {2: {'enx4ce1734ea1fa': '192.168.42.143', 'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('192.168.42.1', 'enx4ce1734ea1fa', True), ('10.34.34.1', 'wlp116s0', False)]
Addresses by interfaces: {2: {'enx4ce1734ea1fa': '192.168.42.143', 'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('192.168.42.1', 'enx4ce1734ea1fa', True), ('10.34.34.1', 'wlp116s0', False)]
Addresses by interfaces: {2: {'enx4ce1734ea1fa': '192.168.42.143', 'wlp116s0': '10.34.34.111'}}
Then I eventually noticed...
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
get_node_names_and_namespaces()
Interface kind: 2, info: [('10.34.34.1', 'wlp116s0', True)]
Addresses by interfaces: {2: {'wlp116s0': '10.34.34.111'}}
Remote shutdown requested
I plugged in a ethernet cable, which seemed to cause the error.
Daemon node was reset
This happens when the underlying network interfaces are changed, before calling the corresponding Node method registered on XMLRPC server.
When this happens, it will destroy the Node(DirectNode) object previously registered, and then recreate new Node(DirectNode) in NetworkAwareNode.
I think that this invalidates the all method functions registered on XMLRPC server.
probably we need to re-register method functions on XMLRPC server with new Node(DirectNode) object? i might be mistaken, comments are welcome.
CC: @ivanpauno
I think that this invalidates the all method functions registered on XMLRPC server.
That may be the case, I'm not completely sure. I don't remember this error happening before.
I am also getting this issue with the same behaviour. But previously I never had this issue when I was on ubuntu 20.04 with galactic (now I am on ubuntu 22.04 with humble). It does seem to be a network problem because I notice that this error come up more often when I try to read messages from the mqtt broker running on the same machine at the same time.
Issue is still occuring on my system. Solution at the moment is to run
ros2 daemon stop && ros2 daemon start
This seems to get the ros2 daemon back to a working state until the error appears again.
I was having this issue quite frequently on Humble as well. I have since downgraded to Foxy and I don't have any issues since.
That being said, I was not able to reliably reproduce the issue on Humble. The only way to fix it is restart the daemon or restart the computer.
From memory, I got this issue fairly frequently when I was using RTABMAP without setting up the required transformation in TF2. I was experimenting RTABMAP package, so I would launch it, kill it, repeat a couple of times. And sometimes, this error would just occur. However, I also remember my team member who is running a very simple script also had this issue happened to him once or twice.
I was using a laptop with WIFI but not connected to Ethernet, and a desktop with Ethernet only. There is no switching of network or network interface as far as I aware.
I can try to jump back into Humble and see if I can reproduce this issue with RTABMAP again in a reliable way in a few days.
https://github.com/ros2/ros2cli/pull/785 should fix the issue. @timonegk @austin-InDro @Zachaccino if you can test the patch to confirm it solves the issue that would be great, thanks!
I am getting this issue as well, The issue popped up while using ros2 service list command, I have no idea how to reproduce it as it occurred today very randomly. complete error log is provided below
ros2 service list
Traceback (most recent call last):
File "/opt/ros/humble/bin/ros2", line 33, in <module>
sys.exit(load_entry_point('ros2cli==0.18.4', 'console_scripts', 'ros2')())
File "/opt/ros/humble/lib/python3.10/site-packages/ros2cli/cli.py", line 89, in main
rc = extension.main(parser=parser, args=args)
File "/opt/ros/humble/lib/python3.10/site-packages/ros2service/command/service.py", line 41, in main
return extension.main(args=args)
File "/opt/ros/humble/lib/python3.10/site-packages/ros2service/verb/list.py", line 39, in main
service_names_and_types = get_service_names_and_types(
File "/opt/ros/humble/lib/python3.10/site-packages/ros2service/api/__init__.py", line 23, in get_service_names_and_types
service_names_and_types = node.get_service_names_and_types()
File "/usr/lib/python3.10/xmlrpc/client.py", line 1122, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python3.10/xmlrpc/client.py", line 1464, in __request
response = self.__transport.request(
File "/usr/lib/python3.10/xmlrpc/client.py", line 1166, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python3.10/xmlrpc/client.py", line 1182, in single_request
return self.parse_response(resp)
File "/usr/lib/python3.10/xmlrpc/client.py", line 1354, in parse_response
return u.close()
File "/usr/lib/python3.10/xmlrpc/client.py", line 668, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'rclpy._rclpy_pybind11.InvalidHandle'>:cannot use Destroyable because destruction was requested">
as @austin-InDro suggested I tried
Issue is still occuring on my system. Solution at the moment is to run
ros2 daemon stop && ros2 daemon start
and the issue was resolved. Is there a way to fix this, I have noticed that my ros2cli version is older than the version mentioned in this issue how can I upgrade that?
@lonebots do you use humble? https://github.com/ros2/ros2cli/pull/786 is not released as package yet, i believe this will be included in next patch release. in the meantime, something you can do is to build the source code, see https://docs.ros.org/en/humble/Installation/Alternatives/Ubuntu-Development-Setup.html
@lonebots do you use humble?
@fujitatomoya Yes, I use humble.
in the meantime, something you can do is to build the source code, see https://docs.ros.org/en/humble/Installation/Alternatives/Ubuntu-Development-Setup.html
Okay I will try this out. Thank you!