thingsboard
[BUG] RuntimeError: cannot schedule new futures after interpreter shutdown
Describe the bug Just installed latest release of the TB Gateway usign apt packet. Configured from scratch. Created OPCUA connector with no issues in the communication between TB server and Gateway but cannot TB-GW is not contacting the OPCUA server. It's not a network problem (TB-GW, TB server and OPC server are on the same lan, ping is ok, ports are reachable, no firewall). Tried with type OPCUA and also OPCUA ASYNCIO, same result. The output from "systemctl status thingsboard-gateway" are costantly saying "RuntimeError: cannot schedule new futures after interpreter shutdown" (I'll list you below the complete log) but I didn't see anything in the log files...
If I start the tb-gateway with the command
/usr/bin/python3 -c "from thingsboard_gateway.tb_gateway import daemon; daemon()"
I see the same erros below, shown in the std output.
I attack you the relevant config files tb-gateway-config.zip
Connector name (If bug in the some connector): OPC-UA and OPC-UA ASYNCIO Connector
Error traceback (If available):
2024-06-21 12:55:51 - |ERROR| - [tb_logger.py] - tb_logger - exception - 127 - cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/opcua/opcua_connector.py", line 175, in start_client
async with self.__client:
File "/var/lib/thingsboard_gateway/.local/lib/python3.12/site-packages/asyncua/client/client.py", line 94, in __aenter__
await self.connect()
File "/var/lib/thingsboard_gateway/.local/lib/python3.12/site-packages/asyncua/client/client.py", line 305, in connect
await self.connect_socket()
File "/var/lib/thingsboard_gateway/.local/lib/python3.12/site-packages/asyncua/client/client.py", line 369, in connect_socket
await self.uaclient.connect_socket(self.server_url.hostname, self.server_url.port)
File "/var/lib/thingsboard_gateway/.local/lib/python3.12/site-packages/asyncua/client/ua_client.py", line 300, in connect_socket
await asyncio.wait_for(asyncio.get_running_loop().create_connection(self._make_protocol, host, port), self._timeout)
File "/usr/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
return await fut
^^^^^^^^^
File "/usr/lib/python3.12/asyncio/base_events.py", line 1080, in create_connection
infos = await self._ensure_resolved(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/base_events.py", line 1456, in _ensure_resolved
return await loop.getaddrinfo(host, port, family=family, type=type,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/base_events.py", line 901, in getaddrinfo
return await self.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/base_events.py", line 863, in run_in_executor
executor.submit(func, *args), loop=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/thread.py", line 172, in submit
raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
Versions:
- OS: Ubuntu 24.04 LTS
- Thingsboard IoT Gateway version: 3.5.1-1
- Thingsboard server: 3.6.4-1
- Python version: 3.12
Hi @cos-gl,
Thank you for information, you are right, this issue appears if you restart the connector, could you try to use a master branch version and let us know about the results? We have added a fix. Probably, we will create a hotfix release on next week.
Hi @imbeacon, sorry for the delay in this reply. I will take a look in the next days and I will let you know. In the meantime I was using gateway on docker (as a test) and I didn't experienced this behaviour. Thanks
Hi, just updated from master branch. The gateway doesn't reconnect to the OPC server and after a complete restart (OPC server, thingsboard server, thingsboard gateway) the gateway logs shown this
{"log":"2024-06-27 06:36:36 - |INFO| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _process_config_request - 204 - Configuration update request received.\r\n","stream":"stdout","time":"2024-06-27T06:36:36.432180021Z"}
{"log":"2024-06-27 06:39:19 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - _on_disconnect - 343 - MQTT client was disconnected with reason code 7 (The connection was lost.) \r\n","stream":"stdout","time":"2024-06-27T06:39:19.589941921Z"}
{"log":"2024-06-27 06:39:49 - |INFO| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 352 - MQTT client \u003cpaho.mqtt.client.Client object at 0x72fc227f44d0\u003e - Connected!\r\n","stream":"stdout","time":"2024-06-27T06:39:49.997990973Z"}
{"log":"2024-06-27 06:39:49 - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 236 - Subscribed to *|* with id 2 for device *\r\n","stream":"stdout","time":"2024-06-27T06:39:49.998595612Z"}
{"log":"2024-06-27 06:42:05 - |INFO| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _process_config_request - 204 - Configuration update request received.\r\n","stream":"stdout","time":"2024-06-27T06:42:05.75746973Z"}
{"log":"2024-06-27 06:42:05 - |ERROR| - [tb_gateway_service.py] - tb_gateway_service - __connect_with_connectors - 872 - ['33089995-5dbc-40a9-ac72-c9a4165ac8f6'] Error on loading connector 'test_connector_GL': RuntimeError(\"There is no current event loop in thread 'Remote Request Processing'.\")\r\n","stream":"stdout","time":"2024-06-27T06:42:05.762057989Z"}
{"log":"2024-06-27 06:42:05 - |ERROR| - [tb_logger.py] - tb_logger - exception - 127 - There is no current event loop in thread 'Remote Request Processing'.\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767821097Z"}
{"log":"Traceback (most recent call last):\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767847074Z"}
{"log":" File \"/thingsboard_gateway/gateway/tb_gateway_service.py\", line 857, in __connect_with_connectors\r\n","stream":"stdout","time":"2024-06-27T06:42:05.76785169Z"}
{"log":" connector = self._implemented_connectors[connector_type](self,\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767855207Z"}
{"log":" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767870459Z"}
{"log":" File \"/thingsboard_gateway/connectors/opcua/opcua_connector.py\", line 81, in __init__\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767873497Z"}
{"log":" current_loop = asyncio.get_event_loop()\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767875648Z"}
{"log":" ^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767935449Z"}
{"log":" File \"/usr/local/lib/python3.11/asyncio/events.py\", line 681, in get_event_loop\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767939309Z"}
{"log":" raise RuntimeError('There is no current event loop in thread %r.'\r\n","stream":"stdout","time":"2024-06-27T06:42:05.768140747Z"}
{"log":"RuntimeError: There is no current event loop in thread 'Remote Request Processing'.\r\n","stream":"stdout","time":"2024-06-27T06:42:05.768157543Z"}
I really don't understand the meaning... The telemetry wasn't collected since yesterday (when I restarted TB server and TB gateway).
EDIT: after a TB gateway service restart the connection with the OPC server started and telemetry is now collected (but I'm sure I restarted the gateway also before, I completely rebooted the machine)
We have improved OPC-UA connector and in the latest version it works more stable, please try to use it.