Suddenly unable to send data

[RobWPink]

Component

• Generic

Description Suddenly, all attempts to send data to devices via MQTT began to timeout. I checked the log and saw this on repeat:

2023-05-30 13:26:42,235 [DefaultTelemetryWebSocketService-57-1] WARN  o.t.s.c.plugin.TbWebSocketHandler - [e9376ff3-f0ed-46f1-8bd0-dab76bf0c71d] Failed to find session by external id
2023-05-30 13:26:42,239 [http-nio-0.0.0.0-8080-exec-2] INFO  o.t.s.c.plugin.TbWebSocketHandler - [3b4f58c0-c435-11ed-8f9a-b925c845f9fa][e9376ff3-f0ed-46f1-8bd0-dab76bf0c71d][10666b05-de0b-aa9e-490b-1bd4c6abfa90] Session is closed
2023-05-30 13:26:44,903 [TB-Scheduling-2] INFO  o.t.server.actors.ActorSystemContext - Rule Engine JS Invoke Stats: requests [74] responses [0] failures [0]
2023-05-30 13:26:44,903 [TB-Scheduling-1] INFO  o.t.s.s.s.DefaultTbEntityDataSubscriptionService - Stats: regularQueryInvocationCnt = [261], regularQueryInvocationTime = [3028], dynamicQueryCnt = [0] dynamicQueryInvocationCnt = [0], dynamicQueryInvocationTime = [0], alarmQueryInvocationCnt = [5], alarmQueryInvocationTime = [32]
2023-05-30 13:26:45,500 [http-nio-0.0.0.0-8080-exec-10] INFO  o.t.s.c.plugin.TbWebSocketHandler - [3b4f58c0-c435-11ed-8f9a-b925c845f9fa][756ecb5a-dbf2-404a-8339-3afd19100ac8][9046d879-5cf0-12ae-8a06-cc83ea0607c1] Session is opened from address: /XX-XXX-XXX-XXX:62455
2023-05-30 13:26:47,671 [sql-log-1-thread-1] INFO  o.t.s.dao.sql.TbSqlBlockingQueue - Queue-1 [TS] queueSize [0] totalAdded [22] totalSaved [22] totalFailed [0]
2023-05-30 13:26:47,676 [sql-log-1-thread-1] INFO  o.t.s.dao.sql.TbSqlBlockingQueue - Queue-2 [TS] queueSize [0] totalAdded [142] totalSaved [142] totalFailed [0]
2023-05-30 13:26:47,736 [sql-log-1-thread-1] INFO  o.t.s.dao.sql.TbSqlBlockingQueue - Queue-1 [TS Latest] queueSize [0] totalAdded [22] totalSaved [22] totalFailed [0]
2023-05-30 13:26:47,737 [sql-log-1-thread-1] INFO  o.t.s.dao.sql.TbSqlBlockingQueue - Queue-2 [TS Latest] queueSize [0] totalAdded [140] totalSaved [140] totalFailed [0]
2023-05-30 13:26:47,929 [sql-log-1-thread-1] INFO  o.t.s.dao.sql.TbSqlBlockingQueue - Queue-2 [Attributes] queueSize [0] totalAdded [143] totalSaved [145] totalFailed [0]
2023-05-30 13:26:49,421 [DefaultTelemetryWebSocketService-22-6] WARN  o.t.s.c.plugin.TbWebSocketHandler - [756ecb5a-dbf2-404a-8339-3afd19100ac8] Failed to find session by external id

For reference XX-XXX-XXX-XXX is the IP that the aws ssh client shows I'm connecting from when I start an ssh session. And the port in the message seems to be increasing per loop, the amount it increases isnt consistnat but it seems to mostly increase by 2. Ive restarted EC2, the connected device, and the thingsboard service.

Any help would be greatly appreciated, I'm not sure where to go from here.

Environment

• OS: AWS EC2 Ubuntu 20.04.06 LTS
• ThingsBoard: 3.4.4
• Browser: chromium

[RobWPink]

it solverd itself breifly, until a reboot caused it to happen again,

[xiddjp]

I meet the same problem..

[pon0marev]

The log file you attached does not include any messages about a mqtt transport issue. If this issue still occurs, add advanced logging to the mqtt transport and attach the log file. Modify the logback file /usr/share/thingsboard/conf/logback.xml as follows:

<logger name="org.thingsboard.server.transport.mqtt" level="TRACE" />

[pon0marev]

The issue has been closed due to inactivity. If you have additional questions or comments, you can reopen it.