Socket extreme high latency

[TonKro]

Describe the bug

The latencey from HA to Node-Red is after some Node-Red restarts exteremely slow. Sometimes it responds seconds (5-20) later.

Running HA in a docker and Node-Red directly installed.

Can someone help me how to debug if this is unkown?

To Reproduce

Only some restarts are needed

Expected behavior

Low latency

Screenshots

No response

Example Flow

No response

Environment Information

Version:

Home Assistant version: 2023.12.4 Companion version: 3.1.2

Node-RED version: 3.0.2 Docker: no Add-on: no

Node.js version: v16.20.1 OS: Raspbian Bullseye

Additional context

No response

[TonKro]

No one does have this issue?

Any help how to debug?

[zachowj]

You might have a flow that is causing the slowdown. Try disabling all the flows and then enable one tab at a time.

[TonKro]

Node red itself is very fast. When hit any inject the result is directly there. Only the websocket is extremely slow. When hit a switch sometimes it takes seconds. This sounds to me not it is caused by node red but by the websocket.

[zachowj]

import

[{"id":"d4020dde040b9344","type":"debug","z":"120358abd7c22d30","name":"debug 164","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1002,"y":6608,"wires":[]},{"id":"6e9b73a713fdd24a","type":"server-events","z":"120358abd7c22d30","name":"","server":"","version":3,"exposeAsEntityConfig":"","eventType":"state_changed","eventData":"{\"entity_id\": \"light.bedroom\"}","waitForRunning":true,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"eventData"},{"property":"time","propertyType":"msg","value":"","valueType":"date"}],"x":804,"y":6608,"wires":[["d4020dde040b9344"]]}]

change the entity id to a valid one in your setup. Then trigger a change in that entity and compare time_fired to time. You can click on the time value in the debug window and it will convert it to an easier format to compare with time_fired.

time_fired is the time Home assistant recorded the change and time will be when NR received the event.

[TonKro]

Great, going to test it. Will update you here.

[TonKro]

It seems it is not possible to measure it like this. In HA when turn the switch, nothing happens for seconds and then does change. The time of the switch is also forwarded to NR:

{"payload":{"event_type":"state_changed","entity_id":"switch.nr_timing_test","event":{"entity_id":"switch.nr_timing_test","old_state":{"entity_id":"switch.nr_timing_test","state":"off","attributes":{"icon":"mdi:electric-switch-closed","friendly_name":"NR Timing TEST"},"last_changed":"2024-02-10T12:20:30.154588+00:00","last_updated":"2024-02-10T12:20:30.154588+00:00","context":{"id":"01HP9FCHSTGSWNQE9V92BH8FGK","parent_id":null,"user_id":"da06c29453e0415f8b7e60d6ad8c191d"}},"new_state":{"entity_id":"switch.nr_timing_test","state":"on","attributes":{"icon":"mdi:electric-switch-closed","friendly_name":"NR Timing TEST"},"last_changed":"2024-02-10T12:39:42.804123+00:00","last_updated":"2024-02-10T12:39:42.804123+00:00","context":{"id":"01HP9GFK65MEJD4N783MGDN45N","parent_id":null,"user_id":"da06c29453e0415f8b7e60d6ad8c191d"}}},"origin":"LOCAL","time_fired":"2024-02-10T12:39:42.804123+00:00","context":{"id":"01HP9GFK65MEJD4N783MGDN45N","parent_id":null,"user_id":"da06c29453e0415f8b7e60d6ad8c191d"}},"time":1707568783386,"_msgid":"aabe69e23f3a0601"}

"time_fired":"2024-02-10T12:39:42.804123+00:00 NR Time: 1707568783386 -> 13:39:43.386

This is just 0,5 second.

[zachowj]

The delay you're seeing is not the connection between HA and NR but the time it takes for the device to update HA that it has changed state.

[TonKro]

I do not understand what you mean exactly. Both are on the same device running. Node red as application and home assistant in a docker container.

Where should the delay be? For this test I did use your example. So there is no device on the end just the NR component from your example.

[mgiggs]

Hi, I'm also getting quite high latency between Node Red and Home Assistant.

The things I have done to confirm this is the issue:

• Move node red to separate docker rather than HA Add-on
• Disable above node-red and only run add-on with single flow, no automations in flow, only inject and service calls vs mqtt calls.
• Disable reverse proxy as I was seeing "Client unable to keep up with pending messages." api errors in logs from companion apps (separate issue) so only access is via internal ip which nothing uses or has access to.
• Compare timing turning on and off a light using HA Service call vs mqtt message with both an inject node and from a zigbee button using zigbee2mqtt
• Compared to zigbee button using HA Automation

So I feel I've excluded many variables and it comes down to NR accessing HA through the websocket.

With the zigbee button triggering an mqtt messgae, the light is essentially instant, but using the light service call via HA, it takes approximately a second give or take, does not sound like much but the delay is incredibly noticeable.

[mgiggs]

Further to my previous post, I have run some other tests.

I setup another Home Assistant and connected the same non-addon Node Red to it. Testing a basic light toggle with this Home Assistant was instant!

So I was thinking it is the websocket on my home assistant that is the issue.

I have then setup Postman and connected to the websocket of my main Home Assistant and testing a basic light toggle is instant.

So there is no latency in calling a service through the websocket API for my Home Assistant but there is latency between Node Red and my main Home Assistant but not a brand new empty one.

So my best conclusion is that it must be the subscribing (or similar) to state changes and events from Node Red combined with the volume of states etc in my Home Assistant that is causing the latency. I do not believe that it is the nodes in node red that are looking for state changes etc as I previously tested a separate node red with only 2 flows and just re-validated by disabling all but one flow and got the same latency.

Happy to provide any further info, but it would seem that the latency is only between NR and HA and most likely because I have such a large HA with so many events and state changes.

[TonKro]

Is there any way to debug it? I see delays of more then 10 seconds regularly.

[mgiggs]

I still have this issue.

I've confirmed that it is my current Home Assistant config combined with the Node Red websocket use only.

Fresh install of Home Assistant, Node Red is fine.

Any other websocket API use (eg Postman), responses are instant.

Even triggering a light on using Node Red via MQTT -> Home Assistant automation -> Service call is close to instant compared to the 5s+ it takes for Node Red to call HA through the websocket.

My Home Assistant runs on 4 cores of an i5-9600 with a max of 15% CPU usage along with 10Gbe network between Node Red and HA and ~0.2ms ping latency so I feel environment is ruled out.

I'm left with having to rebuild my Home Assistant from the ground up to try and solve this but I would love advice on how to troubleshoot otherwise. It is like the way node-red connects to the websocket is flooded with background traffic or similar based on the size of my home assistant implementation.

[github-actions[bot]]

This issue has been automatically marked as abandoned because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.