Get Entities node is extremely slow and cause Home assiatnt to disconnect

[smarthomeaf]

trafficstars

Describe the bug

A simple query with a single entity, takes 2 seconds, while a more complex one with multiple queries or using an "in group" takes up to 10 seconds and causing connectivity issues and tons of Home assistant Disconnects. I rolled back to .65 and these queries are instant. Not sure exactly what version this started at, but my guess is that this is what is causing the majority of the issues that are listed on GitHub (ie disconnect, crashes etc)

To Reproduce

a simple flow as outlined in the screenshot

Expected behavior

In older version (.65 tested) this was instant

Screenshots

Example Flow

[
    {
        "id": "dfd6fc951c155ad8",
        "type": "tab",
        "label": "Flow 1",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "70151ef02612fa4f",
        "type": "inject",
        "z": "dfd6fc951c155ad8",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 280,
        "y": 380,
        "wires": [
            [
                "1f31c4b6275e9762"
            ]
        ]
    },
    {
        "id": "1f31c4b6275e9762",
        "type": "ha-get-entities",
        "z": "dfd6fc951c155ad8",
        "name": "",
        "server": "5ea56d96.083004",
        "version": 1,
        "rules": [
            {
                "condition": "state_object",
                "property": "entity_id",
                "logic": "is",
                "value": "device_tracker.unifi_default_00_50_56_9f_bc_22",
                "valueType": "str"
            }
        ],
        "outputType": "array",
        "outputEmptyResults": false,
        "outputLocationType": "msg",
        "outputLocation": "payload",
        "outputResultsCount": 1,
        "x": 630,
        "y": 380,
        "wires": [
            [
                "19dab93174b6898f"
            ]
        ]
    },
    {
        "id": "19dab93174b6898f",
        "type": "debug",
        "z": "dfd6fc951c155ad8",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 960,
        "y": 380,
        "wires": []
    },
    {
        "id": "5ea56d96.083004",
        "type": "server",
        "name": "Home Assistant",
        "addon": true
    }
]

Environment Information

The button not in my nodered isnat and there is no way to download the diagnostic flow from your https://zachowj.github.io/node-red-contrib-home-assistant-websocket/guide/diagnostics.html#overview

Additional context

No response

[zachowj]

How many entities does your Home Assistant instance have? You can find out by running this template in the Developer Tools > Template section:

{{ states | count }}

[smarthomeaf]

7768

I can see this being a high count, but the problem did not exist 0.65

[zachowj]

Try updating to the latest release, v0.74.2, and check if there's any improvement in the output speed.

[smarthomeaf]

Looks much better, not sure it was still as fast as before as i still see a bit of a delay, but at least it is not causing nodered to crash and disconnect. The other thing I noticed that that I can use inject nodes again in the Home Assistant UI rather than having to go directly connecting to Node Red. So thank you for that as well.

Can we keep this open for a few more days to see if the issue resurfaces

[N3rdix]

I had observed weak performance for get_entities as well, but already with small groups ~20 members which took around 1 second each... Version 0.74.2 is a lot faster, back down to what feels like milliseconds 👍

[chrhen]

In my setup with 1600 entities 0.74.2 is faster than 0.73, but still far from the state before get-entities was significantly extended (0.65.1 for me).

For comparison see the CPU load of my the VM where I run HA with the Node-RED addon. Not much in the setup changed except the version of the Node-RED addon and consequently the version of the ha-websocket-contrib:

• Pre October 10: node-red-contrib-home-assistant-websocket 0.65.1 with around 8% of CPU load. Get Entities node had no noticeable performance impact iterating through 1600 entities and matching some by regex name and state every 5 seconds via inject node.
• October 10: Upgrade from ha-websocket-contrib from 0.65.1 to 0.67.2. Drastic change of CPU load to average of 25%. Unchanged node setup.
• October 30: Switch inject node to every 2 seconds, CPU load goes to 50 on same node setup.
• Late November 1: Upgrade to node-red-contrib-home-assistant-websocket 0.74.2. CPU load decreases to average of 15%, but still far from the previous 8% where the Node-RED-addon was almost no contributor to CPU load at all.

If I disable my flow that regularly calls several Get Entities nodes my CPU load is back to 8% levels.

For me the situation improved but it is still far from 0.65.1.

Note: More details on my setup in above mentioned issue.

[smarthomeaf]

I agree with @chrhen, it's acceptable, but def much slower than it used to be.

[smarthomeaf]

seems to be related, but I just noticed that action nodes with action on todo.get_items is super slow and sometimes cause nodered to crash as well.

[JanickGers85]

Really slow for me too, running the latest version (0.74.2)

[
	{
		"id": "bad59dc1fc5a6beb",
		"type": "ha-get-entities",
		"z": "76837880fe4808b5",
		"name": "Tutti gli schedule",
		"server": "70d27884.6241b8",
		"version": 1,
		"rules": [
			{
				"condition": "state_object",
				"property": "entity_id",
				"logic": "starts_with",
				"value": "switch.schedule_",
				"valueType": "str"
			},
			{
				"condition": "state_object",
				"property": "state",
				"logic": "is",
				"value": "on",
				"valueType": "str"
			}
		],
		"outputType": "array",
		"outputEmptyResults": false,
		"outputLocationType": "msg",
		"outputLocation": "entity",
		"outputResultsCount": 1,
		"x": 570,
		"y": 180,
		"wires": [
			[
				"f4788fc485325dde"
			]
		]
	},
	{
		"id": "70d27884.6241b8",
		"type": "server",
		"name": "Home Assistant",
		"version": 5,
		"addon": false,
		"rejectUnauthorizedCerts": true,
		"ha_boolean": "y|yes|true|on|home|open",
		"connectionDelay": true,
		"cacheJson": true,
		"heartbeat": false,
		"heartbeatInterval": 30,
		"areaSelector": "friendlyName",
		"deviceSelector": "friendlyName",
		"entitySelector": "friendlyName",
		"statusSeparator": "at: ",
		"statusYear": "hidden",
		"statusMonth": "short",
		"statusDay": "numeric",
		"statusHourCycle": "h23",
		"statusTimeFormat": "h:m",
		"enableGlobalContextStore": true
	}
]

[zachowj]

The latest release, 0.75.0, includes significant speed improvements for the get-entities node.

[N3rdix]

Happy to test 0.75, could you trigger a release to make it updatable in the NR palette directly?

[chrhen]

I did some testing on 0.75.0. Things seem to have improved noticeably, but we have not yet reached the old performance from before the major change of get-entities.

Analysis:

• Before 12:10 version 0.74.2 was in use.
• At around 12:10 I've upgraded to 0.75.0 and restarted Node-RED. CPU load of the complete HA VM decreased from 20% to 13% on average.
• At around 18:40 I set the incriminating flow to inject every 2 seconds versus previously every 5 seconds. Load increased to 27% as expected.
• At 19:00 I deactivated the incriminating flow, so only the background load of the HA VM with other Node-RED flows remains, which produces around 6% of load.

Conclusion: A flow with several get-entities nodes that are all queried at 5 second intervals and loop through more than 1000 entities previously produced a load of 14% alone on 0.74.2. On 0.75.2 the same flow alone produces 7% of load, which is an improvement, but still non-negligible.

Further assessment: To be fair, there's now much more querying going on since labels, areas, floors, etc. are also queried, which was not the case before. So we also have to be realistic in the expectation. However I still have some concerns using the get-entities node in its current state on less performant machines (RPi ...) on an HA installation with a sizeable number of entities.

Edit: Thanks for having put in the work to come up with a clever optimization, this is really appreciated!

[zachowj]

@chrhen In my testing, performance when checking only entity properties should be comparable between version 0.75.0 and version 0.65.0. What specific conditions are you testing in your flow? Do you have usage data from before version 0.66.0 to compare with the current results?

[chrhen]

@chrhen In my testing, performance when checking only entity properties should be comparable between version 0.75.0 and version 0.65.0. What specific conditions are you testing in your flow? Do you have usage data from before version 0.66.0 to compare with the current results?

Indeed, have a look at my post from Nov 1: CPU load of the complete HA VM including Node-RED with the same flow on 0.65.1 was around 8%. Currently I see 13% on 0.75.0.

To note that in my setup I have 8 get-entities nodes that are triggered at the same time every 5 seconds, so this is a bit of a specific scenario where the results might differ from a unit-level test that evaluates the performance of one node without looking at any form of concurrency.

I now also realize that under these circumstances it's not the ideal solution as I am trading simplicity and commonality against performance and I might have to change my model to obtain the results I am looking for.

Again, thanks for being so involved!

[github-actions[bot]]

This issue has been automatically marked as abandoned because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.