[Feature request]: separated "last_seen" topic

[emandtf]

Is your feature request related to a problem? Please describe

In Homeassistant all Entities/Sensors have their own "last_updated" and "last_changed" values. The last one is updated when the value changes and the first one also if it remains the same as before. Those are usually enough to determinate if a device/sensor is still working, but.....not always.

Unfortunately many Entities/Sensors doesn't have a valid update to be published to their respective topics: for example a "MiBox Zigbee Dimmer" doesn't publishes any value if its light remains at the same state and at the same brightness level. But I would like to know if the device is ONLINE besides the need to wait for each "availability" timeout....and many of them (usually the powered/online ones) sends something like a "ping" that could signal that they are online but without publish any useful data/changes that HA can store/use.

A solution could be achieved by enabling Z2M "last_seen" BUT on Homeassistant each different received payload is stored in a separated record in the database (so HA can store all possible values and reuse them, which are usually big Strings, without the need to store the whole String at every new value even if it's the same of a previous one), so if the JSON key "last_seen" changes of few milliseconds, a new record in the database will be stored containing the big whole String. This affects Database size A LOT due to big JSON payloads usually published, especially if the device is sending data very often but which remains most of the times as the same value (like a Temp or an AC voltage/current).

Describe the solution you'd like

A solution could be: publish Devices "last_seen" values BUT in a separated topic like under the "/bridge" one or something like that. A single separated topic/subtopic that contains a combination of "device_name: last_seen". In this way someone could read data from this topic to know the last published message from each device without the bad Homeassistant behavior.

Describe alternatives you've considered

I tried to use "last_seen" option and it works, but Homeassistant Database grows 100MB per-day due to the behavior I described before.

Additional context

None

[github-actions[bot]]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days

[emandtf]

Up

[github-actions[bot]]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days

[toine512]

Hi. Do you know you can set an availability timeout per device? And #23910 would be a nice enhancement.

[emandtf]

Hi. Do you know you can set an availability timeout [per device]?

Unfortunately it has nothing to do with what I described here. The "problem" it's not the possibility to enable Availability per-device or per-type (battery or AC power) but the fact that in BOTH of these types the LAST_SEEN timestamp is added in the payload. Obliviously this is the expected behavior but, as I written, Homeassistant (starting from 1 year ago I think) stores any "not yet received" payload in a separated record and uses its record ID for the reference. So if a bulb sends this:

{"state": "on", "last_seen": "2025-01-01 10:09:08"}

a new record with ID (I.e) 15 is stored in Homeassistant. When a second "state: ON" payload is sent HA tries to reuse a previously saved ID to avoid to re-store JSONs already present, but OBLIVIOUSLY the new payload has a different "last_seen" value and the old "state: ON" record ID (the one already stored in DB) cannot be reused due to the hash/checksum of it is differs from the newer just received.

This generates the issue I described: tons of similar records that has only the "last_seen" part different from each others.

I have some Zigbee Tuya mmWave sensors that sends data every 1 second 24h/24 7d/7 and I had dozen of thousand records of similar JSONs stored that differs only from "last_seen" value. So I choosed to disable "last_seen" feature at all. My Homeassistant Database size growth by 70MB per-day due to this behavior (and because mmWave sensors have a big payload). Turning last_seen OFF was THE solution but obliviously I miss it in some situation.

[toine512]

I'm pointing out that you can use availability with last_seen disabled for the specific use case you mentioned. Indeed it doesn't do anything to help you using last_seen.

(Why the f is Home Assistant storing MQTT messages at all? Forever???)

[emandtf]

Why you think to have "right" to tell someone that "it doesn't do anything to help using last_seen"? Do you already know all the possibilities and needs of all persons? Why have I to explain WHY I need last_seen if it has nothing to do with this "feature request"? Those are two different things and, morover, most of normal Homeassistant users doesn't know anything about HOW it stores data in database but complains why their database grows so fast with just few sensors....and this "unwanted behavior" is the culprit in most cases. There isn't written anywhere that HA works like I analyzed and neither Z2M warns with something like "enabling last_seen could give unexpected DB grow in your Homeassistant".

So adding a separated Topic where someone could extract "last_seen" value will help BOTH to take DB size small and who needs "last_seen" info.

[toine512]

Why you think to have "right" to tell someone that "it doesn't do anything to help using last_seen"?

That's not what I wrote.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

[jamesmyatt]

(Why the f is Home Assistant storing MQTT messages at all? Forever???)

The HA recorder database (https://www.home-assistant.io/integrations/recorder/) records the attributes of all of the state updates within the history interval (default = 10 days); this drives the entity history in the Home Assistant UI. It deduplicates the values, but only at the level of the attributes dictionary rather than individual attribute values. Z2M's sensors are configured to take the whole message as the attributes dictionary (https://www.home-assistant.io/integrations/sensor.mqtt/#json_attributes_topic) hence, each Z2M message results in a unique attributes dictionary when "last_seen" is included.

Although, I'm not 100% sure if the discovery configuration message has changed in Z2M v2.

[emandtf]

The only "solution" I found was this:

1. configure HA Recorder to NOT record any Z2M entity
2. manually create a Template/Helper Sensor to replicate all MQTT/Z2M attributes and states except for "last_seen"
3. manually create a Template/Helper Sensor for each "last_seen" attribute (not Recorded as well)

In this way "last_seen" is well available at HA-level and only few Z2M JSONs are recordered to Database. It's a bit tricky but it works....however this makes MQTT entities twice in HA and them could be A LOT.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days