Huge amount of data in Home Assistant history database

[igogold]

The bug description:

When state of climate entity of a Smart Thermostat changes it writes to the history database. The current state and all attributes are saved. But internally recorder integration try to deduplicate attributes set between records, saving all attributes set in the dedicated table and a link inserted to it in the record of state changes. But your integration uses many attributes for debugging/technical use and every state update creates a new set of attributes, so database grows quickly. It was found in the issue about long history database purging. See comment https://github.com/home-assistant/core/issues/70309#issuecomment-1110559335

To Reproduce

Nothing need to do explicitly, it happens all time.

Expected behavior

Create additional sensors instead of frequently changed attributes. Correct usage of attributes described in HA docs: https://developers.home-assistant.io/docs/core/entity/?_highlight=extra_state_attributes

Versions:

• Integration version: 2022.2.1
• OS: Home Assistant OS 7.6
• Home Assistant: core-2022.4.7

[ScratMan]

Hello, that's something I suspected as I noticed a performance issue on my raspberry. Unfortunately, I have no time for development now due to family life and heavy workload at office at the moment 😟. If someone could implement the required changes and make a pull request it would greatly help.

[T81]

@igogold @ScratMan if you can point me in some direction with examples I can try and refactor the code accordingly

[T81]

I think I got it. Issue is at https://github.com/ScratMan/HASmartThermostat/blob/master/custom_components/smart_thermostat/climate.py#L638 as @igogold mentioned in https://github.com/home-assistant/core/issues/70309#issuecomment-1110559335 where a lot of entries need to be converted to sensor entities. @ScratMan can you give me a sample of one of these entries converted to sensor entity?

[ScratMan]

Yes, that's it. The extra state attributes I added are useful for debugging and tuning the PID coefficients, so we should remove them from extra states and create sensor entities instead, so that it can be activated on demand in HA. It requires to add a sensor.py file alongside the climate.py ; but I don't know yet how to create both climate and sensors entities together and have the sensors reporting values from climate's properties. I planned to get inspiration from some other integrations using the same principle (I looked at https://github.com/jeroenterheerdt/HAsmartirrigation as I already used this integration).

[T81]

I see, so comparing how extra states are implemented between the two codes it's a good start, I suppose

[ScratMan]

I see, so comparing how extra states are implemented between the two codes it's a good start, I suppose

I quickly looked at how it's done, and in fact the init.py should create a coordinator that will interface with both climate entity and additional sensors, grabbing properties and variables that are in extra states attributes from the climate today, to expose them into dedicated sensors that could be kept inactive by default and that can be enabled directly in HA UI. That's a big architecture rework, but it should be simple once the coordinator is ready.

[ScratMan]

I could get some time to work on this, but unfortunately I could not figure out how to make the integration create both the climate entity and additional sensors entities getting their state from the climate entity. I tried to use the detailed_hello_world_push and example_load_platform examples from example-custom-config but it would either lead to no YAML configuration passed to the climate platform, or the platform loading called twice (first with the correct config, and then with no config, so second call crashes).

I finally ended adding a debug boolean parameter that defaults to false to be able to only have the debugging attributes in the extra state when needed : https://github.com/ScratMan/HASmartThermostat/blob/a7c2e30d701ae98d18a52a7f20b9dae11a91b905/custom_components/smart_thermostat/climate.py#L566-L615 I kept the attributes that should be restored from previous state upon reboot (as they can be updated using services), and the control_output that is needed for some users with analog valves. @bdraco Would this be enough to reduce the database footprint ?

[jasperweiss]

I just installed smart thermostat but can’t find any attributes. Did you merge this already by any chance? Or am I looking in the wrong spot

[ScratMan]

I just installed smart thermostat but can’t find any attributes. Did you merge this already by any chance? Or am I looking in the wrong spot

The changes are still to be released. So there should be no difference for the moment, attributes should be visible in the states development tools of HA.

[jasperweiss]

Ah, I see.. Found it. Thanks! Usually you can find attributes on the info tab of an entity. Thanks for making this, it's much appreciated 😃