Trend sensor too brittle on HA restart because it has no minimum duration option

[melyux]

The problem

The Trend sensor's sample_duration and max_samples options let you track "long-term" trends (meaning anything trend involving more than 2 sample points). However, all samples are deleted upon HA restart, and the sensor starts trying to fit trends the first 2 samples it gets and often "detects" a trend, which defeats the whole purpose of setting these options.

A min_duration attribute is required on the Trend sensor to make the existing options make sense. In fact, the current options are not very useful. It's certain that most people using the Trend sensor with multiple samples want the trend to be over a certain amount of time, nothing more and nothing less. Otherwise, the findings would be quite useless. A "trend" involving multiple samples that meets a required change gradient tells you nothing unless you know over how long of a period that trend is happening.

The ideal solution: The sample_duration and max_samples options should be removed entirely because a binary sensor tracking trend just doesn't make sense over a variable amount of time. They should be replaced with a single, optional duration. If it's not set, then the Trend will just track trends between the current state and the immediately preceding state, as is already the case. If it is set, then the Trend will track the trend over the set duration, and will not falsely fire when the required length of samples is unavailable (like when HA restarts).

The temporary, non-breaking solution: Add a min_duration option that will add to the effect of the existing sample_duration and max_samples options by simply mandating that the binary sensor not fire when the collected data does not span the minimum duration specified (like when HA restarts). This is a stopgap solution and a no-brainer.

Environment

• Home Assistant Core release with the issue: 0.114
• Last working Home Assistant Core release (if known): N/A
• Operating environment (OS/Container/Supervised/Core): Container
• Integration causing this issue: Trend
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/trend

Problem-relevant configuration.yaml

  - platform: trend
    sensors:
      outside_suddenly_cool:
        entity_id: sensor.outside_temperature
        friendly_name: "Outside Suddenly Cool"
        device_class: cold
        sample_duration: 1200
        max_samples: 20
        min_gradient: -0.00241

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[melyux]

Nah not stale yet

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[freakshock88]

I'm also still affected by this issue.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[melyux]

No change in behavior. Would at least like to have some discussion on the question: do we use the better, more breaking solution by switch simply to a single duration? Or do we have to do a non-breaking change first? What's the roadmap for switching to the better solution?

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[melyux]

No.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[melyux]

No, no change in behavior.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[melyux]

No

[slovdahl]

I'm also having a similar issue. As it stands, I can't set min_gradient as low as I would like to, because a restart combined with a tiny bump in the value of the source sensor would trigger the trend sensor. Instead, it means the sensor won't trigger every time it should (which in my case is still better than the opposite, triggering when it shouldn't, but still not ideal).

The simple solution to my problem would be a "minimum samples" setting. Alternatively, read history from the recorder and backfill the samples according to sample_duration and max_samples before evaluating it the first time after a restart.

[lbouriez]

Also looking for a solution on this one, every reboot makes my trend sensor unknown for a moment...

[melyux]

I've switched to using the new statistics sensors that do exactly same thing but take into account historical database data even after restarts. HA really needs to stop making HA restarts affect things; they should be ignored for all intents and purposes. In the meantime though, the trend integration is basically dead in the water with the new stats stuff.

[lbouriez]

I've switched to using the new statistics sensors that do exactly same thing but take into account historical database data even after restarts. HA really needs to stop making HA restarts affect things; they should be ignored for all intents and purposes. In the meantime though, the trend integration is basically dead in the water with the new stats stuff.

Ho thx, i will have a look on it !

[slovdahl]

I've switched to using the new statistics sensors that do exactly same thing but take into account historical database data even after restarts. HA really needs to stop making HA restarts affect things; they should be ignored for all intents and purposes. In the meantime though, the trend integration is basically dead in the water with the new stats stuff.

Interesting, thanks for the tip, I hadn't thought about using https://www.home-assistant.io/integrations/statistics/ for this.

Just out of curiosity, how do you use it as replacement for a trend sensor? I guess a combination of a statistics sensor with state_characteristic: change_sample or state_characteristic: change_second and a template binary sensor that returns true if the statistics sensor is greater than the min_gradient from the trend sensor could yield the same behaviour at least. 🤔

[lbouriez]

I've switched to using the new statistics sensors that do exactly same thing but take into account historical database data even after restarts. HA really needs to stop making HA restarts affect things; they should be ignored for all intents and purposes. In the meantime though, the trend integration is basically dead in the water with the new stats stuff.

Interesting, thanks for the tip, I hadn't thought about using https://www.home-assistant.io/integrations/statistics/ for this.

Just out of curiosity, how do you use it as replacement for a trend sensor? I guess a combination of a statistics sensor with state_characteristic: change_sample or state_characteristic: change_second and a template binary sensor that returns true if the statistics sensor is greater than the min_gradient from the trend sensor could yield the same behaviour at least. 🤔

It's less intuitive than the trend sensor unfortunately and I am still playing with it. For now, I use it like this:

---
platform: statistics
name: "main_bathroom_humidity"
entity_id: sensor.group_humidity_bathroom
state_characteristic: change
max_age:
  minutes: 15
sampling_size: 150
precision: 1

So for example, if my humidity sensor sensor.group_humidity_bathroom go from 50 to 55% in 2 min, my sensor.main_bathroom_humidity (the statistics sensor) will have a value of 5.

And I plugged my automation on this;

...
trigger:
    entity_id: sensor.main_bathroom_humidity
    above: 2
...

[slovdahl]

Ah, thanks! Way easier than trying to emulate what the trend sensor does 😁

[melyux]

I use this Statistics sensor to find the trend for outside temperature:

  - platform: statistics
    name: "Outside Temperature Trend"
    entity_id: sensor.outside_temperature
    state_characteristic: change_second
    sampling_size: 50
    max_age:
      minutes: 26 # 25 minutes, but upped to avoid situation where the immediately preceding 25-minute period has only 49 samples (which is always the case b/w any two measurements)
    precision: 6

I then use this Threshold sensor based upon it to replace the trend for outside temperature suddenly becoming cold:

  - platform: threshold
    name: "Outside Suddenly Cool"
    entity_id: "sensor.outside_temperature_trend"
    lower: -0.001705 # -0.00241 is the lower-than threshold
    hysteresis: 0.000705
    device_class: cold

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[melyux]

Shouldn't be stale. The trend sensor should probably be removed in favor of the newer and more capable statistics integration, however.

[drpump]

If we are to keep the trend sensor, I would suggest having either duration (original request) or min_samples. A trend of two samples is impossible to work with for sensors that have some degree of inaccuracy. For example, I’m using BLE beacons to measure distance and want to know when something is approaching vs leaving. BLE beacon measurements bounce around a bit initially before stabilising.

I will probably use the statistics integration but it is also missing a minimum samples configuration option, so I need to explicitly check the count before acting on the sensor value.

[UmairEEE]

I have configured a trend sensor for water level. I want it to alert me when distance decreases like 0.04 meter over a period of 25 minutes. Issue is that trend sensor sometimes turns on with only 2 samples (error in ultrasonic reading of 0.01 meter in 1 minute) ignoring my setup of 0.04 meter in 25 minutes.

- platform: trend sensors: water_incoming: entity_id: sensor.tank_level_meter sample_duration: 3600 max_samples: 60 min_gradient: -0.000026

Edit: Following are the screenshots when it happened at 02:58:11 at night. When i checked in the morning sample count was not going above 2 and trend sensor was still ON until i restarted HA then it changed back to OFF.

UPDATE: Increasing decimal accuracy coming from Ultrasonic sensor seems to have fixed this issue. IMO trend sensor sample count does not go up if value remain same.

[UmairEEE]

I am still having this issue of Trend sensor turning ON sometimes upon HA restart.

[slovdahl]

I am still having this issue of Trend sensor turning ON sometimes upon HA restart.

I have never had that problem, until I updated from 2023.5 to 2023.6. Now it happens almost every restart :thinking:

[UmairEEE]

Just updated HA (to 2023.7.3). After the update HA restarted, trend sensor turned on for a minute then turned off again. It's really annoying because I have some automations for notifications. Waiting for a fix for this issue.

[Mrten]

patch for min_samples seems pretty trivial

[drpump]

#100332 doesn’t really solve this problem. The trend sensor will still trigger off just 2 samples, which is often insufficient as per various examples described above. Could you please re-open? Thanks.

[jpbede]

Will add min_samples later this day.