Derivative integration gives incorrect values when source does not change

[EDelsman]

The problem

The derivative of a constant value is 0: https://owlcation.com/stem/The-Derivative-of-a-Constant-With-Examples

However, the derivative integration only updates if the source value changes. Therefore, the derivative integration is never 0, even if the source stays the same for a very long time. This makes it impossible to see when the source is unchanging. Instead the derivative indicates a constant rising or falling value, which isn't true.

In other words: the progressing of time, without changes to the source, should be considered as a possible change of the derivative too. The derivative should change towards zero while time progresses without changes to the source.

What version of Home Assistant Core has the issue?

2022.11.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

derivative

Link to integration documentation on our website

https://www.home-assistant.io/integrations/derivative/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

Hey there @afaucogney, mind taking a look at this issue as it has been labeled with an integration (derivative) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of derivative can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Change the title of the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign derivative Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

derivative documentation derivative source _{^{(message by IssueLinks)}}

[afaucogney]

Hello, thanks for posting your question. Why a source could not update the value with the same numerical value ?

Please post a screenshot of the source and it's derivatives signal to better understand your issue. Thanks.

[EDelsman]

Sensors could post more values, but most battery ones usually don't when there's no change. A.f.a.i.k. template sensors also do not store any data if there's no change. I would not know how to force them to.

[EDelsman]

I've been thinking: a regular sensor and/or sensor graph assumes the value is unchanged if no new sensor update is known. Most sensors "expect" this behaviour: they send frequent updates when the value changes quickly, als slow (or no) updates when the value is stable.

For derivative,this assumption is currently the same: no new updates means derivative stays the same. But when the derivative is non-zero, that assumption clases with the above assumption.

For derivatives, if the real value is stable, the derivative changes toward 0. Basically, what it should periodically do is recalculate, assuming the value now() is equal to the last known value of the original sensor - the same as the assumption above. When you do, you'll see the derivative change.

This could be optimized because when the derivative is already 0, the change of time is irrelevant, because the derivative is in line with the assumption of a stable base sensor.

Trend sensors will likely suffer from the same wrong assumption, as they internally depend on a derivative.

[Petro31]

A proper way to handle long pauses within the HA paradigm is to schedule an update in x seconds. This update would only occur if isn't a natural update that occurs within those x seconds. Treat it as a fallback so that it does not impact sensors that have a periodic update frequency. The calculation would remain unchanged as the time window would still be honored, i.e. change/time.

This would most likely require a new optional field for the schema. The default would be none to keep backward compatibility. None would not add a next-scheduled-update. (1 through N) would add a next-scheduled-update in (1 though N) seconds from the last update. If no update was received between the last+update and next-scheduled-update, an update will fire at next-scheduled-update's time. Lastly, every update would reset next-scheduled-update.

Going this route would keep current functionality while adding granularity to the derivative calculations.

[EDelsman]

That sounds like a good way to solve it. But I am curious why you think the old behaviour would be beneficial to anyone. If sensors update regularly, the fix wouldn't change a thing. And if it does not, why would anyone want to keep the old behaviour? Because now the value of the sensor is determined by what happened prior to the long pause.

[Petro31]

That sounds like a good way to solve it. But I am curious why you think the old behaviour would be beneficial to anyone. If sensors update regularly, the fix wouldn't change a thing. And if it does not, why would anyone want to keep the old behaviour? Because now the value of the sensor is determined by what happened prior to the long pause.

Because we all have sensors that constantly update, I don't need this. My temperature and humidity update on a 30 second basis and the resolution is large enough that values are always different. It provides the correct result.

[ildar170975]

Could be related: https://github.com/home-assistant/core/issues/84088

[neil1111]

I’m having a similar problem. I setup a derivative to track temperature changes from an Aotec sensor. The temp is changing, but the derivative never does. See the following charts for both:

Here are the settings:

[finity69x2]

Every person who uses a derivative sensor should expect that the derivative sensor should react as a true derivative.

By definition a derivative is a rate of change.

If the source sensor never changes then the derivative by definition is 0.

The derivative sensor shouldn't need to see a change in the source sensor for it to re-calculate. if the source doesn't change then the derivative sensor should be 0.

I created a test setup to see what the true behavior of the sensor is.

Here are the results:

the source sensor stopped updating at 3:38 PM:

the derivative sensor stayed at the same value ever since:

the derivative sensor never goes to 0 as expected when the source stops changing.

Any user who actually understands how a derivative is supposed to work would be confused by the current behavior.

we shouldn't need to create a work around for the incorrect behavior of the derivative sensor.

[flo-wer]

In https://github.com/home-assistant/core/issues/67627#issuecomment-1374407886 I wrote a detailed comment about both the derivative integration and the statistics integration that both provide the same derivative function. Both implementations have the same bug that a value of 0 is never reached but with different origin.

[sophof]

[snip]

we shouldn't need to create a work around for the incorrect behavior of the derivative sensor.

It's off topic but I still felt the need to react. I really don't like the condescending tone of your reaction, it reads as if you expect people did this almost out of spite or some absurd misunderstanding of what the definition of a derivative is. I'd suggest starting from the assumption that people not only tried their best but that any mistake is an honest mistake and not because they are idiots. Maybe they even know something you don't (like that it isn't trivial/automatic within HA to implement that a "sensor shouldn't need to see a change in the source sensor for it to re-calculate" unless you go full polling which you generally want to avoid).

Not just that, your reaction is in a issue where everyone including the code owner agrees it happens, that it shouldn't happen and people are actively trying to fix it in the best way possible.

You also seem to be overstating how easy it is to spot edge cases like this one. Most sensors will report with some frequency and most sensors will report values that trend to a zero derivative before being stable. At least this is the case for all my sensors where I use it for. Therefore the only way to notice - at least for me - is an issue like this one. It might have been realized earlier, each specific edge case on its own seems obvious in hindsight when you notice them, but take them all together and you'll always miss a few. This is also obvious from the fact that the statistics sensors shares this bug.

I don't understand why you reacted at all to be honest, all you added here was a sour taste for someone who contributed to this integration (speaking for myself) and a general condescending rant.

[neil1111]

I’m having a similar problem. I setup a derivative to track temperature changes from an Aotec sensor. The temp is changing, but the derivative never does. See the following charts for both:

Here are the settings:

Hi. I’m bumping the above post from last month, please. My derivative is always 0, even when the underlying sensor is changing. Can someone please help me with the configuration? Thanks very much!

[finity69x2]

I really don't like the condescending tone of your reaction

Sorry that you took my reply as condescending.

It was just meant to clarify and specify what the problem was.

it reads as if you expect people did this almost out of spite

not at all. I honestly fail to see where you could read that unless you are reading more into it in order to be upset about something.

Not just that, your reaction is in a issue where everyone including the code owner agrees it happens, that it shouldn't happen and people are actively trying to fix it in the best way possible.

Sorry again but I got none of that by the replies to this issue.

nowhere in this issue did anyone imply that the issue was actively being worked on. if they did then I missed it.

the only suggestion in the issue was that someone suggested a workaround for it and I'm pretty sure it wasn't the code owner. And the codeowner has only replied once in the issue and it was over a month ago now. So I'm not sure how anyone could know that it's being actively worked on.

I only commented that it shouldn't take a work around for this to work as expected.

(just to head this off...) I'm not saying that anyone should be required to fix anything, either.

You also seem to be overstating how easy it is to spot edge cases like this one.

No I'm not.

I literally never said anything of the sort.

Of course, the whole purpose of issues are likely due to edge cases. If it wasn't an edge case then the issue likely would have been fixed long before now.

I, again literally, just added a bit more information in the form of a couple of examples.

But as you said this is not helping the issue get resolved so at that I'll leave it alone and apologize again that you took what I said negatively. No offense was meant to you or anyone else.

[Petro31]

@neil1111 you need a time window. You have zero currently

[sophof]

@neil1111 you need a time window. You have zero currently

Unless it got changed without me noticing (which could be the case) you shouldn't need to set any window. Without a window it should simply always take the derivative based on the last two measurements. You could try checking by setting a 1 second time window, which effectively would have the same effect (the underlying assumption is linear, so the derivative is constant between two data points)

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[ildar170975]

up

[sparkelende]

Having issues with this too. Derivative sensor keeps showing a change value even when change is 0. Any workaround on this?

[mekaneck]

The workaround is to create a template sensor that forces the repeated values to be stored in the database, which then gives the derivative sensor data for it to actually calculate a zero slope. The way to enforce recording of unchanging states is to add an attribute to your template sensor that changes as often as you want to force an entry in the recorder database.

https://github.com/home-assistant/core/issues/67627#issuecomment-1451255785

When you create a derivative sensor based off this new template sensor you should no longer have an issue.

[sparkelende]

The workaround is to create a template sensor that forces the repeated values to be stored in the database, which then gives the derivative sensor data for it to actually calculate a zero slope. The way to enforce recording of unchanging states is to add an attribute to your template sensor that changes as often as you want to force an entry in the recorder database.

#67627 (comment)

When you create a derivative sensor based off this new template sensor you should no longer have an issue.

Not very experienced with HA coding, usually trial and error.. Tried for an hour or 2 but couldn't get a workaround working. Could you please help me in the right direction, maybe even give me a litle bit of code I can use? :)

I have a Wemos d1 mini flashed with tasmota connected to a YF-B6 hall-effect sensor for measuring flow for a pump. Set it up as a counter in Tasmota.

So that gives me sensor.tasmota_systeempomp_counter_c1 in my HomeAssistant that counts the total of pulses. But I need pulses per minute to calculate flow, so i created:

• platform: derivative name: systeempomp_puls_min unit_time: min source: "sensor.tasmota_systeempomp_counter_c1" time_window: "00:00:10" round: 1

This gives me the pulses per minute i need to calculate using 6.6Hz per l/min as a given spec for my YF-B6 sensor. So pulses per minute / (6.6Hz * 60 seconds) gives me the actual flow.

• platform: template sensors: systeempomp_liter_min: unit_of_measurement: "l/min" value_template: '{{ ((states("sensor.systeempomp_puls_min") |float) / 396) | round(1) }}'

So now I have the flow in liters per minute, which is pretty much is 99% accurate compared to the flowmeter on the Grundfos Alpha 2 pump itself.

The issue here is when the pump is turned off, flow drops from 15l/min to readings like 3 or 6 l/min, but never to 0 which messes up all sorts of calculations used for efficiency for my heatpump. Noticed this issue day before yesterday ,explaining my lack of heatpump efficiency in recent weeks.

But it also looks totally stupid in the system overview i created.

Would really appreciate any help. Thanks in advance!

[sparkelende]

So I finally managed to get the workaround going. Ill share it for everyone experiencing similar issues.

First create the template sensor for forcing database entry of the counter. In my case for one of my pumps.

template:

• sensor: name: "warmtepomp_pulsen_totaal_templatesensor" state: > {{ (states("sensor.tasmota_warmtepomp_counter_c1")) }} attributes: minute_counter: "{{ now().minute }}"

After that, create the derivative sensor:

• platform: derivative name: wp_derivative_van_totaal_pulsen_templatesensor unit_time: min source: "sensor.warmtepomp_pulsen_totaal_templatesensor" time_window: "00:00:10" round: 2

Now you have the pulses per minute and you can create a different sensor with calculations to flow. Before, it didn't go to 0 when stopping the pump, now it does.

@mekaneck thank you for pointing me in the right direction.

[g1za]

Another solution/implementation, like the one initially suggested by @Petro31, is proposed here: https://github.com/home-assistant/core/pull/88655

[edgarveersel]

For me this was working well until 2023.03 and only went wrong since 2023.04. I don't know the cause, but I assume the deduplication that was done in the database could be it...

Just for informational purposes: I created a workaround by using Node Red. It reads the incoming values, calculates the derivative and writes the result to a numeric helper. Here are the differences:

Derivative Helper

By Node Red

[jbsmi]

I have in the same problem with derivative that don't turn to 0, can you please share the Nod-Red code to create a helper with the right value? I don't have this problem with mqtt value's, because of the "force_update: true" function with custom sensors, to update the value to HA even there is no change But with modbus sensors this function is not supported in modbus.yaml

[edgarveersel]

Hi jbsmi, I am currently on vacation, and I can't reach Node Red at the moment. But I can give you a functional description if that helps. In my case, I read the source values from MQTT, of which I know come in every 180 seconds (Tasmota telemetry setting). So I use an MQTT in node to read the value, then use a Change Node to calculate:

msg.derive = (msg.sourceValue- $flowContext('prev_sourceValue')) * (60/3)

Then I use a Change node to store the current value as a flow parameter flow.prev_sourceValue Lastly, I store the derivative value back to a Number Helper in HA.

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[ildar170975]

up

[WMP]

I have this same on HA 2023.07, derivative is based on mqtt sensor from Mew 01:

    - name: '#0 Electricity Meter (Total reverse active energy - balanced)'
      availability:
        topic: supla/devices/zamel-mew-01-3a23c/connected
        payload_available: 'true'
        payload_not_available: 'false'
      device:
        identifiers: zamel-mew-01-3a23c
        manufacturer: Zamel
        name: ZAMEL MEW-01
        sw_version: 2.8.11
      unique_id: supla_cc50e33a23c0_0_4
      qos: 0
      unit_of_measurement: kWh
      state_topic: >-
        supla/devices/zamel-mew-01-3a23c/channels/0/state/total_reverse_active_energy_balanced
      value_template: '{{ value | round(5) }}'
      state_class: 'total_increasing'
      device_class: 'energy'
      expire_after: 300

I need to add to this mqtt sensor force_update: true to workaround this bug.

[edgarveersel]

Hi @WMP, Thanks for the tip. I tried it, but not sure if I did it the right way.

In my case the entity that is the source to the derivative helper is created by the Tasmota integration, sensor.koffiemachine_energy_total

So, I opened the customize.yaml file and added the source entity. So, I my case that is:

sensor.koffiemachine_energy_total:
  force_update: true

Reloading all yaml configurations might have been sufficient, but just to be sure, I rebooted HA. Unfortunately, no luck...

Still not 0.