Derivative value is reset to zero upon HA restart

[lazyoldbear]

trafficstars

The problem

Derivatives are being calculated appropriately, following the settings for time windows and precision. Unfortunately, upon restart the values are zeroed.

I suppose, that just means that the integration does not speak to the Recorder. Well, that just renders it not usable.

As a workaround, I reverted to using Statistics and a bunch of templates.

Should the maintainers insist that this behaviour is intentional, I recommend to make it very clear in the documentation.

What version of Home Assistant Core has the issue?

2022.5.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Core

Integration causing the issue

Derivative

Link to integration documentation on our website

https://www.home-assistant.io/integrations/derivative/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant[bot]]

derivative documentation derivative source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @afaucogney, mind taking a look at this issue as it has been labeled with an integration (derivative) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[Swamp-Ig]

While it could be recalculated from the history from before the shutdown, how would you manage values that were missing while HA was down?

[lazyoldbear]

Excellent point.

I, personally, would introduce a parameter to choose from three behaviours:

• Keep the last derivative value (recalculate on two previous points)
• Take the two last available points and honest time between them
• Abandon hope, as it does now

There are smarter ways of dealing with this, but I am unsure what is realistic without big changes.

[Swamp-Ig]

It's probably not worth the effort to manage this edge case. Without looking at the code, I expect it would require a fair bit of rework, plus it's such an edge case it would end up buggy because it wouldn't often get used.

If you really want to pull stats from history and calculate from what's returned, you can do this with some code in node red.

For the most part though the derivative probably will be zero, unless you're dealing with some really long time windows.

[lazyoldbear]

HA restart is an edge case? I am sorry, but that is hardly acceptable: it is tolerable to see some delays, but when a plot stops making sense for a day (the visible part), that is big. This is especially noticeable for slightly bigger time windows, for parameters that are rarely zero, like hourly water flow.

[lazyoldbear]

It is not a question whether it can be done elsewhere. Statistics are one example. But right now, derivative, which is a much, much better concept for the purpose, is simply not usable for the reasons described.

[lazyoldbear]

Wait. There is another option, much simpler. Just stay unavailable/unknown until there is a legit history of values.

[Swamp-Ig]

The main issue is that for every option introduced, there's potentially an extra set of bugs to introduce. For this reason you don't want to introduce too much configuration was more what I was trying to say.

If the derivative was originally undefined and then became defined after some time period the next issue is, what time period? Mathematically the derivative of two points is defined, but it does tend to oscillate around a lot for small sets of data. which is the whole point of having a smoothing window in the first place. There's a trade off - the sampling shot-noise increases the shorter the period becomes, while the value is delayed for longer periods.

Do we wait for a whole period to elapse first? Half a period? Just use the data you have? Pull data from the (potentially incomplete) history?

The optimal way to do this is going to depend on the use case, but only when HA is first starting up. Yes it's an edge case, because for most people HA is (or should be) constantly running. For complexity reasons you have to pick one.

[lazyoldbear]

Well, first of all, the new option introduced is derivative itself. We are talking about a mild flaw that would make it a great option.

Now, the existence of time window suggests that if undefined path is taken, that is what it is: the value stays undefined until there is enough continuous data. Otherwise, a restart will inject artefacts.

And I would recommend to assume HA restart to be a routine event. Even if it is not, it is not an edge case: new stable versions appear several times a month. And any restart will cause a drop on the plot. An edge case would be something that happens in a rare combination of conditions that apply to a minority of installations rarely enough.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.