home-assistant
Fix time-based statistics (peaks, incorrect values)
Fixing issues with time-based statistics (peaks, incorrect values) and provide values for statistical function even if only a single input value exists
Breaking change
Proposed change
The statistics functions have several issues:
- Currently, many functions require at least two sensor values although a well defined value also exists when just one sensor value is there (e.g. the average of [7] is just 7. Instead of returning "unknown", we can return a proper value.
- Currently, time-based functions (average step / linear) don't compute a proper average over the specified time interval. Instead the average is only computed between the first and last change within the interval. This produces unexpected results when values are not changing over a longer period.
- During the initialization phase of the sensor values can be added to the internal queue before values have been loaded from the database. This messes up the order of the queue, which needs to be sorted for a correct computation of the time based functions. This creates incorrect values and extreme spikes that will only go away over time. Correct values are only computed after the faulty values were kicked out of the queue (when max age was reached).
- Currently, the computation of the time-based functions is only triggered when input values change or values are dropping out of the age range. However, the actual average changes over time, because the time interval is shifting.
The changes will fix the above mentioned issues. Also they add a new parameter "refresh interval" that allows to updated the computed averages even if the input states are not changing.
Type of change
- [ ] Dependency upgrade
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New integration (thank you!)
- [X] New feature (which adds functionality to an existing integration)
- [ ] Deprecation (breaking change to happen in the future)
- [ ] Breaking change (fix/feature causing existing functionality to break)
- [ ] Code quality improvements to existing code or addition of tests
Additional information
- This PR fixes or closes issue: fixes #119738 #98262 #67627
- This PR is related to issue:
- Link to documentation pull request:
Checklist
- [X] The code change is tested and works locally.
- [X] Local tests pass. Your PR cannot be merged unless tests pass
- [x] There is no commented out code in this PR.
- [X] I have followed the development checklist
- [ ] I have followed the perfect PR recommendations
- [X] The code has been formatted using Ruff (
ruff format homeassistant tests) - [X] Tests have been added to verify that the new code works.
If user exposed functionality or configuration variables are added/changed:
- [x] Documentation added/updated for www.home-assistant.io
If the code communicates with devices, web services, or third-party tools:
- [ ] The manifest file has all fields filled out correctly.
Updated and included derived files by running:python3 -m script.hassfest. - [ ] New or updated dependencies have been added to
requirements_all.txt.
Updated by runningpython3 -m script.gen_requirements_all. - [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
To help with the load of incoming pull requests:
- [ ] I have reviewed two other open pull requests in this repository.
Please take a look at the requested changes, and use the Ready for review button when you are done, thanks :+1:
Learn more about our pull request process.
![home-assistant[bot] avatar](https://avatars.githubusercontent.com/in/97978?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hey there @thomdietrich, mind taking a look at this pull request as it has been labeled with an integration (statistics) you are listed as a code owner for? Thanks!
Code owner commands
Code owners of statistics can trigger bot actions by commenting:
@home-assistant closeCloses the pull request.@home-assistant rename Awesome new titleRenames the pull request.@home-assistant reopenReopen the pull request.@home-assistant unassign statisticsRemoves the current integration label and assignees on the pull request, add the integration domain after the command.@home-assistant add-label needs-more-informationAdd a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.@home-assistant remove-label needs-more-informationRemove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.
Hello @unfug-at-github, its generally not a good idea to mix so many changes in one PR. Still, please give me some time to review, this looks like some good additions! Thanks for the contribution :)
Hello @unfug-at-github, its generally not a good idea to mix so many changes in one PR. Still, please give me some time to review, this looks like some good additions! Thanks for the contribution :)
I agree. I should have at least put the fixes for the average and the changes for the "compute with only one value" functions into different requests.
With the average functions it's a little harder to split things up, because these changes are somewhat depending on each other.
Let me know if things need to be put into separate requests. It's my first contribution here, and I have to admit that I only read some of the how-tos after I fixed all the issues that I had with my solar management solution.
I could really use this fix from this PR, @ThomDietrich could you give an estimate on when you have time to approve this PR?
I think this PR should try to split up more, for me as a reviewer its not clear why we exactly change what since this PR tries to solve 4 bugs, so I can't discover if this is the proper way to solve the bug.
Also, there is a merge conflict.
I have created a separate pull request that only addresses the issue of the spikes (#127268). I'll add further ones to reduce the scope of this one.