springfall2008
"Current PV" sensor no longer tracking pv_today properly.
Describe the bug A clear and concise description of what the bug is.
The "Current PV' sensor is supposed to mirror the sensor in the pv_today config entity. However at some point in the last few days it seems to no longer be mirroring properly.
By the looks of it, there seems to be sometimes a residual amount from the previous day that's being subtracted from it.
Expected behaviour A clear and concise description of what you expected to happen.
Current PV and pv_today should be the same.
Predbat version V8.27.21
Environment details
- Inverter and battery setup
- Standard HAOS installer or Docker
- Anything else?
Screenshots If applicable, add screenshots to help explain your problem. The most useful ones can be your battery chart, the Predbat HTML plan and your current settings in HA.
Log file Can you capture a log file from the time of the issue, debug mode is not normally required.
You can download the logfile from the WebUI on the Dash tab select predbat_debug.log
Predbat debug yaml file This is important for any plan related issues.
Once you have captured the issue go to the Web UI, in the Dash tab click on predbat_debug.yaml and wait 30 seconds for it to download. Select 'keep' if your Web Browser thinks the file is dangerous.
Rename the download file to predbat_debug.yaml.txt and upload it to Github. This will allow your plan to be reproduced and also stores all your settings for review.
I guess this comes from this line:
https://github.com/springfall2008/batpred/blob/de20ec124b85434ddb96c0574268e38cc28b84de/apps/predbat/fetch.py#L517-L521
That's by the looks of it removing the value at the start of the day (00:00:00) presumably to allow for sensors that are lifetime totals.
For the SolaX setup (#259) we use daily tally's that reset at midnight. However due to polling delays reading the values and slight RTC mismatches, this reset happens between 00:00:10 and 00:00:25 in my case. The days where the PV today is wrong are days where this is at the slower end.
This will also be affecting the day imported and exported energy values too if used.
Two possibilities to fix if support for day resetting sensors is desirable (the name in apps.yaml suggests they are):
- Add a config option to day "energy sensors reset at midnight", which if true means the start of day value is not subtracted. Would have no impact on non-resetting sensors.
- Using 00:01:00 (
self.minutes_now - 1) as the start of day values to accommodate slight latencies in measurement. The impact on both resetting and non-resetting sensors would be a likely minimal inaccuracy.
If daily sensors are unsupported, for SolaX I can just switch to the total sensors rather than the todays sensors.
For the SolaX setup (#259) we use daily tally's that reset at midnight. However due to polling delays reading the values and slight RTC mismatches, this reset happens between 00:00:10 and 00:00:25 in my case. The days where the PV today is wrong are days where this is at the slower end.
Pretty sure predbat requires these to be properly 'today' sensors. We've been having problems in predbat with the _today sensors from our givenergy inverters (via givtcp) not resetting properly at midnight. If you don't have a reliable today sensor, can you wrap the 'total' sensor up in a utility meter that HA will reset each day?
@gcoan The sensors are "today" resetting on the inverter, but reading in takes 10 to 25 seconds. I imagine most integrations reading remote registers will have a similar latency.
Looking at the logic there it seems like predbat would be happier with non-resetting total sensors based on the inclusion of the logic subtracting the value from the start of the day. It seems like a sledgehammer to crack a nut adding extra utility meters to remove a sub-minute latency.
The UM was just a suggestion. It would be better if predbat didn't need today sensors and could take total sensors as well, so why don't you try changing the code and see if it works? And then PR it in if successful. Or I can include in my next PR, but you should take the credit !
Had a go adding in an option for adding am optional start of day offset to the todays load/import/export/pv sensor data (https://github.com/springfall2008/batpred/compare/main...TCWORLD:batpred:today-sensor?expand=1).
Wasn't sure whether the option was best added under APPS_CONFIG or APPS_SCHEMA, I tried the former but it was causing odd errors so I'm trying the latter for now.
It appears to work, though today the reset latency was on the early side so I'm going to have to keep checking for a random day that would have otherwise triggered the problem. Looking at the logs of the load_today sensor though it does seem to work - I can see when I change the start delay to 1 minute from 0 minutes, the load_today_now value drops by 0,01kWh (the amount it increased in that first minute of the day).
Hmm, this has happened again yesterday.
Was trying to debug why the new "check minimum" logic wasn't working. It seems I was looking in the wrong place originally, assuming the data was not accumulative. However predbat is converting the data into a continuously incrementing value over the full 7 days of history requested, rather than retrieving the data as is, so there are no dips for it to find the minimum.
I made it log the 5 minute values either side of midnight at the start of the data and they are all the same, and equal to a value a bit higher than the cumulative sum of the whole week prior.
The sum of the 7 days prior comes out to be 11kWh, whereas predbat had a midnight (+/-5 min) value of 11.7kWh. This leads to the difference between the sunday peaks is 0.7kWh,
Not entirely sure what is going on, will try and do some more debugging.
Right, so after much head scratching, I think its this part of the minute_data function that is causing the problem:
https://github.com/springfall2008/batpred/blob/cc9f31857e35df295ce078344ee2f9b64ebba99b/apps/predbat/utils.py#L452-L461
This is the bit that is detecting the drop to zero at the end of the day. As long as the drop is >=1 (kWh) this logic is fine. However not all days yield more than 1kWh of solar. Indeed looking back on all the messed up days, they are ones which followed a day that had <1kWh produced.
In those low yield days, it seems to be basically ignoring the drop to zero, and thus messing up the date. This also explains why it's not been been noticeably affecting import or export (I usually export 1kWh due to the grid bias). Although yesterday there was only 0.9kWh export, and todays export data is messed up.
Changing that limit to >=0.1 fixes the data in my case (only because it has a resolution of 0.1kWh). However I have no idea what side-effects that might have as lots of places use that function. Given they could also be used it for different units (kWh vs W, a hard-coded threshold probably doesn't make sense.
Before:
43585 | 2025-12-01 22:15:01.954068: Current data so far today: load 14.23 kWh, import 9.9 kWh, export 0.2 kWh, pv 0.2 kWh
43296 | 2025-12-01 22:10:02.980436: Current data so far today: load 14.03 kWh, import 9.6 kWh, export 0.2 kWh, pv 0.2 kWh
43155 | 2025-12-01 22:05:01.881231: Current data so far today: load 13.87 kWh, import 9.1 kWh, export 0.2 kWh, pv 0.2 kWh
After:
44486 | 2025-12-01 22:30:02.482781: Current data so far today: load 14.71 kWh, import 11.1 kWh, export 1.1 kWh, pv 0.2 kWh
44233 | 2025-12-01 22:25:28.995158: Current data so far today: load 14.56 kWh, import 10.8 kWh, export 1.1 kWh, pv 0.2 kWh
43853 | 2025-12-01 22:20:35.851885: Current data so far today: load 14.41 kWh, import 10.5 kWh, export 1.1 kWh, pv 0.2 kWh
I'm curious as to why there is a limit there at all. If it's in "incrementing" mode, then any drop is technically invalid, so the >= check is probably not strictly required.
I don't know if its related @TCWORLD but the way predbat determines load by day (which must share some of the same logic) frequently throws up warnings that I have gaps in my load energy sensor that predbat has to fill.
e.g. right now:
2025-12-01 19:55:01.134646: Warn: Historical day 8 has 110 minutes of gap in the load history data, filled from 34.12kWh to make new average 36.94kWh (percent 92%) 2025-12-01 19:55:01.151043: Warn: Historical day 3 has 130 minutes of gap in the load history data, filled from 31.31kWh to make new average 34.42kWh (percent 91%)
and yet the sensor (which updates every 5 minutes) has no obvious visual gaps in it, or sections where the load is flat, unchanging
its not just me who gets this. I could define a base load in apps.yaml but that seems wrong, covering up the behaviour
@gcoan Not sure about that one, I don't get any load gap warnings.
Out of curiosity, how does your predbat.load_energy_h0 sensor compare to the actual house load sensor? That should show the data predbat has imported with the various "smoothing" and "spike filtering" that it applies in that import function (though also has load_scaling applied).
Are there any flat sections in the imported data?
There's definitely something perculiar though, as this is a bit from yesterday (source bottom, pretbat top) where the predbat import has somehow taken strictly increasing data and added a dip into it:
I believe I've fixed the interpolation code that caused the above dip. Basically a rounding problem that meant the delta time used for linear interpolation was inaccurate leading to the interpolation overshooting. Will let it run for a day or so to check if it's resolved.
@TCWORLD I see the same dips on mine on the load_today:
The sensors are not the same, the predbat h0 is about 0.5-1kWh higher:
The dip at 00:05 in the top (my house load) is caused by the sensor being a time trigger template that updates every 5 minutes and at precisely 00:00 the underlying sensors have not yet changed, but 5 minutes later when it next updates they are all at zero
The predbat h0 is one of the sensors I purge the history of every night so I can only go back 2 days. I'll stop the purge for now to let it build up some history
Be interested in your interpolation fix and see if that makes any difference
@gcoan I've tacked the interp changes onto #3012 for now - I think it fixes it anyway.
The h0 sensor for load being larger is expected - it should be larger by a constant factor as the import multiplies it by the load_scaling config setting (1.05 in my case).
Would you mind downloading the data from that 22:00 to 23:00 history view so I can try and create a test case to see if it fixes it?
Here's the house load and h0 for all of yesterday:
You probably only wanted house load
Good point about load scaling, I'd forgotten that. Mine is 1.05 as well
Hmm, so I can't actually get it to reproduce the dips in the unit testing even before any of my changes, how weird. Must be something I'm missing. Will have to do some further testing tomorrow.
Although I can see how the load gaps it's complaining about appear. In your data, I see the following section where there is a tiny dip (there are a couple of these):
sensor.house_load_today | 2.33 | 2025-12-02T07:40:00.155Z
sensor.house_load_today | 2.23 | 2025-12-02T07:55:00.163Z
sensor.house_load_today | 2.33 | 2025-12-02T08:00:00.152Z
The dip will get ignored, and so you end up with two consecutive data points of the same value, so they smooth to a flat line (no load) rather than being a gentle slope. My changes don't rectify that problem.
I remember now having missing history issues originally. I then ended up changing my todays house load source to be an template integration sensor based on house power, which is then fed it through a "Utility Meter" to get daily data. Because the utility meter is set to be strictly incrementing, so there can be no duplicate data points in the house load data so I never get these missing load data warnings.
Interesting.
My house load is calculated from a template sensor of import-export+PV-battery charge+battery discharge
I have to do this as I have multiple inverters and batteries that share the house load so the givtcp load sensors are meaningless
Maybe I need to add a UM in as well to wrap the house load
Another option for your case that may work better (and save the extra sensory) is a condition on your trigger template that skips updating the value if the new value is less than the old (with the exception of the midnight case).
I've finally worked out what I was missing.
Every time the prediction runs, predbat is recalculating the full history of the sensor starting from N days ago. It then runs through calculating smoothing and essentially integrating the data points before subtracting the value at the start of today to get the "now" value which it updates into the h0 sensor history.
The problem is that it will therefore have a different starting point for the data, and essentially produce a different history curve each time because of rounding to the nearest minute. Any errors in the curve add up until it inevitably produces a value for now that happens to be smaller than the previous run.
Each of the data points in the h0 history are basically from different discontinuous sets, so there is no expectation for them to form a perfectly incrementing waveform.
While not strictly a bug, it's kind of ugly. But unrelated to the original issue.
Another option for your case that may work better (and save the extra sensory) is a condition on your trigger template that skips updating the value if the new value is less than the old (with the exception of the midnight case).
I've created a utility meter this morning and will see what it looks like when its built up some history, but these are good ideas, only looking at the raw sensor history can I see these small dips due to rounding of the underlying sensors to the nearest 0.1kWh.
I might have a go at creating a revised template as well and test them in parallel. Thanks
Every time the prediction runs, predbat is recalculating the full history of the sensor starting from N days ago. ... While not strictly a bug, it's kind of ugly.
Yes this is the way that predbat works out the cost today entities and similar. I agree its ugly, and it leads to some annoying consequences:
-
if you have set a rate override for say a power up event, the event happens and you then modify the override for the next power up event, the recalculation causes the cost to spike as the original cost override is forgotten
-
more of an issue is people on IOG. They get an IOG slot, do some charging at cheap rate, and then soon after Octopus remove the slot from the IOG sensor history so again the cost today jumps to the wrong value
Whilst the current approach is logical and deals with the situation of predbat being offline or not running for a period of time, I don't like the way it works. Personally I think it would be better if a 'cost to date/time' attribute was stored (probably the last_updated is good enough already) and then predbat calculates the cost from that time to now. This would solve the rewriting of history issue and be more efficient as well