springfall2008
Predbat Log showing missing data history which does not correct itself over time.
Routine inspection of predbat log file indicates that selected history days are missing several hours of historical data. Reboot of PB, Givtcp v3 and MQTT have no effect on outcome. Same error has been evident for over a week now. No data has been missing to the best of my knowledge
Example of error messages:
2025-04-05 17:05:05.977187: Warn: Historical day 2 has 205 minutes of gap in the data, filled from 7.3 kWh to make new average 8.51 kWh (percent 86%) 2025-04-05 17:05:05.991365: Warn: Historical day 3 has 235 minutes of gap in the data, filled from 5.9 kWh to make new average 7.05 kWh (percent 84%) 2025-04-05 17:05:48.795890: Info: record_status Demand 2025-04-05 17:10:06.874377: Warn: Historical day 2 has 205 minutes of gap in the data, filled from 7.3 kWh to make new average 8.51 kWh (percent 86%) 2025-04-05 17:10:06.888351: Warn: Historical day 3 has 235 minutes of gap in the data, filled from 5.9 kWh to make new average 7.05 kWh (percent 84%) 2025-04-05 17:10:49.742108: Info: record_status Demand 2025-04-05 17:15:05.636640: Warn: Historical day 2 has 205 minutes of gap in the data, filled from 7.3 kWh to make new average 8.51 kWh (percent 86%) 2025-04-05 17:15:05.651563: Warn: Historical day 3 has 235 minutes of gap in the data, filled from 5.9 kWh to make new average 7.05 kWh (percent 84%) 2025-04-05 17:15:48.756103: Info: record_status Demand 2025-04-05 17:20:05.709863: Warn: Historical day 2 has 205 minutes of gap in the data, filled from 7.3 kWh to make new average 8.51 kWh (percent 86%) 2025-04-05 17:20:05.724882: Warn: Historical day 3 has 235 minutes of gap in the data, filled from 5.9 kWh to make new average 7.05 kWh (percent 84%) 2025-04-05 17:20:52.413165: Info: record_status Demand 2025-04-05 17:25:05.261230: Warn: Historical day 2 has 205 minutes of gap in the data, filled from 7.3 kWh to make new average 8.51 kWh (percent 86%) 2025-04-05 17:25:05.275405: Warn: Historical day 3 has 235 minutes of gap in the data, filled from 5.9 kWh to make new average 7.05 kWh (percent 84%) 2025-04-05 17:25:48.132300: Info: record_status Demand 2025-04-05 17:30:05.168314: Warn: Historical day 2 has 205 minutes of gap in the data, filled from 7.41 kWh to make new average 8.65 kWh (percent 86%) 2025-04-05 17:30:05.182218: Warn: Historical day 3 has 235 minutes of gap in the data, filled from 5.8 kWh to make new average 6.93 kWh (percent 84%) 2025-04-05 17:30:48.370354: Info: record_status Demand 2025-04-05 17:35:05.420484: Warn: Historical day 2 has 200 minutes of gap in the data, filled from 7.49 kWh to make new average 8.69 kWh (percent 86%) 2025-04-05 17:35:05.433031: Warn: Historical day 3 has 240 minutes of gap in the data, filled from 5.8 kWh to make new average 6.96 kWh (percent 83%) 2025-04-05 17:35:48.386914: Info: record_status Demand 2025-04-05 17:40:05.317029: Warn: Historical day 2 has 200 minutes of gap in the data, filled from 7.36 kWh to make new average 8.55 kWh (percent 86%) 2025-04-05 17:40:05.329687: Warn: Historical day 3 has 240 minutes of gap in the data, filled from 5.94 kWh to make new average 7.12 kWh (percent 83%) 2025-04-05 17:40:48.140971: Info: record_status Demand 2025-04-05 17:40:49.583932: Info: Refresh config entities due to their age of 480.0 minutes 2025-04-05 17:40:58.006252: Warn: Historical day 2 has 200 minutes of gap in the data, filled from 7.36 kWh to make new average 8.55 kWh (percent 86%) 2025-04-05 17:40:58.018968: Warn: Historical day 3 has 240 minutes of gap in the data, filled from 5.94 kWh to make new average 7.12 kWh (percent 83%) 2025-04-05 17:41:34.636796: Info: record_status Demand 2025-04-05 17:45:05.473360: Warn: Historical day 2 has 200 minutes of gap in the data, filled from 7.33 kWh to make new average 8.51 kWh (percent 86%) 2025-04-05 17:45:05.486463: Warn: Historical day 3 has 240 minutes of gap in the data, filled from 6.0 kWh to make new average 7.2 kWh (percent 83%)
I'm getting this too,
We all do.
It has been raised before, see #1239. Trefor suggested it was due to periods of zero load but we weren't sure
I'm getting this too,
We all do.
It has been raised before, see #1239. Trefor suggested it was due to periods of zero load but we weren't sure
The thing is, even if the load is zero, zero is still a figure it can't be classed as missing. I've been away on holiday with everything turned off and my load was sometimes as low as 150w, but its still a valid load so it can't be missing. Also it tells me that it has been missing for 420 mins (7 hours). That is clearly wrong.
A value of 150w isn't zero, this is saying you have zero load which is impossible unless your system isn't talking correctly.
Please pull up the load chart of the period in question
I get this all the time @springfall2008 as well, e.g.:
22856 2025-04-12 18:45:02.073589: Warn: Historical day 8 has 85 minutes of gap in the data, filled from 21.61 kWh to make new average 22.96 kWh (percent 94%) 22855 2025-04-12 18:45:02.068888: Warn: Historical day 4 has 25 minutes of gap in the data, filled from 22.88 kWh to make new average 23.29 kWh (percent 98%) 22854 2025-04-12 18:45:02.066968: Warn: Historical day 3 has 15 minutes of gap in the data, filled from 21.85 kWh to make new average 22.08 kWh (percent 99%) 22853 2025-04-12 18:45:02.064750: Warn: Historical day 2 has 45 minutes of gap in the data, filled from 24.65 kWh to make new average 25.45 kWh (percent 97%)
These are completely bogus as I create my load today as a time based trigger that fires ever 5 minutes to calculate my house load, so the only way it should ever be zero is if the house load calculation produces a negative value, or HA is off - which it isn't.
My template sensor:
# Home consumption sensor, updated every 5 minutes instead of the default of every sensor state change
- trigger:
- platform: time_pattern
minutes: "/5"
sensor:
- name: "House Load Today"
unique_id: "house_load_today"
unit_of_measurement: kWh
state_class: total
device_class: energy
state: >
{% set x=(states('sensor.g_solar_energy_today')|float(0)
+ states('sensor.h_solar_energy_today')|float(0)
+ states('sensor.fit_solar_energy_today')|float(0)
+ states('sensor.total_battery_discharge_today')|float(0)
- states('sensor.total_battery_charge_today')|float(0)
+ states('sensor.grid_import_today')|float(0)
- states('sensor.g_xxx_export_energy_today_kwh')|float(0) )
%}
{{ max(x,0)|round(1) }}
And here's the sensor graph for yesterday:
day before:
day before that:
day before that:
As you can see there are no periods where the sensor goes back to zero.
But, a possible cause of this issue? I can see in my graphs, and its even more obvious when you zoom the time range in:
That there are small periods of time when the load_today value goes up then goes back down then goes back up again. In my case this is due to the granularity of the underlying kWh sensors, when summed and subtracted, are I think creating tiny rounding errors. Might this be why Predbat thinks there are gaps in the sensor history?
So, as I hope you can see when I commit this text, I have 60 mins of data missing from day 3. Now because I can't be exactly sure which day 3 is, I have taken a good few days into ge past just to be sure. As you can see, the data never drops to zero.
Running HA 2025.3.4 and 15.0 on pi5 in HAOS.
I have had these historical day missing hours issues for months. I now just ignore the warnings although they make a mess of my logs. From time to time, I use predai and the historical hours warnings disappear.
2025-04-12 20:00:37.439902: Info: record_status Exporting target 74%-4%
2025-04-12 20:05:02.525336: Warn: Historical day 3 has 295 minutes of gap in the data, filled from 13.34 kWh to make new average 16.78 kWh (percent 80%)
2025-04-12 20:05:02.533508: Warn: Historical day 4 has 60 minutes of gap in the data, filled from 16.19 kWh to make new average 16.89 kWh (percent 96%)
2025-04-12 20:05:02.538994: Warn: Historical day 5 has 155 minutes of gap in the data, filled from 14.19 kWh to make new average 15.9 kWh (percent 89%)
2025-04-12 20:05:02.553183: Warn: Historical day 6 has 170 minutes of gap in the data, filled from 14.27 kWh to make new average 16.19 kWh (percent 88%)
2025-04-12 20:05:02.572251: Warn: Historical day 7 has 160 minutes of gap in the data, filled from 16.08 kWh to make new average 18.09 kWh (percent 89%)
2025-04-12 20:05:02.588748: Warn: Historical day 8 has 90 minutes of gap in the data, filled from 15.26 kWh to make new average 16.28 kWh (percent 94%)
2025-04-12 20:05:04.387944: Info: record_status Exporting target 72%-4%
Predbat and debug logs:
Rob
I have taken a good few days into ge past just to be sure
@Eeebygum23 I think you have the wrong givtcp sensor. Predbat is complaining about gaps in load history, which is whatever load_today is configured to in apps.yaml
Most likely it is sensor.givtcp_xxx_load_energy_kwh_today (or something like that name). i.e. the energy consumed
your graph is of load_power which is instantaneous power consumed, not cumulative energy
I have arrached a portion of my apps.yaml file and I have the file that you suggested. Does this look correct??Kind RegardsChrisOn 12 Apr 2025, at 22:19, Geoffrey Coan @.***> wrote:
I have taken a good few days into ge past just to be sure
@Eeebygum23 I think you have the wrong givtcp sensor. Predbat is complaining about gaps in load history, which is whatever load_today is configured to in apps.yaml Most likely it is sensor.givtcp_xxx_load_energy_kwh_today (or something like that name). i.e. the energy consumed your graph is of load_power which is instantaneous power consumed, not cumulative energy—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>
gcoan left a comment (springfall2008/batpred#2210)
I have taken a good few days into ge past just to be sure
@Eeebygum23 I think you have the wrong givtcp sensor. Predbat is complaining about gaps in load history, which is whatever load_today is configured to in apps.yaml Most likely it is sensor.givtcp_xxx_load_energy_kwh_today (or something like that name). i.e. the energy consumed your graph is of load_power which is instantaneous power consumed, not cumulative energy
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>
Comparison of running predbat using standard HA historical data and predai
I have restarted predbat twice, each time with debug.enable switched on. For each run, I have attached the predbat log and debug log. The critical part is between the .py files being watched and the first status info. This is where any historical hour warnings show up.
To initiate predai, I uncommented the respective lines in apps.yaml:
# Load forecast can be used to add to the historical load data from predai
load_forecast_only: True
load_forecast:
- sensor.givtcp_{geserial}_load_energy_today_kwh_prediction$results
When running predai, the warnings are absent. Does this offer a clue or am I pursuing a line that is invalid?
1. Using HA historical
predbat_debug_10_35_00.yaml.txt
Predbat files are installed correctly for version v8.18.0
**** Starting Standalone Predbat ****
2025-04-13 10:38:07.786402: Loading apps.yaml
2025-04-13 10:38:07.844841: Info: Add-on slug is 6adb4f0d_predbat
2025-04-13 10:38:07.847892: Info: Connected to Home Assistant at http://192.168.68.83:8123
2025-04-13 10:38:07.847973: Creating task: <coroutine object HAInterface.socketLoop at 0x7f93144e50>
2025-04-13 10:38:07.848361: Info: Web Socket task started
2025-04-13 10:38:07.848481: Creating task: <coroutine object WebInterface.start at 0x7f931fec50>
2025-04-13 10:38:07.848797: Info: Start socket for url http://192.168.68.83:8123/api/websocket
Web interface started
2025-04-13 10:38:07.852777: Info: Web Socket active
Watching ['./compare.py', './execute.py', './fetch.py', './web.py', './userinterface.py', './alertfeed.py', './prediction.py', './output.py', './energydataservice.py', './download.py', './ha.py', './gecloud.py', './inverter.py', './unit_test.py', './plan.py', './predbat.py', './utils.py', './db_manager.py', './config.py', './octopus.py', './apps.yaml', './predheat.py', './hass.py', './solcast.py', './futurerate.py'] for changes
2025-04-13 10:38:12.199853: Warn: Historical day 3 has 465 minutes of gap in the data, filled from 9.72 kWh to make new average 14.36 kWh (percent 68%)
2025-04-13 10:38:12.213421: Warn: Historical day 4 has 70 minutes of gap in the data, filled from 15.37 kWh to make new average 16.15 kWh (percent 95%)
2025-04-13 10:38:12.220152: Warn: Historical day 5 has 130 minutes of gap in the data, filled from 15.66 kWh to make new average 17.21 kWh (percent 91%)
2025-04-13 10:38:12.230353: Warn: Historical day 6 has 185 minutes of gap in the data, filled from 13.89 kWh to make new average 15.94 kWh (percent 87%)
2025-04-13 10:38:12.250544: Warn: Historical day 7 has 160 minutes of gap in the data, filled from 14.99 kWh to make new average 16.87 kWh (percent 89%)
2025-04-13 10:38:12.266538: Warn: Historical day 8 has 95 minutes of gap in the data, filled from 16.2 kWh to make new average 17.34 kWh (percent 93%)
2025-04-13 10:38:47.233302: Info: record_status Demand
2. Using predai
predbat_debug_10_45_00.yaml.txt
**** Starting Standalone Predbat ****
2025-04-13 10:43:55.383801: Loading apps.yaml
2025-04-13 10:43:55.442251: Info: Add-on slug is 6adb4f0d_predbat
2025-04-13 10:43:55.446280: Info: Connected to Home Assistant at http://192.168.68.83:8123
2025-04-13 10:43:55.446509: Creating task: <coroutine object HAInterface.socketLoop at 0x7f8acf8e50>
2025-04-13 10:43:55.447201: Info: Web Socket task started
2025-04-13 10:43:55.447456: Info: Start socket for url http://192.168.68.83:8123/api/websocket
2025-04-13 10:43:55.447984: Creating task: <coroutine object WebInterface.start at 0x7f8ad99d50>
Web interface started
2025-04-13 10:43:55.451189: Info: Web Socket active
Watching ['./compare.py', './execute.py', './fetch.py', './web.py', './userinterface.py', './alertfeed.py', './prediction.py', './output.py', './energydataservice.py', './download.py', './ha.py', './gecloud.py', './inverter.py', './unit_test.py', './plan.py', './predbat.py', './utils.py', './db_manager.py', './config.py', './octopus.py', './apps.yaml', './predheat.py', './hass.py', './solcast.py', './futurerate.py'] for changes
2025-04-13 10:44:33.681097: Info: record_status Demand
2025-04-13 10:45:09.493075: Info: record_status Demand
2025-04-13 10:45:47.121764: Info: record_status Demand
2025-04-13 10:50:03.370639: Info: record_status Demand
2025-04-13 10:55:22.491854: Info: record_status Demand
Rob
To add to my post above and linking to recent post from @gcoan This is a chart for my 'last hour' for sensor.givtcp_ce2218g352_load_energy_today_kwh, as linked in apps.yaml:
load_today:
- sensor.givtcp_{geserial}_load_energy_today_kwh
Predbat documentation (https://springfall2008.github.io/batpred/apps-yaml/#historical-data) states that house load must be incrementing but load_today goes up and down. Could this be the origin of the missing hours?:
load_today - Entity name for the house load in kWh today (must be incrementing)
Rob
Was looking to see if there was an easy way of creating an 'always incrementing' load sensor that would present the last value if the new value was lower.
Would be a way to test this hypothesis.
Its not straight forward https://community.home-assistant.io/t/only-allow-a-sensor-value-to-increase/298648/9
Need to create a helper entity for last value, the template sensor itself, and would need an automation or some other mechanism to reset the last value to zero at midnight.
If you look at your givtcp load today above you see what Predbat would conisder zero load sections as it will ignore the little up and down and see it as a flat line.
You could use the load power instead as Predbat now supports this but its less accurate trying to integrate power to load.
If you look at your givtcp load today above you see what Predbat would conisder zero load sections as it will ignore the little up and down and see it as a flat line
interesting, thanks Trefor
So I guess the question is, should Predbat be considering these zero load sections as being an error and interpolating a load figure in them, or should we be increasing the load_filter_threshold in apps.yaml to say 60 minutes?
My load sensor is in kWh to 1 decimal place, so a 99W half-hour load (198W an hour) could be recorded by Predbat as zero load and interpolated from historical data. Add in that Predbat ignores ups and downs of the underlying sensor and it could be 199W in a half hour.
You could try 60 minutes instead, it should be fine provided you average multiple days of data
I don't know whether this data is useful but see below:
Looking back at my original 1 hour snapshot of load_today, with threshold = 30 min I had 17 dips of 1 kWh each = 17 kWh 24 hours of same pattern would give 408 kWh
I have commented out predai and have gone back to normal historical data. I have increased the threshold to 60 minutes and then to 90 minutes. Here is the comparison, albeit from different time periods:
Threshold = 30 min (default). Overall gap = 1015 min 2025-04-13 10:42:41.022170: Warn: Historical day 3 has 470 minutes of gap in the data, filled from 9.72 kWh to make new average 14.43 kWh (percent 67%) 2025-04-13 10:42:41.035958: Warn: Historical day 4 has 70 minutes of gap in the data, filled from 15.37 kWh to make new average 16.15 kWh (percent 95%) 2025-04-13 10:42:41.042835: Warn: Historical day 5 has 130 minutes of gap in the data, filled from 15.51 kWh to make new average 17.05 kWh (percent 91%) 2025-04-13 10:42:41.053135: Warn: Historical day 6 has 185 minutes of gap in the data, filled from 14.04 kWh to make new average 16.11 kWh (percent 87%) 2025-04-13 10:42:41.073582: Warn: Historical day 7 has 160 minutes of gap in the data, filled from 14.99 kWh to make new average 16.87 kWh (percent 89%) 2025-04-13 10:43:29.068051: Info: record_status Demand
Threshold = 60 min. Overall gap = 635 min 2025-04-13 21:35:01.840272: Warn: Historical day 3 has 310 minutes of gap in the data, filled from 9.36 kWh to make new average 11.93 kWh (percent 78%) 2025-04-13 21:35:01.847700: Warn: Historical day 4 has 200 minutes of gap in the data, filled from 12.75 kWh to make new average 14.81 kWh (percent 86%) 2025-04-13 21:35:01.851889: Warn: Historical day 6 has 20 minutes of gap in the data, filled from 14.19 kWh to make new average 14.39 kWh (percent 99%) 2025-04-13 21:35:01.855262: Warn: Historical day 7 has 35 minutes of gap in the data, filled from 14.38 kWh to make new average 14.74 kWh (percent 98%) 2025-04-13 21:35:01.857382: Warn: Historical day 8 has 70 minutes of gap in the data, filled from 15.85 kWh to make new average 16.66 kWh (percent 95%) 2025-04-13 21:35:33.590657: Info: record_status Demand
Threshold = 90 min. Overall gap = 360 min 2025-04-13 21:45:02.027344: Warn: Historical day 3 has 175 minutes of gap in the data, filled from 9.24 kWh to make new average 10.52 kWh (percent 88%) 2025-04-13 21:45:02.030439: Warn: Historical day 4 has 140 minutes of gap in the data, filled from 12.86 kWh to make new average 14.24 kWh (percent 90%) 2025-04-13 21:45:02.032706: Warn: Historical day 7 has 5 minutes of gap in the data, filled from 14.81 kWh to make new average 14.87 kWh (percent 100%) 2025-04-13 21:45:02.034768: Warn: Historical day 8 has 40 minutes of gap in the data, filled from 15.78 kWh to make new average 16.23 kWh (percent 97%) 2025-04-13 21:45:03.674451: Info: record_status Exporting target 39%-4%
I am now going back to predai for now.
Rob
You could try 60 minutes instead, it should be fine provided you average multiple days of data
that has made a difference
with load_filter_threshold set to 30 I get errors on 4 days:
8597 | 2025-04-13 21:50:02.303553: Warn: Historical day 6 has 10 minutes of gap in the data, filled from 21.46 kWh to make new average 21.61 kWh (percent 99%)
8596 | 2025-04-13 21:50:02.298715: Warn: Historical day 5 has 25 minutes of gap in the data, filled from 23.18 kWh to make new average 23.59 kWh (percent 98%)
8595 | 2025-04-13 21:50:02.294119: Warn: Historical day 4 has 15 minutes of gap in the data, filled from 23.42 kWh to make new average 23.66 kWh (percent 99%)
8594 | 2025-04-13 21:50:02.287794: Warn: Historical day 3 has 45 minutes of gap in the data, filled from 21.97 kWh to make new average 22.67 kWh (percent 97%)
but changing to 60 I only get error from 1 day:
9426 | 2025-04-13 21:55:02.047643: Warn: Historical day 3 has 10 minutes of gap in the data, filled from 21.69 kWh to make new average 21.84 kWh (percent 99%)
I have changed my template to use round(2) for the load sensor so will let that run for a few days to build up history and see if that makes a material difference
To create more stats, I am now back on normal historical data from predai. I have set threshold = 60 min I will leave now and see what happens.
Rob