Consumption History includes Cheaper Overnight Battery Top-up

[Pyinthesky99]

Describe the bug Predicted Consumption appears to include historical overnight charging, to take advantage of cheaper rates, of domestic storage. Does this skew setting of charging slots to earlier, and maybe costlier periods, to cover for later predicted consumption that doesn't happen.

To Reproduce Permanent

Expected behavior Consumption should be Domestic use only, charging of batteries in technically for domestic use but only when they discharge. This could be rather difficult to achieve.

Screenshots

pv_opt.log

Desktop (please complete the following information):

• OS: Win 11
• Browser Edge

Smartphone (please complete the following information):

• N/A

Additional context Am currently on v4.0.11-Beta-3, I inadvertently loaded a V5 and it didn't go well. Should I be on just v4.0.11?

[stevebuk1]

A prediction of consumption is made from the last 7 days of historic consumption, whicn is in turn obtained from an entity from your inverter. It is the inverter that has to do the various additions and substractions to come up with what your house actually consumes, as opposed to what comes in in from the grid and / or whats flowing in and out of the batteries. For exampe, for Solis inverters using SolisCloud, this is called sensor.solis_energy_today which is an incrementing sensor measuring kWh, reset at midnight. It does not include any battery charging.

If your config is using a HA entity from the inverter that includes overnight charging then its the wrong sensor and needs changing. It will be a case of looking at the various entities your inverter provides and checking your config doesnt point to one that is actually grid kWh or battery kWh. Your inverter may not provide this, in which case a prediction based on historic consumption may not be possible and you'd need to use a fixed daily consumption with a consumption profile. All that said I'm pretty sure we spent some time locating the right sensor for this when developing the Solarman integration?

[stevebuk1]

I've checked your log, and it is using this sensor to calculate the consumption prediction:

Just had a look at solis_hybrid.yaml and its this sensor that should be used.

Have a check of it in HA using the History tool and see if it includes any overnight charging?

[stevebuk1]

Am currently on v4.0.11-Beta-3, I inadvertently loaded a V5 and it didn't go well. Should I be on just v4.0.11?

Best to move to v4.0.11 production release (the various v4.0.11 beta releases get incorporated to v4.0.11 production). v5 versions should work but I've only tested them with Appdaemon v0.17.x, which was what its primary purpose is for. I don't think the versions here impact what you are seeing though, so just stick with what you've got for now if its easier.

[Pyinthesky99]

Hi Steve Thanks for the comprehensive investigation, as always.

Here is a snip of a few days history of that sensor which doesn't look as if it does include overnight charging.

Also here is the data..

history.csv

I was going by the lump in Forecast Consumption, probably wrong thing to do :-)

[stevebuk1]

You certainly have a lump of overnight consumption at around 3am, perhaps you are not expecting this?

If you turn on debug logs and set the debug categories to include the letter "P" I can take a closer look at both historic consumption and predicted.

[Pyinthesky99]

Never expect things to go bump in the night.

This is current snip and log, too soon?

pv_opt.log

[stevebuk1]

Unfortunately the log has no debugging, did you change debug_categories to include the letter P and also turn debugging on? Lines in config.yaml would be:

debug: true and debug_categories: OP

In the interim, you could have a look at sensor.solis_house_load_power in HA which gives you instantaneous house consumption but in W, which should be easier to find overnight peaks. Plotting together with house_load_today would give something that looks like this (I was on hols Thursday and Friday, the 7kW peaks on Monday night are car charging, which is seen as house load and Pv_opt removes using consumption data from the car charger, etc etc)

[Pyinthesky99]

Whoops, my error, changed to include P but logging was turned off!

This any better?

pv_opt.log

This is what my plot looks, there are some gaps as we have had multiple power cuts, a tree took out some overhead cables in the storm, they installed a generator which kept falling over.

[stevebuk1]

Great, Pv_opt.log now contains the power consumption logging. There is no 3am peak in either the historic data or the predicted data. Was the dashboard showing a 3am peak when you last posted?

It might be worth just copying the dashboard yaml file and attaching it just to check what is being plotted. Click the pencil icon on top right, then the 3 dots icon to get you to the raw configuration editor, then copy and paste the text directly here.

[Pyinthesky99]

Can't recall exactly if there was a peak at 3am predicted at the time of the log, isn't currently.

This is the yaml for the relevant card

type: custom:stack-in-card cards:

• type: vertical-stack cards:
  • type: markdown content: Results
  • type: entities entities:
    • entity: sensor.solis_battery name: Current Battery SOC
    • type: custom:template-entity-row name: Next Timed Slot Start state: >- {{(as_local(as_datetime(states('sensor.pvopt_charge_start')))|string)[:16]}} icon: mdi:timer-sand-complete
    • type: custom:template-entity-row name: Next Timed Slot End state: >- {{(as_local(as_datetime(states('sensor.pvopt_charge_end')))|string)[:16]}} icon: mdi:timer-sand-complete
    • entity: sensor.pvopt_charge_current name: Optimum Charge Current
    • entity: sensor.solis_battery_charge_current_limit name: Charge Current Limit Now
    • entity: sensor.solis_battery_current name: Battery Current Now state_color: true show_header_toggle: false
• type: custom:apexcharts-card apex_config: chart: height: 325px yaxis:
  • id: power show: true decimals: 0 apex_config: forceNiceScale: true header: show: true show_states: true colorize_states: true title: Solar Forecasts vs Actual graph_span: 2d stacked: false span: start: day series:
  • entity: sensor.solis_inverter_ac_power float_precision: 0 extend_to: now name: Inverter AC Load (15min avg) stroke_width: 2 color: '#ff0000' group_by: func: avg duration: 30min
  • entity: sensor.solis_inverter_dc_power float_precision: 0 extend_to: now name: Actual PV (Offset) stroke_width: 1 type: column color: '#34eb52' group_by: func: avg duration: 30min offset: +15min show: name_in_header: true in_header: true in_chart: true legend_value: true offset_in_name: false
  • entity: sensor.pvopt_opt_cost type: line name: Forecast Consumption color: yellow opacity: 0.7 stroke_width: 1 offset: +15min show: in_header: false legend_value: false data_generator: | return entity.attributes.consumption.map((entry) => { return [new Date(entry.period_start), entry.consumption]; });
  • entity: sensor.solcast_pv_forecast_forecast_today type: area name: '' color: '#dea4dd' opacity: 0.1 stroke_width: 0 unit: W show: in_header: false legend_value: false offset_in_name: false data_generator: | return entity.attributes.detailedForecast.map((entry) => { return [new Date(entry.period_start), entry.pv_estimate90*1000]; }); offset: +15min
  • entity: sensor.solcast_pv_forecast_forecast_today type: area name: ' ' color: '#1c1c1c' opacity: 1 stroke_width: 0 unit: W show: in_header: false legend_value: false offset_in_name: false data_generator: | return entity.attributes.detailedForecast.map((entry) => { return [new Date(entry.period_start), entry.pv_estimate10*1000]; }); offset: +15min
  • entity: sensor.solcast_pv_forecast_forecast_today type: line name: ' ' color: '#dea4dd' opacity: 1 stroke_width: 2 unit: W show: in_header: false legend_value: false offset_in_name: false data_generator: | return entity.attributes.detailedForecast.map((entry) => { return [new Date(entry.period_start), entry.pv_estimate*1000]; }); offset: +15min
  • entity: sensor.solcast_pv_forecast_forecast_tomorrow type: area name: '' color: '#dbd5db' opacity: 0.1 stroke_width: 0 unit: W show: in_header: false legend_value: false offset_in_name: false data_generator: | return entity.attributes.detailedForecast.map((entry) => { return [new Date(entry.period_start), entry.pv_estimate90*1000]; }); offset: +15min
  • entity: sensor.solcast_pv_forecast_forecast_tomorrow type: area name: ' ' color: '#1c1c1c' opacity: 1 stroke_width: 0 unit: W show: in_header: false offset_in_name: false legend_value: false data_generator: | return entity.attributes.detailedForecast.map((entry) => { return [new Date(entry.period_start), entry.pv_estimate10*1000]; }); offset: +15min
  • entity: sensor.solcast_pv_forecast_forecast_tomorrow type: line name: ' ' color: '#dbd5db' opacity: 1 stroke_width: 2 unit: W show: in_header: false legend_value: false offset_in_name: false data_generator: | return entity.attributes.detailedForecast.map((entry) => { return [new Date(entry.period_start), entry.pv_estimate*1000]; }); offset: +15min
• type: custom:apexcharts-card apex_config: chart: height: 234px yaxis:
  • id: soc show: true min: 0 max: 100 decimals: 0 apex_config: tickAmount: 5 header: show: true show_states: true colorize_states: true title: Battery SOC Forecast vs Actual graph_span: 2d span: start: day series:
  • entity: sensor.solis_battery extend_to: now name: Actual stroke_width: 1 type: area color: '#ff7f00' opacity: 0.4 yaxis_id: soc
  • entity: sensor.pvopt_opt_cost type: line name: Optimised color: '#7f7fff' opacity: 0.7 stroke_width: 2 unit: '%' show: in_header: false legend_value: false data_generator: | return entity.attributes.soc.map((entry) => { return [new Date(entry.period_start), entry.soc]; }); yaxis_id: soc
  • entity: sensor.pvopt_base_cost type: line name: Initial color: '#7fff7f' opacity: 0.7 stroke_width: 2 unit: '%' show: in_header: false legend_value: false data_generator: | return entity.attributes.soc.map((entry) => { return [new Date(entry.period_start), entry.soc]; }); yaxis_id: soc
• type: custom:apexcharts-card apex_config: chart: height: 230px header: show: true show_states: true colorize_states: true title: Pricing and Forced Charge/Discharge graph_span: 2d yaxis:
  • id: price decimals: 0 min: -5 max: 100 apex_config: tickAmount: 8
  • id: charge decimals: 0 opposite: true show: true min: -4000 max: 4000 apex_config: tickAmount: 8 stacked: false span: start: day series:
  • entity: >- event.octopus_energy_electricity_21l4303484_1012726052729_current_day_rates yaxis_id: price name: Historic Import Price color: yellow opacity: 1 stroke_width: 1 extend_to: now unit: p/kWh data_generator: | return entity.attributes.rates.map((entry) => { return [new Date(entry.start), entry.value_inc_vat*100]; });
    offset: '-15min' show: in_header: false legend_value: false offset_in_name: false
  • entity: sensor.pvopt_opt_cost type: line name: Future Import Price float_precision: 1 color: white opacity: 1 stroke_width: 1 extend_to: now unit: p/kWh offset: '-15min' show: in_header: true legend_value: false offset_in_name: false data_generator: | return entity.attributes.import.map((entry) => { return [new Date(entry.period_start), entry.import]; }); yaxis_id: price
  • entity: >- event.octopus_energy_electricity_21l4303484_1050002409859_export_current_day_rates yaxis_id: price name: Historic Export Price color: cyan opacity: 1 stroke_width: 1 extend_to: now unit: p/kWh data_generator: | return entity.attributes.rates.map((entry) => { return [new Date(entry.start), entry.value_inc_vat*100]; });
    offset: '-15min' show: in_header: false legend_value: false offset_in_name: false
  • entity: sensor.pvopt_opt_cost float_precision: 1 type: line name: Future Export Price color: green opacity: 1 stroke_width: 1 extend_to: now unit: p/kWh offset: '-15min' show: in_header: true legend_value: false offset_in_name: false data_generator: | return entity.attributes.export.map((entry) => { return [new Date(entry.period_start), entry.export]; }); yaxis_id: price
  • entity: sensor.pvopt_opt_cost type: column name: Forced Charge yaxis_id: charge color: orange opacity: 0.7 stroke_width: 1 unit: W offset: '-15min' show: in_header: false legend_value: false offset_in_name: false data_generator: | return entity.attributes.forced.map((entry) => { return [new Date(entry.period_start), entry.forced]; });

[Pyinthesky99]

There is a bit of a bump unexpected today forecast today for 10am

Would you like me to look out for the next significant and post alongside anything?

[Pyinthesky99]

Here's today's.....

pv_opt.log

EDIT: Snip a few hours later at 19:15

Historic battery charge seems to have moved forward to 11:30pm

[stevebuk1]

I think theres a bug in the consumption prediction, as you have no 3am peak in your historic consumption and is certainly shouldn't be moving!. Can you add a "Q" to the debug categories list that will then generate some more logging info on the various calculation stages of the forecast consumption. No need for any more images, the log will suffice. (Your dashboard entities are fine)

[Pyinthesky99]

Thanks Steve,

Have added a Q.

Here is the resulting log.

pv_opt.log

Let me know if you need anything else.

Regards Alan

[Pyinthesky99]

I don't know if this is part of the same issue, for some reason a charge slot was run at 4:30 this morning @ 14.99ppkWH. I don't know if that was for anticipated usage.

pv_opt.log

[stevebuk1]

I don't know if this is part of the same issue, for some reason a charge slot was run at 4:30 this morning @ 14.99ppkWH. I don't know if that was for anticipated usage.

This is just to cover anticipated consumption. 14.99p is the cheapest price for the rest of the night and the next day, until it gets cheap again at around 9.30pm. At that point the battery is empty, but the cost drops to 16.4p, and its cheaper to use the grid at 16.4p then it is to do more charging at 14.99p. (due to losses in charging and then discharging).

[stevebuk1]

Thanks Steve,

Have added a Q.

Here is the resulting log.

pv_opt.log

Let me know if you need anything else.

Regards Alan

I think whats happening here is that the algorithm doesn't like 10 days of history (when I changed mine from 7 to 10 the predicted consumption looked very strange indeed). It might be doing something wrong with the day of week weighting, not sure yet.

In the interim, set to 7 days and I think you'll find this will remove the peaks.

[Pyinthesky99]

Will do. Cheers Steve.

If all else fails, which it shouldn't, I will switch to Daily Consumption but I am not clear on the Consumption Profile, what are the 'units' involved.?

[Pyinthesky99]

Have set to 7 days and have a peak around 6am tomorrow, curious..

pv_opt.log

[stevebuk1]

If all else fails, which it shouldn't, I will switch to Daily Consumption but I am not clear on the Consumption Profile, what are the 'units' involved.?

Its in Watts. You are defining a point on straight lines between the points, e.g. from the above you are saying you will consume 750W from 5pm to 10pm (as these values are both the same), then it reduces linearly to 300W by midnight.

Note the profile is purely relative to the value of Daily Consumption on the Dashboard, so don't worry about it too much.

I use this after a weeks holiday when historic consumption is inaccurate for the week after I return and I've just left the profile as is. (However I'm on IOG so it probably matters less)

[stevebuk1]

Have set to 7 days and have a peak around 6am tomorrow, curious..

What have you got Weekday Weighting on the Dashboard set to?

[Pyinthesky99]

Have set to 7 days and have a peak around 6am tomorrow, curious..

What have you got Weekday Weighting on the Dashboard set to?

0.5 Steve

[stevebuk1]

Ok, I can't tell from your logs whats going wrong. I've added some more debugging to inspect the various calculation stages but still can't see anything wrong on my system either (but I'm on IOG, which doesnt forward predict as much as Agile does).

Unfortunately things have moved on too far for me to modify v4.0.11-Beta-3 so we will need to get you onto the latest Beta, which is V5.0.0-Beta-5. However, I think you said this didn't go well for you when you tried it? In which case we need to resolve any issues with this first. V5.0.X was designed to work with the latest versions of Appdaemon but I don't think thats a prerequisite. Best if you load V5.0.0-Beta-5 and post me a log.

[Pyinthesky99]

Ok, will try and load v5 tomorrow and post log, am currently on full v4,0.11 release, not beta.

Will have a play with/scientifically evaluate consumption profile.

Thanks

[Pyinthesky99]

Have loaded V5.0.0-Beta-5 and it stops at Updating Inverter.

pv_opt.log

error.log

appdaemon.log

Am running AppDaemon 0.16.7

[stevebuk1]

Try v5.0.0-Beta-6, which contains a fix for the stall at "Updating Inverter" and also the extra logging for this issue.

Leave Appdaemon unchanged at 0.16.7 for now.

[Pyinthesky99]

Thanks Steve, Beta 6 did the trick with the stalling.

Have set to 7 days history

pv_opt.log

This is the current chart..

[stevebuk1]

Thanks, extra logging is making things clear. Theres definitely a Pv_opt problem, the 9pm forecast peak isn't based on history. The day of week weighting guides the contribution of two arrrays, one that is an average of all 7 days and the other from what it was last week on that particular day. Whilst they are the same length (and there is a check for that), for some reason they start at different times so all the final summations I think are off. Its going to take some head scratching to figure this out as its not my code. I've got a few projects on at home (getting my kitchen done and also doing my bathroom, silly to do these both at the same time) so it may be a while to come up with a fix. But good spot!

[Pyinthesky99]

Thanks, Steve, glad you have spotted the root of the problem.

I don't envy you with your home improvements, Home Assistant is enough for me!

I will await events.

Regards

[Pyinthesky99]

Interesting,

Sorry, but it was supposed to charge this morning but did very little.

Predicting consumption to hang out until cheaper slots tonight?

pv_opt.log