Energy Dashboard doesn't display 'consumed solar' in some scenarios

[sexysaxon]

Checklist

• [X] I have updated to the latest available Home Assistant version.
• [X] I have cleared the cache of my browser.
• [X] I have tried a different browser to see if it is related to my browser.

Describe the issue you are experiencing

On some days 'consumed solar' isn't displayed on the energy distribution graphic on energy dashboard. Consumed solar will be shown in the main energy usage graphic for that day but is absent when looking at the week view. I have a battery on my system and this appears to happen only on nights where I charge the battery between 00:30 and 04:30 (I'm using Octopus Go tariff). This scenario only appears to happen when using large charges. on 4/9/22 I gave the battery a small charge and the problem didn't manifest. Similarly on 7/9/22 I only put in 1kWH and the problem didn't appear.

Describe the behavior you expected

Consumed solar to be shown correctly. Attached screen shot of a good day 7/11/2022

Steps to reproduce the issue

1. Charge a battery with over 1kWh between 00:30 and 04:30

...

What version of Home Assistant Core has the issue?

2022.8.4 and previous version

What was the last working version of Home Assistant Core?

No response

In which browser are you experiencing the issue with?

No response

Which operating system are you using to run this browser?

Windows 10

State of relevant entities

No response

Problem-relevant frontend configuration

No response

Javascript errors shown in your browser console/inspector

No response

Additional information

No response

[Ralphy140]

I am seeing this as well, Also if you check the month and week view the last day of the time period often shows very little production/usage, in my case less than 3kWh even if when i go to the actual day it shows 40kWh of production and usage

[sexysaxon]

Update after more investigation: It seems that if the battery is charged with small amounts of less than approx 1kWh overnight then the distribution graphic exhibits different put probably related issues. The screen grab from 24th September shows the segment for self consumed solar okay but the energy flow dots don't show any flow between grid and battery. This seems to work okay when larger charges are used overnight.

[calum-mcfarlane]

I'm seeing the same issue here. HS 2022.11.1, with the Chestnut FoxESS HA integration, also u sing a system with a battery and Octopus Go.

The only day I can see 'Solar Consumed' in the 'Week' view is when I had only just got the integration going and it had not logged the battery charging that happened overnight. Every night since I've charged the battery at least 7kWh and it's not showing up in the week view.

I'm assuming there's a bug in the nuts and bolts of how the energy dashboard works and it'll get fixed in a future version. The actual logging of the data seems to work fine going on the daily stats view.

Edit: I'm seeing the same issue across browsers on Windows, Linux, and mobile apps for both Android and iOS.

[kaitlinsm]

I'm seeing this too.

I think it's to do with how the rollup is done. It looks like there's an implicit assumtion that you never charge your battery from the grid.

Bearing in mind allt hese calculations are only done with four sensors:

• solar_produced(t) - kWh of solar energy produced in time period t
• grid_consumed(t) - kWh drawn from the mains in time period t
• grid_returned(t) - kWh sent to the grid in time period t
• battery_charge(t) - kWh sent to the battery in time period t
• battery_discharge(t) - kWh drawn from the battery in time period t

on the solar usage graph, it looks like we have two calculations done for each 1-hour slot, the stuff that's "positive" (i.e. energy used from all sources combined) and energy not used (energy sent to the grid or to the battery for later use)

// positive side
total_power_used = battery_discharge + grid_consumed + solar_produced - battery_charge

// negative side
total_power_unused = grid_returned + battery_charge

The assumption is - you only charge your battery with solar. So solar consumption: what did I use that I produced?

solar_consumed = solar_produced - total_power_unused
// or 
solar_consumed = solar_produced - grid_returned - battery_charge

If you do that in an hour, it works. For example, for this day between 16:00 and 17:00:

for that hour:

• I produced a total of 0.35kWh of electricity from my solar panels
• I consumed 0.21kWh from the grid
• I sent 0.02 kWh to the grid
• I sent 0.1 kWh to the battery
• I consumed 0.3kWh from the battery

therefore

total_power_used = battery_discharge + grid_consumed + solar_produced - battery_charge - grid_returned
total_power_used = 0.3 + 0.21 + 0.35 - 0.1 - 0.02
total_power_used = 0.74

which matches consumed solar on the image:

total_consumed = solar_consumed + grid_consumed + battery_discharge
total_consumed = 0.23 + 0.21 + 0.3
total_consumed = 0.74 

Where does solar consumed come from? I believe it's assumed that you only charge your battery with solar, so

solar_consumed = solar_produced - grid_returned - battery_charge
solar_consumed = 0.35 - 0.02 - 0.1 
solar_consumed = 0.23

whee! everything balances.

Ok, so we have our values and the sums work out. wonderful.

But what happens when we repeat this calculation over a day?

For that day:

• I produced a total of 1.98kWh of electricity from my solar panels
• I consumed 5.56kWh from the grid
• I sent 0.15 kWh to the grid
• I sent 1.8 kWh to the battery (1.3kWh during the night!)
• I consumed 4.5kWh from the battery

plug that into the equations again:

total_power_used = battery_discharge + grid_consumed + solar_produced - battery_charge - grid_returned
total_power_used = 4.5 + 5.56 + 1.98 - 1.8 - 0.15
total_power_used = 10.09

which does add up:

total_consumed = solar_consumed + grid_consumed + battery_discharge
total_consumed = 0.03 + 5.56 + 4.5
total_consumed = 10.09

and solar consumed?

solar_consumed = solar_produced - grid_returned - battery_charge
solar_consumed = 1.98 - 0.15 - 1.8 
solar_consumed = 0.03

Oh look. Solar consumed for one day is LESS than the solar consumed in just one hour.

[kaitlinsm]

The issue here is aggregation.

The correct formula for solar_consumed(t) is sum(solar_consumed(h)) where h within t and solar_consumed(h) > 0, not solar_produced(t) - grid_returned(t) - battery_charge(t)

[kaitlinsm]

I've created a google sheet with the data for this day -> https://docs.google.com/spreadsheets/d/1BxD7RmeJglqGjI2Rz2wuj-PGNiVkq8ELNbydUS9P5xU/edit?usp=sharing

The columns are

• Grid In (consumed from grid)
• Grid Out (Sent to grid)
• Solar (actual solar production)
• Battery In (Battery charge)
• Battery Out (power drawn from battery)
• Solar consumed (calculated column for solar consumed)
• Solar consumed(+) (calculated column =MAX(0, Solar Consumed)

The totals work out - my spreadhseet shows 0.03 solar consumed using the "wrong" formula, and the last column's total shows the actual solar energy consumed, a value that I think is more accurate.

The telling thing is that some columns have negative solar consumption - this is impossible.

[calum-mcfarlane]

@kaitlinsm nice work, hopefully this can be picked up and implemented into the next version of the energy dashboard.

[kaitlinsm]

Looks like my estimation of the calculation was right: https://github.com/home-assistant/frontend/blob/ac65882fdd722cbb6ae79427f26b5e26ce25801e/src/panels/lovelace/cards/energy/hui-energy-usage-graph-card.ts#L520

[kaitlinsm]

@bramkragten have you got any clues as to why we're seeing this weirdness?

[kaitlinsm]

looks like a similar issue in the energy distribution card - the solar consumption is zero if the total battery charge is more than solar produced: https://github.com/home-assistant/frontend/blob/ac65882fdd722cbb6ae79427f26b5e26ce25801e/src/panels/lovelace/cards/energy/hui-energy-distribution-card.ts#L153

[calum-mcfarlane]

@bramkragten have you got any clues as to why we're seeing this weirdness?

It certainly appears (coming from someone with minimal modern coding experience) that the dashboard was written with the expectation that a battery would only be charged from the panels and not from the battery. So presumably the fix is 'just' to re-factor the aggregation code to take this into account?

[purcell-lab]

Same issue as reported here https://github.com/home-assistant/frontend/issues/11947#issuecomment-1309916376

I have asked for this to be reopened.

I think there is an issue as energy dashboard issues do not get automatically assigned like issues raised against other integrations.

[jprates]

@purcell-lab called up to my attention that we already have several issues opened for what seems to be the same root cause:

https://github.com/home-assistant/frontend/issues/10163

https://github.com/home-assistant/frontend/issues/11947

https://github.com/home-assistant/frontend/issues/13701

https://github.com/home-assistant/core/issues/83549

The oldest of which goes back to 2021, we're almost on 2023 and we still have this issue around, displaying wrong data whenever batteries are used and don't just charge from solar and discharge to house, or so it seems.

It would be really nice to see this fixed, as some of us like me depend on HASS to integrate different systems, manufacturers, etc on one single platform, and as it is right now we can't trust the values it displays.

I would suggest all scenarios get together in just one single issue that would stay open, perhaps the oldest one, and the rest could be closed as a duplicate, but it would be great if we got someone to have a look at this.

Personally I'm not a programmer so I can't help getting my hands on it, but I am fully available to explain the use case scenarios I have and to test anything the devs need to see tested, anything I can do to help, count on me.

Cheers, -jprates

[jprates]

Just to add that after reading all posts on this issue, I'm happy to see someone else (@kaitlinsm) has reached the exact same conclusions as I did, so I would risk saying the bug is understood by now, but unfortunately it has not been fixed.

Anyway, not that anyone has asked me, but I'd vote for keeping this one issue opened as it is the best explained one thanks to @kaitlinsm, and close all others as duplicates.

Now if only anyone from the dev team would pick this... we'd be a happy group, wouldn't we? :)

My use case is here, hope it helps a bit more for us to be all on the "same page":

https://community.home-assistant.io/t/energy-management-in-home-assistant/326854/1423?u=jprates

[jprates]

I think there is an issue as energy dashboard issues do not get automatically assigned like issues raised against other integrations.

Any idea on how to fix this, i.e. who to call here to have a look?

[fanfarestreak]

I have got the same issue. Battery installed on 19th December and since then the energy dashboard shows no solar consumption.

[jprates]

Hi @frenck, terribly sorry for calling upon you on this issue, and wish you a happy new 2023 before anything else.

As one of the most knowledgeable and experienced developers on HASS, can you please let us know how can we get someone assigned to this issue that as you see dates back from a couple of years already? Are we doing something wrong in this issue or can we do something to move up this in the priority list?

Again, sorry for calling you here, but as you can see we got no traction at all so far, dating back to 2021.

Happy new year everyone.

[fxstein]

I can confirm this issue is still there. Whenever the charge from the grid is larger than the actual solar production, no solar self-consumption is shown.

[mpredfearn]

This is still an issue in 2023.3. The source of the problem seems to me to have been correctly identified by @kaitlinsm as the aggregation function that is applied does not take into account a battery being charged from / discharged to the grid. I'll add a screenshot of my system just to add another data point. Here we can see that the battery was charged with ~5.5kWh during the night, and then ~1kWh from the solar. Around 1.5kWh was used directly. This is calculated correctly in each hour. But the totals for the day are incorrect in that they assume all power to the battery comes from the solar and as such the self-consumption is less than / equal to 0. Unfortunately my ts isn't good enough to attempt a fix - we seem to need @emontnemery or @bramkragten to look into it.

[Ralphy140]

I have just had a battery installed and am seeing the exact behaviour where when the battery is charged from the grid my consumed solar is incorrect. I agree on the above view of the root of the issue so commenting to make sure this does not go stale

[BrentVerhaegen]

Also seeing the same behaior here but with a different scenario. I'm usually not charging (importing) from the grid but discharging (exporting) energy from the battery to the grid.

The self-consumed solar gauge also stays at 0, and the house consumed power 'wheel' shows that it all came from battery, whilst there was also some house consumed solar energy.

On top of that, the HA Energy dashboard considers the total consumed battery energy as house consumed energy and ingnores how much of that battery energy was exported. But that might be a separate issue.

[sexysaxon]

Another symptom of display issues on energy dashboard. On this particular day 28/4/23, self consumption of solar was significant but home graphic for solar consumption in Energy Distribution graphic is clearly too small and doesn't correlate with actual solar consumption.

This error is reflected in weekly and other views, 28/4/23 in weekly view shows insignificant solar consumption

[ryanevans-dev]

Here is another data point, I also charge and discharge the batteries. On 31st May in the day view you can see the consumed solar values:

In the week, month and year view though, you see no aggregation

[bxm6306]

I'm chiming in to also +1 a fix for this issue. I too have a solar system with a battery that can be charged from Grid but the energy dashboard counts both the energy the battery pulled from Grid as well as the energy taken out of the battery which result in inaccuracies.

I'm thinking a potential solution would be to allow users to optionally choose a configured Battery System as the target for a Grid consumption entity and re-adjust the energy calculation formulas accordingly. Here's a mockup:

[iainJersey]

Add me to the list of people with the same issue. It has only just become apparent (UK 'summer') but it renders the graphs useless much of the time

[no2chem]

Hi all,

I recently added a battery which charges from grid+solar and then exports during peak and can confirm that this is still an issue.

I'm happy to try to fix it (it looks like no one has tried to fix this issue for awhile now?) as I figure out what is going on based on the thread. It looks like the data in HA is correct but the problem is in the presentation from the cards?

[purcell-lab]

'm happy to try to fix it (it looks like no one has tried to fix this issue for awhile now?) as I figure out what is going on based on the thread. It looks like the data in HA is correct but the problem is in the presentation from the cards?

Your assistance would be very welcome.

[jprates]

I'd suggest one needs to figure out all of the use cases in the first place, and how to interpret the data coming into HASS from all sensors to map to the identified scenario/use case.

It would not be much of a surprise to me if more sensors would be needed in order to fully map all possible flows to/from producers/consumers.

Starting by identifying all battery use scenarios would probably fit most of posted situations here I reckon.

[iamnotralphy]

The mapping of each flow is the hard part, I built my own graph to solve this using Apex Charts and needed to build the following calculated sensors to solve the issues.

These are the flows that you need a a minimum to make this work Solar2House Solar2Grid Solar2Battery Grid2House Grid2Battery Battery2House Batter2Grid

The challenge was the conditions that were required to discover each flow, If you have a look further down the page in this repo you will see https://github.com/reptilex/tesla-style-solar-power-card AFP (Actual Power Flows) This goes a long was to solving the issue but there are requirements for it to work that are beyond just "add your sensor" and it works. I think that the maths is ok but having something that is user friendly enough and also caters for diffrent types of battery topology (AC vs DC coupled) will be a bit tricky.

[no2chem]

I see. It seems that perhaps different systems have different types of metering available?

Isn't it enough to have:

Solar Production Battery In/Out Grid In/Out

And then you can build all the sensors that @iamnhotralphy enumerated from that - and incidentally, that is all the information the energy panel asks for currently.

Yes, there are various configurations, if you have DC coupled solar that charges the batteries directly, you will need some way to measure solar production and battery input, but as long as you break it down to those 5 sensors, we can derive all the flows.

Do you see a use case that I'm missing? I think the issue is still as @kaitlinsm pointed out, aggregation isn't happening correctly for the daily totals, this shouldn't require any additional information that what the energy page already has. I'll look into the code sometime this evening.