Preserve peaks across all time filters

[jasonacox]

Problem The current Live Monitoring queries uses mean() to display line graphs for Home, Solar, Powerwall, etc. While this normally works well, the problem with this approach is that when you expand the time range (e.g. from past 24 hr to 30 days), it will average out the peaks. I often like to look up "spike events" and ask questions like: When did I see the largest Powerwall charge event, highest sell-back of power to the grid, or highest solar production moment last 90 days? Peaks are how you would spot this over those larger time intervals. Unfortunately, mean() makes it more difficult to find these.

Here is an example of the issue: A zoom in of a peak moment for home power shows a peak at 6.41kW:

The same data over 24 hours causes that peak to now show as 6.14kW.

The same data over 7 days causes that peak to now show as 5.57kW.

It gets even more averaged out by the time you look at 30 days with same peak now at 3.01kW

The issue is that as you expand the time filter span, the "time interval" that is used to run the mean() calculation expands, slowly averaging out the peaks. The challenge then becomes, so what value do you select in that larger time interval period? You could pick max() which would find the positive peak but would erase the negative ones or last() to randomly select the last "true" value in that interval which would also miss the peak if it happened before the last.

Proposal

So here is what I suggest we do to change dashboard.json to increase peak fidelity for longer time intervals:

• Solar to use max() - assumption here is that we would be more interested in seeing the "highs" than the "lows." Solar would always go to zero every day anyway (so long as the world keeps going around).
• Home to use max() - while it would be good to see both peaks and valleys with home, it feels like the more important signal would be max spikes. Downside on this one is that during high spike moments (hot summer with AC) the graph will represent the AC demand more than the troughs as you expand the view to greater time intervals.
• Powerwall and Grid will replace mean() with max() so that peaks both positive and negative show up (from/to powerwall or grid), eg. SELECT max(from_grid) - max(to_grid). This captures both sides of the peaks for the most part. If the interval happens to capture a moment where the Grid or Powerwall go from high outflow to high inflow, it would be negated (zeroed), but I couldn't even find an event like that in my data for the past year and instead, the fidelity of getting to/from spikes was much better using max().
• Charge and Reserve would stay the same at last(), basically picking the last sample in the time interval. This seems logical for both charge and discharge trends. Except for playing ping-pong with the reserve settings, these trends are gradual rather than spiky.
• Weather data would stay at mean() to average the data over the time interval. This would miss peaks and lows but would help answer "what was the average temperature during that time". I could also see using last() to just select a random (last) true data point to represent temperature.

When I apply that to the above example, you can see the result, the true 6.41kW even at 30 days:

I'm interested in hearing other thoughts. What am I missing? Is there a better way that my proposal?

[BJReplay]

I've noticed the same thing, and thought the same thing, but not got as far as thinking about a solution. I think what you've proposed makes sense.

• Weather data would stay at mean() to average the data over the time interval.

Since I'm a weather geek, too, now that I've got one minute weather data going back a couple of months, I can see how Max() or Min() for temp also makes sense - but given I've got my own dashboards, its no big deal for me to customise those for analysis of weather maxima / minima.

[jasonacox]

Thanks @BJReplay. I'm setting it up to see how I like it before issuing a PR.

How are you doing the max/min for your weather dashboards? We have the two panels at the bottom that may benefit from that.

[BJReplay]

How are you doing the max/min for your weather dashboards? We have the two panels at the bottom that may benefit from that.

For the moment, I've got this big-ass dashboard for my desktop (2560x1600) that displays the last 24 hours over three lines and four charts. The charts are still displaying mean(), and the queries behind the stat panels are still also using mean() but the stat panels are using reducer function of Max or Min or Last.

Having a look at the width of the panel (2735 pixels) this means that the panel has 2735 data points, or 30 second resolution, so there is no reduction in the raw data feeding into the 24 hour dashboard, so dumb luck means that I get actual min and max for the day :).

However, if I expand out to a week, I lose resolution, and I know why, now, so I know what to do.

[BuongiornoTexas]

@jasonacox, could we keep this open a bit longer?

I've had a quick look at the max functionality, and while I can see that they can be useful for some applications and some users, it's not universal. For instance, we mainly use the plots for shorter term load balancing (should we plug in the EV, run the dishwasher etc). So we tend to be interested in average load, while spikes don't matter that much. Unfortunately, this change makes it much harder to read this information as it effectively makes the data noisy. From my perspective, reporting max only is also effectively losing data - I'm more interested in typical system performance and not the extremes - and with this reporting, I lose the middle ground and the minimum loads.

I know one of the approaches to this issue in industrial applications is to run two curves - the mean curve, and then a maximum envelope (in our case, probably a rolling average for the max), which provides both sets of information in a fairly readable way. I know we can't do this here because too many plots, but there may be other alternatives that allow windows onto both sets of usage data.

[jasonacox]

I can revert, but can you show an example of what it does in terms of data loss? It would be good to document.

[BuongiornoTexas]

Sorry - poor phrasing. No data is lost - it's an issue of emphasis. I'm more interested in mid-point behaviour than peaks, which the max version does not show. Mainly because I use this data day to day do decide how to operate loads in the house. This is a reasonable example:

This is our solar generation from two days ago showing peak and mean. For us, it's much more useful to be able to see that the solar is putting out a typical 7.5kW supply between 10:00 and 12:30 - so we can then pick the loads we want to be operating at that time. If we use the max, then we would incorrectly assume we have typical supply around 9kW - overestimating our capacity by 1.5kW. This really becomes an issue on intermittently cloudy days, as average generation can be low and peaks can be very misleading - we already run into this issue a bit if we want to charge the EV and do other high load things in the house on these days (first world problems, eh?).

TLDR: Your solution for highlighting peaks in longer term data hurts my ability to manage my loads in the short term.

I'm not suggesting reverting just yet - just holding this issue open to see if more feedback comes in. I may well be in the minority in how I use this data, and can easily setup an alternative panel using mean if needed.

If typical use is to keep the panel on today/last 24 hours, then I'd suggest the old version is better, and the longer term analysis could be done with a separate diagnostic panel. If people are mostly running longer views, then the max version may be better.

[jasonacox]

Thanks @BuongiornoTexas ! That helps. I can't find a 24hr render that show a visible difference between mean() and max() but clearly your examples show it!

To have your cake and eat it to, you just need to bake two cakes. 😄 I have an idea based on your industrial application comment. Why not render a max (and min) as a translucent line to show the extremes?

Maybe something like this (7 day):

I would hide the legend and tooltips (or maybe keep). Doing that and adding the same for Solar, Home, Grid and Powerwall:

I would go so far as to say that it even adds a nice asthetic! I like the fant extremes. What do you all think?

It also strikes me that we have "max" listed in the legend for all series. That isn't really true and I wonder if it is even needed or useful.

[BuongiornoTexas]

Why not render a max (and min) as a translucent line to show the extremes?

More than happy to give it a try as a working compromise - I didn't suggest it initially, as I suspect a lot of people will find the graph just gets too busy - there's a lot going on in that plot already!

I would hide the legend and tooltips (or maybe keep).

I'd stay selective about this - keep the legend for the mean values, weather, charge and reserve. Hide legend and tooltip for max and min?

It also strikes me that we have "max" listed in the legend for all series. That isn't really true and I wonder if it is even needed or useful.

Agreed. I find the "Current" value much more useful! (I know it shows up elsewhere on the page - but this plot is pretty well the go to for me and my partner.)

[jasonacox]

Understood... makes sense! Here is my test implementation below. Does it look too busy?

24 Hours

7 Days

Dashboard.json

If you want to try it: Powerwall.json.txt

[BuongiornoTexas]

Looks good.

I've also realised - my previous comparison was based on a fixed 5min time interval - where the mean and max are very different. The latest version of the dashboard uses the automatically calculated $_interval - which seems to be short enough duration that mean, min and max are nearly the same. So my argument earlier may be moot (I'd need an intermittently cloudy day to be sure).

Regardless, min/mean/max version looks good.

[jasonacox]

Ok, that makes me feel a little better about the max() update. I love the look of the min-mean-max version, but I still wanted to be able to tooltip find the spikes for 7+ day views. I'm saving the min-mean-max version in our dashboards folder: https://github.com/jasonacox/Powerwall-Dashboard/tree/main/dashboards#alternative-dashboards for anyone wanting to explore or tweak it.

Keep checking on your system @BuongiornoTexas. Please do let me know if you can spot an issues with the 24hr view. I'm still open to reverting or pivoting to more of the min-mean-max rendering. I sure don't want to break the utility of the 24hr view.

[BuongiornoTexas]

Keep checking on your system @BuongiornoTexas. Please do let me know if you can spot an issues with the 24hr view. I'm still open to reverting or pivoting to more of the min-mean-max rendering. I sure don't want to break the utility of the 24hr view.

Will do. I guess it's time to update my 4 month old dashboard ...