Error in daily totals

[drRax]

Hi, I've been using this integration since November with a Vue 2. The integration works great, but I started to get a jump in the daily data about two weeks ago. Every day, the 1D information resets at 00:00, but it does a little jump on some days, close to 01:00 am. You can see it in the following screenshot:

It is not every day, and it is not precisely at 1:00 am (which could be from some daylight-saving time issue). In this example, on April 1st, the jump starts at 00:50.29 and ends at 01:04.29. The daily data does not constantly increase, something already raised in Issue #103, but this is something different as it does a big jump to a value similar to the maximum of the previous day and then back down to a value similar to the one it had before the jump. This problem could also be the cause of Issue #112 and #114 since I noticed that the jump is recorded in the Energy Dashboard and makes it show a much larger value than the daily total when compared to the 1D entity (see this example - where the early-morning jump almost entirely explains the difference):

Are there any settings or configurations I should be aware of correcting or is this something to do with the noisy data from the sensors?

[magico13]

The amount of time it stays up there is likely one whole cycle of the daily sensor, which only updates from the API once every 15 minutes. The steps in between those updates are the minute data being added to the last sensor reading to give a smoother update. So what's likely happening is that somehow the integration gets the wrong day's value from the API (whether Emporia's API is giving bad data or the integration is asking for the wrong thing somehow, is to be determined). I'm working on an update to swap to the "total" state class rather than "total_increasing" which might at least get it to balance out but I'm afraid I'm going to have to find a way to filter the data and ignore these weird artifacts.

Just thinking out loud here but I wonder if their API is just spitting out a cached value sometimes, like if they've got a maintenance period or something that makes it give bad data at that time. I have the same issue actually. I'm considering streamlining things a bit and swapping to just using 1 minute sensors and setting those up to work with the energy dashboard properly and just cut out the daily and monthly ones entirely since they are being a real pain and I'm not sure if anyone is using them for automations.

[jfroy]

I'm considering streamlining things a bit and swapping to just using 1 minute sensors and setting those up to work with the energy dashboard properly and just cut out the daily and monthly ones entirely since they are being a real pain and I'm not sure if anyone is using them for automations.

You mean doing the energy integration in HA and not bothering with the energy sensors in the integration (only exporting 1 minute power)?

[magico13]

only exporting 1 minute power

That would be the intent, yes. The integration would get the total energy used in the last minute and Home Assistant would automatically integrate it. If the API was more consistent I'd feel more comfortable doing that so I'm not currently going down that route. If we had local access maybe. I have instead been testing using the total state and while it still has issues the overall energy count doesn't get out of whack when the API gives old data and then corrects itself.

[jfroy]

Thanks. I just imported the 0.8-pre release and testing it (I just installed a Vue Utility Connect, so I don't really have data loss concerns).

[jfroy]

So far so good. I'm using the "1D" sensor for the energy dashboard. Not sure if the 1M makes more sense? I read in some other issues that the 1D sensor resetting to 0 every day might confuse HA?

[magico13]

Monthly doesn't work properly in the 0.8 prerelease, daily is still the one I'd recommend and that prerelease should mostly fix the issue that this GitHub issue references

[cromulent1]

I'm pretty sure I'm having the same problem, but mine is showing up in the form of the balance. Some days between 12:00 AM and 1:00 AM there will be a huge spike in my balance reading. I monitor every circuit, so in normal circumstances my balance reading is zero. I don't see these spikes in the Emporia app, but I'm not usually watching for it at 1:00 AM either. Anyway, just thought I'd add another data point for your analysis.

[andsee]

I'm using 0.8pre5 but still get the blips after 0:00 and also I've noticed other areas where it looks like it reports other old values for a minute on the minute sensor. (First 3 arrows)

Note too that the 1 minute sensor has missing sections.

[magico13]

Those are all pretty well known issues with the API unfortunately. I'm trying to address some of them but some there's not much I can do about since the Emporia API just returns bad data sometimes. With that build the totals should still even out correctly, since the drops balance out the jumps and don't trigger a new cycle, in the regular build the drops will cause a new cycle to start and that's why the totals will sometimes be double or more of the real value.

[andsee]

I was just looking at the HAS plugin and noticed it's using the get_device_list_usage method for instantaneous values. Now I'm thinking we should expect that to fail occasionally be it through local internet issues or whatever. I'm also thinking that sampling a 1 minute sensor every minute when there is an arbitrary internet delay is bound to throw up issues as our sampling time is occasionally before or after the server has updated it's minute stat.

Have you tried using the get_chart_usage and requesting the time period of now (or a bit in the future) and the last time we received data for?

[magico13]

The get_device_list_usage call returns all devices and channels at once in one call, get_chart_usage would require a separate call for every channel. For a Vue 2 with 16 extra monitors that's 17 requests per update. Emporia has already asked that the integration not make more than 2000 requests per day so that gives a minimum of one update per 5 minutes. If you've got more than one device then that means even more time between updates.

If only one channel is misbehaving I could maybe make one call to that endpoint every now and then, but that may end up with it's own off-sync issues. I'll have to think about that a bit and if the extra complexity is worth it.

[andsee]

Ok, good to know the rate limit, that does make solutions to the problems harder. Although I'd probably be tempted to accept the sensor updating once every 15 minutes or faster for fewer devices.

Alternatively as we know the timestamp of the instant reading we get back we can track how often we miss a minute (and ignore duplicate readings which I assume occur, you may already do this). If that is infrequent then as you suggest on the next request we could use the chart API. (What a pain!)

[magico13]

I'm not confident that the timestamp on the response to the device_list_usage call really means anything beyond what timestamp you pass in. It's been a bit since I did testing on that call, it might also just be that it ignores what time you pass in and always returns the most recent data.

The drops at the red arrows are when I'm making the call to the API for what I'm hoping will be the "true" value, which is done every fifteen minutes. The individual updates once a minute between those fifteen minutes are just the one minute data tacked onto the previous value. Most of the time those end up close enough to not be noticeable on the 15 minute updates, but the API seems to give random data sometimes. I am curious now if the returned timestamp at those arrows is wildly different, I think I'll throw a log message in if the difference from the requested time to the returned time is more than a minute and if so I may have to throw out that data and get new data, thanks for that idea.

[andsee]

Some of the following blip in the 1D I think can be explained by last_day_update being out of sync with a rounded 15 minute time.

At 0:00.24 we manually sum our minute data and spot midnight has passed so reset to 0 Up to 0:03:24 we are adding out minute samples At 0:04:24 15 minutes has passed since we last asked emporia for true values but thier server has probably not updated it's day count yet, the reading is likely to be the last time they did this. At 0:20:24 16 minutes have passed (Notice we have time passed > 15 minutes which may be a bug if we want to update every 15, even with >= 15 minutes we can still expect some updates to be 16 minutes apart though) Here we request another value from emporia and they have updated their value.