core icon indicating copy to clipboard operation
core copied to clipboard

APSystems EZ-1 M Inverter gets unavailable every 5 minutes

Open Roycinger opened this issue 9 months ago • 37 comments

The problem

Hello comm, I noticed that my APSystems EZ1-M inverter gets unavailable every 5 minutes as the graph of the power output has a periocial dip every 5 minutes.

Image

Looking at the logs, my suspicion seems to be true, the error shown is "Inverter returned an error" with no further details.

Image

I already tried to increase the log level by adding the following to my configuration.yaml: logger: default: warning logs: homeassistant.components.apsystems: debug Unfortunately, the inverter doesn't seem to provide more log details.

Is this a known issue? Can I provide more details for you?

Thanks!

What version of Home Assistant Core has the issue?

core-2024.12.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

apsystems

Link to integration documentation on our website

https://www.home-assistant.io/integrations/apsystems

Diagnostics information

home-assistant_2025-02-16T09-44-36.254Z.log

Example YAML snippet


Anything in the logs that might be useful for us?

2025-02-16 08:13:46.366 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-16 08:13:46.367 DEBUG (MainThread) [homeassistant.components.apsystems] Finished fetching APSystems Data data in 2.369 seconds (success: False)
2025-02-16 08:13:59.173 INFO (MainThread) [homeassistant.components.apsystems] Fetching APSystems Data data recovered

Additional information

No response

Roycinger avatar Feb 16 '25 09:02 Roycinger

Hey there @mawoka-myblock, @sonnenladengmbh, mind taking a look at this issue as it has been labeled with an integration (apsystems) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of apsystems can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Renames the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign apsystems Removes the current integration label and assignees on the issue, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)


apsystems documentation apsystems source (message by IssueLinks)

home-assistant[bot] avatar Feb 16 '25 09:02 home-assistant[bot]

Confirm this issue. Happened after updating yesterday to HASS Core 2025.2.4. Before there was no problems.

redwulf301 avatar Feb 16 '25 14:02 redwulf301

To add a detail: When the issue appears, it's cyclic, but the issue doesn't always appear, see graphs from yesterday and today (which is also reflected in the logs):

Image

Roycinger avatar Feb 16 '25 16:02 Roycinger

Some further analysis: I sniffed the wifi traffic between the inverter and the access point at the time when the issue appears. On most polling intervals, the tcp streams looks normal to me:

Image

When the issue appears, the datastream just delivers "FAILED" as a message:

Image

I can provide the .pcap file for further analysis if needed, I just can't attach it here.

My initial idea was that the polling interval might be too short but this wouldn't explain why the inverter gets unavailable every 5 minutes (pretty accurately in 5 minute steps) and the issue doesn't always appear, see screenshot from my last comment.

@mawoka-myblock can you assist with this matter?

Roycinger avatar Feb 18 '25 16:02 Roycinger

This behavior isn't normal and you're the first ones reporting it. It's quite common for the inverter to return an error a few times a day, but not like that. The first screenshot you've shown is about the status (Short Circuit, Offgrid). Those are supposed to be 0. The data delivering endpoint is /getOutputData. The getAlarm is unimportant.

Here's the docs for the "getAlarm" endpoint: https://github.com/SonnenladenGmbH/APsystems-EZ1-API/blob/main/APsystemsEZ1/init.py#L188

And here for the proper data endpoint: https://github.com/SonnenladenGmbH/APsystems-EZ1-API/blob/main/APsystemsEZ1/init.py#L215

So, I'd kindly ask you provide the capture data for the data endpoint. But I appreciate your will to help here!

mawoka-myblock avatar Feb 19 '25 19:02 mawoka-myblock

Hi @mawoka-myblock , thanks for getting back! I checked the pcap file for /getOutputData streams and they all look pretty normal over the entire capture, similar to this:

Image All the requests returned with "message":"SUCCESS"

What I'm wondering about is the 2 /getAlarm streams (like in the 2nd screenshot) with "message":"FAILED" over the entire capture (~1m30s). From my understanding this points to the inverter being unavailable (or whatever returns "message":"FAILED") which might be the root cause here.

Just to be safe: Wifi signal is good, no IP conflicts or anything of that sort.

Roycinger avatar Feb 19 '25 20:02 Roycinger

Hello,

I do have the same problem as @Roycinger. I installed the inverter at my parents home and presented with the same issue. It's not every 5 minutes, but now and then during the day. If you need something from me just say it.

MAGICCC avatar Feb 23 '25 20:02 MAGICCC

Have you all check the issue with the APsystem support? The problem is that I/we don't have any other information on what's going on inside the inverter. I only know what we all see and that is that some requests return "FAILED", some quite often, some only a few times a day.

mawoka-myblock avatar Feb 23 '25 20:02 mawoka-myblock

Have you all check the issue with the APsystem support? The problem is that I/we don't have any other information on what's going on inside the inverter. I only know what we all see and that is that some requests return "FAILED", some quite often, some only a few times a day.

No, not yet as I'm not sure if the issue is on the integration / API side or in fact on the hardware side. I'll send a message to APSystems and keep you in the loop here.

Roycinger avatar Feb 25 '25 09:02 Roycinger

Hello, is this one originaly 600w or 800w? which version of Firmware has the EZ-1, cant find this info above.

gotsila avatar Feb 25 '25 10:02 gotsila

Hello, is this one originaly 600w or 800w? which version of Firmware has the EZ-1, cant find this info above.

Should be be 800W originally. You're right, I missed that crucial piece of information. My inverter is running on firmware EZ1 1.7.5

Roycinger avatar Feb 25 '25 11:02 Roycinger

Thanks, i am wonder, my Inverter has EZ1 1.7.0 and on the app (direct connection) it says "Auf dem laufenden" also i cant update it. Any idea how to update if the app says "nothing to do"

gotsila avatar Feb 25 '25 11:02 gotsila

Hi, I also have the same problem. Every metric of the integration becomes unavailable for 13s every 5min (available for 4m47s). The repetition of the gap is very consistent and all day long.

My inverter is EZ1-M 800W running firmware 2.0.1_b. I've got it in a Zendure kit and it was originally running 1.2.0_b but thanks to AP-System Technical support I upgraded it to this version in order to be able to enable local mode.

My homeassistant instance is HAOS v2025.2.5 on a raspberry pi 5.

remip2000 avatar Feb 25 '25 15:02 remip2000

Just as information, I have the same issue:

2025-02-25 12:26:51.866 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 12:31:51.899 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 12:36:52.032 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 12:41:52.065 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 12:46:51.996 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 12:51:51.931 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 12:56:51.954 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:01:52.102 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:06:52.029 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:11:51.962 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:16:51.886 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:21:52.039 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:26:52.062 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:31:51.895 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:36:52.020 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:41:51.951 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:46:51.994 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:51:52.028 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 13:56:51.962 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:01:51.881 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:06:51.922 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:11:51.957 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:16:51.882 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:21:51.913 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:26:52.056 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:31:51.982 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:36:52.004 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:41:52.051 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:46:51.862 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:51:51.998 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 14:56:51.928 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 15:01:52.068 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 15:06:51.994 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error
2025-02-25 15:11:51.920 ERROR (MainThread) [homeassistant.components.apsystems] Error fetching APSystems Data data: Inverter returned an error

It's not every day, but most of them. This is why i created #131930, to at least catch the error. I have this since ever. Firmware: 1.7.7

Thomas55555 avatar Feb 25 '25 15:02 Thomas55555

No, not yet as I'm not sure if the issue is on the integration / API side or in fact on the hardware side. I'll send a message to APSystems and keep you in the loop here.

Will do as well in the next days...

Thanks, i am wonder, my Inverter has EZ1 1.7.0 and on the app (direct connection) it says "Auf dem laufenden" also i cant update it. Any idea how to update if the app says "nothing to do"

That's strange because I am on 1.7.1 for a EZ1-M and according to some german posts I found it's not possible to update when the inverter is running on local mode...

MAGICCC avatar Feb 25 '25 16:02 MAGICCC

Some of you miss the obvious: InverterReturnedError. The inverter returned an error. But it doesn't say at all where the error lies, so it's a black box telling us that sth is wrong. There's nothing we/I can do about that. When the data source returns an error, the sensor goes unavailable. Home Assistant rule. And as the error originates from the black box created by APsystems, they are the ones providing support for that. So please stop reporting the InverterReturnedError error to HA or in the repo over at Sonnenladen. Reach out to the support.

mawoka-myblock avatar Feb 25 '25 17:02 mawoka-myblock

I got response from the APSystems support which didn't shed any light: Dear Customer, We suggest you use the EasyPower app to monitor your device. If the problem still exists, please feel free to come back to us. And a new version of API will be released soon, some features may be affected. thanks I asked for elaboration on the new API version, will keep you uptodate.

I agree with @mawoka-myblock, still the exception needs to be handled in a more graceful way imho. Hopefully APSystems support is willing to get into analysis and stop answering with 1st level support style textblocks.

Roycinger avatar Feb 26 '25 10:02 Roycinger

I've patched my APSystems integration to circumvent this problem. Since a few days, I'm not getting any InverterReturnedError, and the values never transition to unavailable (I've added an automation to warn me if this is the case, and it's not firing anymore). Before, I was getting hundreds of those every day, as most of you I suppose. My modification just adds a few retries when querying the API on the inverter, and it seems to be enough, as the errors seem very transient. It'll still return InverterReturnedError after all retries have failed.

I'm not opening a PR because this is not a "fix", just a (arguably dirty) workaround, hence it would probably be refused. For reference, if you want to apply it to your install too:

--- coordinator.py	2025-03-21 21:28:27.000000000 +0100
+++ coordinator.py.patched2	2025-03-25 13:04:00.873903311 +0100
@@ -18,6 +18,7 @@
 
 from .const import DOMAIN, LOGGER
 
+import asyncio
 
 @dataclass
 class ApSystemsSensorData:
@@ -70,11 +71,16 @@
         self.device_version = device_info.devVer
 
     async def _async_update_data(self) -> ApSystemsSensorData:
-        try:
-            output_data = await self.api.get_output_data()
-            alarm_info = await self.api.get_alarm_info()
-        except InverterReturnedError:
-            raise UpdateFailed(
-                translation_domain=DOMAIN, translation_key="inverter_error"
-            ) from None
-        return ApSystemsSensorData(output_data=output_data, alarm_info=alarm_info)
+        for tries in range(5):
+            await asyncio.sleep(tries)
+            try:
+                output_data = await self.api.get_output_data()
+                alarm_info = await self.api.get_alarm_info()
+            except InverterReturnedError:
+                if tries >= 4:
+                    raise UpdateFailed(
+                        translation_domain=DOMAIN, translation_key="inverter_error"
+                    ) from None
+                else:
+                    continue
+            return ApSystemsSensorData(output_data=output_data, alarm_info=alarm_info)

The file is located at /usr/src/homeassistant/homeassistant/components/apsystems/coordinator.py (in the homeassistant container if you're using HAOS)

speed47 avatar Mar 25 '25 17:03 speed47

Awesome, thank you! To those that may also complain about this not being part of the official integration: It is the proper way to handle retries: Throw an error. HA automatically retries within a time frame. So this more or less bypasses HA's retry system which is not wanted, so it's not implemented.

mawoka-myblock avatar Mar 25 '25 19:03 mawoka-myblock

I can confirm for HA 2025.4.1, Inverter FV 1.7.5. I'm pretty sure I didn't have any problems before HA 2025.4.0 (maybe also with 2025.4.1 everything was still fine).

My suspicion would be this commit https://github.com/home-assistant/core/pull/140086/files, however it doesn't explain why you have experienced this long before.

lu-x avatar Apr 07 '25 06:04 lu-x

The commit has absolutely nothing to do with it. Just complain to APsystems.

mawoka-myblock avatar Apr 07 '25 13:04 mawoka-myblock

The commit has absolutely nothing to do with it. Just complain to APsystems.

Ok. Just wanted to point out that for me there have been no problems before last weekend.

lu-x avatar Apr 08 '25 09:04 lu-x

I've patched my APSystems integration to circumvent this problem. Since a few days, I'm not getting any InverterReturnedError, and the values never transition to unavailable (I've added an automation to warn me if this is the case, and it's not firing anymore). Before, I was getting hundreds of those every day, as most of you I suppose. My modification just adds a few retries when querying the API on the inverter, and it seems to be enough, as the errors seem very transient. It'll still return InverterReturnedError after all retries have failed.

I'm not opening a PR because this is not a "fix", just a (arguably dirty) workaround, hence it would probably be refused. For reference, if you want to apply it to your install too:

The file is located at /usr/src/homeassistant/homeassistant/components/apsystems/coordinator.py (in the homeassistant container if you're using HAOS)

Thank you very much for the solution, even if it's dirty. Unfortunately, it didn't change much for me:

Image I made the change as a custom_component (so the apsystems integration from the core gets overwritten by the custom_components folder).

Roycinger avatar Apr 08 '25 10:04 Roycinger

I can confirm, since the last converter firmware update it's a drop each 5 minutes. This sucks...

DanielMisch avatar Apr 22 '25 07:04 DanielMisch

i'am afraid there are many different FW Versions on many EZ-1. on my one it is 1.7.0 and i cant update to newer on, because the app "says" i have the newest Which one do you mean, if you say "...the last..."?

gotsila avatar Apr 22 '25 16:04 gotsila

have the same ! behavior: drops down to "0" Watt for a minute or 2 then serving again some hundreds watt, as an average it is arround 200W in the 5 minutes overview. Within the AP Easy Power App "live" data and updating the data quickly you see these drops to zero.

I have the inverter since~4 weeks, first 3 weeks work well, since saturday (26.04) the issue , the same next day. The issue start and ends more or less at the same time, but having reviewed it it looks like reaching an amount of Watt when it starts dropping. e.g. starts dropping at ~11 AM (500Watt+ and ends ~ 4 PM ~450Watt) and Inverter is 800W. Changed the output power to lower numbers, today 500W and it looks to work constantly in that case, raising it will falldown again.

Version 1.7.5 , 800W, not using local mode, only Cloud with "AP easy power" and "solakon". The AP systems App does not through out any alarm, also not the App from the APsystems OEM provider "solakon". So, not sure where to search for the origin without any hint from the inverter why/reason it drops down ( also red light indicator is on).

MarkusH1206 avatar Apr 29 '25 09:04 MarkusH1206

If it's also dropping in the official app, reach out to APsystems.

mawoka-myblock avatar Apr 29 '25 13:04 mawoka-myblock

I can make sure that inverting and deliver energy works while reported outage.

DanielMisch avatar Apr 29 '25 17:04 DanielMisch

Installed EZ-1 today. And I have the same problem very 5 minutes become unavailable. Firmware 1.7.0. I see that micro inverters is not dropping the networks and becomes connected since I installed 3 hours ago. But become unavailable in home assistant.

Any solution ?

aperezva avatar Jun 07 '25 13:06 aperezva

Same here for one of my two EZ1, never had this problem before, firmware 1.7.1 installed a year ago, HA 2025.5.3. This behavior cause my time based energy automations not to work properly ... How to help ?

odeBuXTeR avatar Jun 09 '25 15:06 odeBuXTeR