tsun-gen3-proxy icon indicating copy to clipboard operation
tsun-gen3-proxy copied to clipboard

Frequent data gaps with MS600 (GEN3) inverter

Open s-allius opened this issue 1 year ago • 9 comments

With an MS600 (GEN3) inverter with DSP version 5.1.09, I regularly see gaps in the data transmission. An initial analysis has shown that the TCP connection is stable and TCP keep-alive packets are also being answered.

We may be able to get the inverter out of this state by reestablishing the connection. I am not aware of a way to restart the DSP using a magic packet. I will try to recognize the situation by time monitoring and then close the TCP connection from the proxy side (sending a FIN packet) in the hope that the inverter will then send data again immediately.

Bild 10 05 24 um 20 23

s-allius avatar May 10 '24 18:05 s-allius

I see the same gaps with my MS600 (using dev-0.8.0, 10 sec interval. I switched off cloud. The gap is very regularly: every ~13 min, 55-58 request (or similar). Looks like your gaps have the same frequency - interval 1 min, gap each hour. Interesting point: the gaps are shorter too -> same amount of request before connection reset?

Maybe the MS600 accepts only a max number of requests per connection. Reset connection by default after 50 requests?

Bildschirmfoto vom 2024-05-10 21-57-54

Same with 20 sec request interval. It works for 55-58 requests. Followed by 11-18 unanswered requests. Than:

  1. a new connection (new port) is accepted.
  2. old connection is reset by peer. (server loop stops.)
  3. modbus 0x70 requests are send to now connection and are answered
  4. (BUG: 0x70 requests are send to closed connection too)

roethigj avatar May 10 '24 20:05 roethigj

That is interesting. Today I also saw very long gaps (up to 15 minutes). So not only the Modbus responses are missing, the inverter itself is not sending any packets either. During the analysis, I noticed that I had forgotten to call the close() function to delete internal structures during a graceful disconnect. Half-open TCP connections could be the result. Fix is checked in the 0.8.0 branch. (That should fix your seen bug - point no 4)

Today I also implemented a monitoring function that resets the connection after 2.5 minutes without receiving any packets. This will hopefully keep the dropouts to a minimum. However, the actual cause is not yet clear to me. Which DSP firmware do you have? I have V5.1.09 on it after an update to 800VA.

s-allius avatar May 11 '24 22:05 s-allius

I have an idea.

Before the Modbus payload, there are four bytes (offset 0x17 directly after the message ID), which could be a packet counter or a reset timestamp. I have set this byte to a fixed value of b'\x00\x01\xa3\x28'. But in my traces, I saw that this value changes with every request from the TSUN cloud. I have five examples:

2024-04-30 10:32:37   00 01 a3 06
2024-04-30 10:37:41   00 01 a3 08
2024-04-30 10:52:54   00 01 a3 0e
2024-04-30 11:33:28   00 01 a3 14
2024-04-30 15:20:23   00 01 a3 28

And you are wright, the resets are very regular. We can see it in the Power On Timegraph: Bild 12 05 24 um 10 50

Now I check, what will happen with all four bytes set to 0x00.

s-allius avatar May 12 '24 08:05 s-allius

Today I saw a gap from 13:15 until 18:00. My timeout handler worked as expected and close the TCP connection properly (server loop stoped). But the inverter didn't establish a new connection.

Maybe I have a network problem. It is possible, that the inverter roamed to another Accesspoint in that moment. (I have three APs here) But wenn the inverter don't make a reconnection, then I see no chance to mitigate the problem from the proxy side.

I will invest some more time to analyse it.

s-allius avatar May 13 '24 17:05 s-allius

I also had gaps larger than usual (AVM AP) this afternoon, so likely no local issue on your side.

I attach my report from TSUN website.

device-report-2024051319514161.xlsx

BTW: I do not use the proxy.

jkrwdf avatar May 13 '24 17:05 jkrwdf

Today a German TSUN handler wrote on his web page, that beginning with the 20th of May customer can switch to the Smart App even for GEN3 inverters. Maybe this will include a new firmware which supports the SolarmanV5 protcol.

The article is only in German available: https://priwatt.de/blog/tsun-wechselrichter-update-800-watt/

s-allius avatar May 14 '24 19:05 s-allius

With the proxy version 0.10 the connection is really stable here. But I have to deactivate the polling mode. It seems to me, that the inverter has a problem with the frequent MODBUS request every minute.

To deactivate the polling mode add a line to your config.toml and set modbus_polling to false for the GEN3 inverter

See also Inverter config

s-allius avatar Aug 10 '24 14:08 s-allius

Did you try to further analyze this? I am also seeing data gaps and reconnects every 2-3 hours on my TMS800 (Gen3). Disabling the polling_mode solves it, but I would like to have more frequent updates.

It seems that there is still something wrong with accepting new connections and closing old connections. Because less than a minute after the disconnect, the inverter is trying to connect again. However it seems the proxy isn't sending any data and closing the connection again after some time. It then takes another connection approach from the inverter to successfully start the proxy again.

´´´ '2025-02-07 10:47:39 INFO | data | [balcony/] MODBUS: grid.Output_Power : 484.0W' '2025-02-07 10:48:37 INFO | data | [balcony/] MODBUS: grid.Output_Power : 502.2W' '2025-02-07 10:49:39 INFO | data | [balcony/] MODBUS: grid.Output_Power : 520.4W' '2025-02-07 10:51:04 INFO | data | [balcony/] Modbus timeout (FCode: 3 Reg: 0x300f, 7)' '2025-02-07 10:51:39 WARNING | conn | [balcony/:22] Dead connection timeout (120s) for ('172.23.0.2', 5005)' '2025-02-07 10:51:39 INFO | conn | [balcony/:22] Server loop stopped for r('192.168.100.180', 54350)' '2025-02-07 10:51:39 INFO | conn | [balcony/:22] disc client connection: [balcony/:23]' '2025-02-07 10:51:39 INFO | conn | [balcony/:23] Peer closed. for ('172.23.0.2', 60370)' '2025-02-07 10:51:39 INFO | conn | [balcony/:23] Client loop stopped for l('172.23.0.2', 60370)' '2025-02-07 10:52:20 INFO | conn | [G3:24] Accept connection from ('192.168.100.180', 54351)' '2025-02-07 10:54:45 INFO | conn | [G3:24] Peer closed. for ('172.23.0.2', 5005)' '2025-02-07 10:54:45 INFO | conn | [G3:24] Server loop stopped for r('192.168.100.180', 54351)' '2025-02-07 10:54:52 INFO | conn | [G3:25] Accept connection from ('192.168.100.180', 57773)' '2025-02-07 10:55:11 INFO | root | [balcony/] Connect to ('logger.talent-monitoring.com', 5005)' '2025-02-07 10:55:11 INFO | root | [G3:26] Connected to ('logger.talent-monitoring.com', 5005)' '2025-02-07 10:55:12 INFO | data | [balcony/] GEN3: collector.Collector_Fw_Version : RSW_400_V1.00.17' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: env.Inverter_Status : 1' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: events.Inverter_Alarm : 0' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: events.Inverter_Fault : 0' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: events.Inverter_Bitfield_1 : 0' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: events.Inverter_bitfield_2 : 0' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: grid.Output_Power : 606.1W' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: total.Daily_Generation : 0.53kWh' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: total.Total_Generation : 44.6kWh' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: inverter.Max_Designed_Power : 800W' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: other.Prod_Compliance_Type : 2' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: inverter.Output_Coefficient : 100.0%' '2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: inverter.Version : V5.0.17' '2025-02-07 11:01:03 INFO | data | [balcony/] MODBUS: grid.Output_Power : 634.7W' '2025-02-07 11:02:03 INFO | data | [balcony/] MODBUS: grid.Output_Power : 678.2W' ´´´

´´´ '2025-02-12 13:47:38 INFO | data | [balcony/] MODBUS: grid.Output_Power : 246.0W' '2025-02-12 13:48:40 INFO | data | [balcony/] MODBUS: grid.Output_Power : 262.3W' '2025-02-12 13:49:11 INFO | data | [balcony/] GEN3: grid.Output_Power : 255.7W' '2025-02-12 13:49:11 INFO | data | [balcony/] GEN3: total.Daily_Generation : 1.2kWh' '2025-02-12 13:49:11 INFO | data | [balcony/] GEN3: total.Total_Generation : 54.54kWh' '2025-02-12 13:49:55 INFO | data | [balcony/] MODBUS: grid.Output_Power : 259.1W' '2025-02-12 13:51:21 INFO | data | [balcony/] Modbus timeout (FCode: 3 Reg: 0x300f, 7)' '2025-02-12 13:51:55 WARNING | conn | [balcony/:239] Dead connection timeout (120s) for ('172.23.0.2', 5005)' '2025-02-12 13:51:55 INFO | conn | [balcony/:239] Server loop stopped for r('192.168.100.180', 56679)' '2025-02-12 13:51:55 INFO | conn | [balcony/:239] disc client connection: [balcony/:240]' '2025-02-12 13:51:55 INFO | conn | [balcony/:240] Peer closed. for ('172.23.0.2', 58116)' '2025-02-12 13:51:55 INFO | conn | [balcony/:240] Client loop stopped for l('172.23.0.2', 58116)' '2025-02-12 13:52:26 INFO | conn | [G3:241] Accept connection from ('192.168.100.180', 56680)' '2025-02-12 13:54:51 INFO | conn | [G3:241] Peer closed. for ('172.23.0.2', 5005)' '2025-02-12 13:54:51 INFO | conn | [G3:241] Server loop stopped for r('192.168.100.180', 56680)' '2025-02-12 13:54:58 INFO | conn | [G3:242] Accept connection from ('192.168.100.180', 55067)' '2025-02-12 13:55:17 INFO | root | [balcony/] Connect to ('logger.talent-monitoring.com', 5005)' '2025-02-12 13:55:18 INFO | root | [G3:243] Connected to ('logger.talent-monitoring.com', 5005)' '2025-02-12 13:55:19 INFO | data | [balcony/] GEN3: collector.Collector_Fw_Version : RSW_400_V1.00.17' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: env.Inverter_Status : 1' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: events.Inverter_Alarm : 0' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: events.Inverter_Fault : 0' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: events.Inverter_Bitfield_1 : 0' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: events.Inverter_bitfield_2 : 0' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: grid.Output_Power : 490.0W' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: total.Daily_Generation : 1.26kWh' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: total.Total_Generation : 54.6kWh' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: inverter.Max_Designed_Power : 800W' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: other.Prod_Compliance_Type : 2' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: inverter.Output_Coefficient : 100.0%' '2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: inverter.Version : V5.0.17' '2025-02-12 14:01:09 INFO | data | [balcony/] MODBUS: grid.Output_Power : 415.5W' '2025-02-12 14:02:10 INFO | data | [balcony/] MODBUS: grid.Output_Power : 601.7W' ´´´

Phil1pp avatar Feb 12 '25 16:02 Phil1pp

Hi Phil1pp,

Did you try to further analyze this? I am also seeing data gaps and reconnects every 2-3 hours on my TMS800 (Gen3). Disabling the polling_mode solves it, but I would like to have more frequent updates.

you are right on models with a serial number starting with R17 or R47 the modes polling isn't stable. The inverter disconnect aber a few hour regular. And there is no solution for it, because the inverter internally hangs, which must be fixed by a reconnect.

So if you need/want faster updates, you must accept this problem.

s-allius avatar Apr 01 '25 21:04 s-allius