openevse_esp32_firmware icon indicating copy to clipboard operation
openevse_esp32_firmware copied to clipboard

Openevse loses mqtt connection

Open rvoosterhout opened this issue 4 years ago • 8 comments

Hello,

I'm using my MQTT connection to regulate my charging speed by the available amps on my main fuse. I do this by reading my smartmeter, converting that to the amount of amps available and forwarding this every second to openevse/excess/power which is used by the ECO mode in Grid mode.

Now I'm noticing the problem where my Openevse loses the MQTT connection, and won't reconnect. There is no problem with my MQTT broker Mosquitto, since the rest of my devices are still connected. When I restart the broker, Openevse won't reconnect. The danger here is that when I plug in my car, and Openevse has no MQTT connection anymore and lets say i'm starting cooking on my induction cooker, i'm going to blow my main fuse.

The only solution to reconnect Openevse to my MQTT broker is to completely restart the WiFi module, even tough I can still reach the openvse GUI by WiFi. Most of the time it starts up in AP mode after the restart and I need to reconnect to my WiFi again. It does remember the WiFi name and password, I only need to press connect.

I saw that RAPI commands are going to be deprecated and I am using 2 RAPI commands, to disable and enable Openevse when i'm not home so that other people can't use the charger. Could this be the issue of losing the MQTT connection?

What can I do for troubleshooting this issue? Is there a way to disable the Openevse when MQTT is not connected?

Solar PV Divert Wifi setup

rvoosterhout avatar Sep 15 '21 14:09 rvoosterhout

I just converted my rapi calls to http api calls, since the amount of RAPI packets is very high. I'll post here if this solves the problem.

rvoosterhout avatar Sep 15 '21 14:09 rvoosterhout

This could be similar to #135 especially if the WiFi connection is poor. These are really hard to replicate in the lab so probably need #244 to be able to get a clearer view on what is going on.

jeremypoulter avatar Sep 16 '21 10:09 jeremypoulter

Yes it looks like this is the same issue. The only difference is i'm not using Unifi AP's. But I don't suspect that those cause the issue. If I can be of any help, or you need remote access to my openvse, let me know.

EDIT: Converting my rapi calls $FE and #FD to http api calls didn't change anything, lost mqtt connection again, 6 hours ago.

rvoosterhout avatar Sep 17 '21 10:09 rvoosterhout

Maybe the upgrade the Espressif SDK will bring improvements to this problem. https://github.com/OpenEVSE/ESP32_WiFi_V4.x/pull/207

ruimarinho avatar Dec 23 '21 00:12 ruimarinho

I encountered the same issue. I'm also using RAPI through MQTT. But even if I don't, sometimes the MQTT connection is lost (needed Solar PV divert).

@jeremypoulter is onto something I believe, it may have to do with losing WiFi for a while. MQTT losing connection and OpenEVSE not re-subscribing when WiFi is back? That should be testable in a lab.

@rvoosterhout is there a specific reason why you are not using the standard functionality of OpenEVSE for Solar PV divert? I have DSMR5 and run Python code to calculate "Feed" as +I/-E from dsmr. Power1Calculator.py.pdf

pdhoogh avatar May 20 '22 17:05 pdhoogh

I can pretty reliably replicate this condition I believe, as generally when something external causes my MQTT connection to bounce (docker reboot, network update) MQTT in OpenEVSE fails to reconnect.

If I restart openevse_wifi, MQTT reconnects without any other intervention.

Ideally, MQTT should try to reconnect every minutes (or seconds?) in my opinion. Even one attempt an hour would eliminate my issue.

Alternatively maybe MQTT could reconnect whenever the WiFi / Ethernet connection bounces (again, plus seconds)?

A less elegant alternative would be a remote command to restart the OpenEVSE WiFi module.

davethetallguy avatar Aug 27 '22 20:08 davethetallguy

I have noticed this as well. Any time the connection is dropped (due to network outage at evse or mqtt broker, reboot/restart broker) OpenEVSE does not reconnect. I have also tried to save the MQTT settings in OpenEVSE but that does not work either. Only solution is to restart OpenEVSE. I have 3 openevse devices and 2 of them are not on the LAN so I have no way of automating a restart on them if this happens.

omriasta avatar Aug 30 '22 15:08 omriasta

It may be important to point out that while MQTT in OpenEVSE seems to not automatically recover after a lost connection, the wifi does and it is possible to reset the WiFi module using the WiFi Restart button on the OpenEVSE webpage. The restart recovers the MQTT connection. On average I need to do this once a week.

Over the past year of using OpenEVSE about two times the WiFi also did not recover. Cycling power to the OpenEVSE resolves the issue.

How the WiFi module of the OpenEVSE interacts with the home WiFi may play a role here. My home WiFi signal on the driveway - coming from the router - is too weak for a stable connection. I'm using Devolo Magic 2 Power Line Connection devices to extend the LAN and WiFi to parts in the house where there is no UTP cable or the WiFi is not strong enough. It meshes with the router. Specifically for the OpenEVSE I installed a PLC receiver in the garden house so OpenEVSE has a direct short path to a WiFi emitter. It works. But I had to disable the 802.11r feature on the PLC, OpenEVSE wifi cannot handle it. As soon as 802.11r is on, OpenEVSE looses the WiFi connection.

pdhoogh avatar Aug 31 '22 06:08 pdhoogh

I don't have a whole lot to add, but would like to mention I have this same issue as well. I have to reset the wifi module via the web interface about once a month. I'm not using anything Ubiquiti either. I haven't been able to tell if it's a dropped MQTT connection that triggers the bug, but I can tell you it's not a docker reset or mosquitto reset.

BScatterplot avatar Nov 15 '22 06:11 BScatterplot

Thanks @BScatterplot.

Since I have the WiFi PLC AP in the garden house blasting WiFi at the OpenEVSE from 5 meters away I have not had any issues... until I start the Robomow... the robotic mower uses a RF signal in a wire buried under the lawn to find its way. The emitter sits close to the WiFi AP. As soon as I turn on that signal for the mower, OpenEVSE loses the WiFi connection. My iPhone doesn't. So it does not look like the PLC AP failing.

OpenEVSE Wifi can be categorized as "flaky". I am happy it has Wifi because it does not require me to pull a CATx cable to the driveway. With the current WiFi performance I understand people may want a cabled connection. I'm not bored out by it sufficiently yet to give up. I am running openevse_wifi_v1 4.1.5.. WiFi seems more stable than with previous firmware versions, but I have not had sufficient runtime on it to make that a reportable observation.

I am wondering why the obvious solution is still not available: Power Line Connection. It is a general issue, not OpenEVSE's. It is possible to use that today by having a separate PLC device near the OpenEVSE and use the cabled connection option on the OpenEVSE. But if that could all be built in, it would be so much more convenient. It would require a generic PLC emitter. I believe they are all proprietary now. Standards organisations have not yet worked on interoperability of PLCs as they have on 802.x.

Another option may be a large external WiFi antenna. The connection to the inside may be done with a cable through a gland as is done with the SolarEDGE inverter. That keeps the IP rating up. If one is a handy crafter, it is a modification that one could do to the current OpenEVSE hardware.

Further on my PLC rant: SolarEdge uses PLC to talk to the optimizers sitting under my PV panels. But that is also proprietary. I cannot use PLC to connect the SolarEdge to the LAN, I need to use wired ethernet or WiFi. Because these are standardized.

pdhoogh avatar Nov 15 '22 09:11 pdhoogh

The MQTT re-connection logic right now is so unreliable I actually started migrating to the newer REST+websockets API, hoping I could turn off the MQTT integration altogether. However, it has some pretty rough edges so the new setup doesn't work quite as well as MQTT does (when the connection is alive), which is understandable given that I am using developer betas and the REST APIs are still shaping up :)

It'd be fantastic if @jeremypoulter could look into this while the newer APIs are developed. I think the testing is pretty easy too - just shut down your MQTT server (I'm using mosquitto) or restart your AP and OpenEVSE will remain disconnected until the WiFi module is restarted. Adding (or fixing) the MQTT reconnection logic would be a lifesaver!

I am actually running an nodemcu32 with an external attached inside the enclosure and the WiFi flakyness is gone. The only other situations where I'm facing disconnects are related to server upgrades, where I need to restart mosquitto regardless, or AP firmware updates.

ruimarinho avatar Nov 15 '22 19:11 ruimarinho

This should be improved in latest version

glynhudson avatar Mar 31 '23 01:03 glynhudson

I can pretty reliably replicate this condition I believe, as generally when something external causes my MQTT connection to bounce (docker reboot, network update) MQTT in OpenEVSE fails to reconnect.

If I restart openevse_wifi, MQTT reconnects without any other intervention.

Ideally, MQTT should try to reconnect every minutes (or seconds?) in my opinion. Even one attempt an hour would eliminate my issue.

Alternatively maybe MQTT could reconnect whenever the WiFi / Ethernet connection bounces (again, plus seconds)?

A less elegant alternative would be a remote command to restart the OpenEVSE WiFi module.

It is possible to remotely reset the wifi module with the following command:

curl --request POST --url http://EVSE_IP_ADDRESS/restart --header 'Accept: application/json' --header 'Content-Type: application/json' --data '{ "device": "gateway" }' --user "USERNAME:PASSWORD"

Don't use the --user option if you have no HTTP security defined.

odaigle avatar Aug 16 '23 21:08 odaigle