Split-Single-Phase-Energy-Meter icon indicating copy to clipboard operation
Split-Single-Phase-Energy-Meter copied to clipboard

Periodic disconnects - add scheduled reboot?

Open amil109 opened this issue 4 years ago • 44 comments

Hi CircuitSetup,

I'm really enjoying my Split Single Phase Energy meter, with only one issue - every now and then, approximately every 10 days, it disappears from the network, and requires a restart.

I've had other ESP devices in the past that needed a periodic reboot (once / week, etc) to perform reliably - I could never quite get to the bottom of what the issue was, and didn't have the time to resolve it 'properly' either. After reading a few things online, I thought maybe the meter might be swapping APs or something (multiple APs in the house), and having issues, but I've switched it to an SSID that is only on one AP (with only the ESP on the SSID), so it should not be this.

Please note that I have read "Using web interface and/or leaving it open causes ESP32 memory leak/crash" - I'm running v2.5.5, so should not be this either (although I do always use Chrome).

Right now, I've got the meter connected through a smart outlet, so if it goes down I can disconnect and reconnect it remotely, but I would rather have this automated on the ESP itself.

Can you help with solving this issue? I can certainly send logs (let me know how), or, alternatively, could you add a scheduled reboot to the WebUI? I think this would 'solve' the issue.

Has anyone else reported this?

Kind regards,

Andrew

amil109 avatar Feb 16 '20 13:02 amil109

Hi @amil109 - I'm sorry it's giving you issues! It looks like you have the kit that came programmed. When you updated to v2.5.5, did you compile it, or just load the binary via the web interface? I ask because if you compiled and the libraries weren't up to date, that could still cause memory leak issues.

It seems that some people still do have this issue, but it's usually related to the signal strength, and the ESP32 being able to reconnect to the AP when it looses the connection. After trying to reconnect, it will go into AP mode, which uses a lot of power. It can sometimes cause the brownout detector to trip, and the ESP32 to freeze.

What specific AC transformer are you using? Is it at least 1000mA?

CircuitSetup avatar Feb 16 '20 16:02 CircuitSetup

Hi @CircuitSetup ,

Thanks for the quick response!

I used the binary to flash via the web interface.

My AC transformer is 2000mA, it's one for 230V, not listed, so I've got a calibrated value in the WebUI.

The ESP is very close to my router (~1m, they're in the same cupboard) I don't know if this might be an issue?

Kind regards,

Andrew

amil109 avatar Feb 16 '20 22:02 amil109

Your transformer is outputting enough current. What is the voltage output? Also is it one that plugs into the wall? Some people have had some issues with low grade transformers.

The closer to ESP32 is to the AP, the better. That's probably not the issue either.

CircuitSetup avatar Feb 17 '20 21:02 CircuitSetup

Hi @CircuitSetup ,

It is a 'wall wart' type transformer, 9V AC 2000mA output - this is the link.

If you can recommend a better / different brand for use in the UK, please let me know - I will happily purchase it!

Kind regards,

Andrew Mills

amil109 avatar Feb 17 '20 23:02 amil109

There isn't too much detail about how that transformer works on that page, but I can see that it's passed UK standards, so it's probably doing what it is supposed to do.

In EmonCMS, you can also take the available memory and make that an input, graph it, and see if it running out of memory. If it is there is an unexpected leak somewhere. If not, then it's likely a connection/power issue.

If it is a power issue the only other suggestion I would have before trying a different ESP32, is to put a 10uf to 50uf through hole style capacitor between the 3v3 and gnd pins on the ESP32. This can either be soldered in or put in the pins that hold the ESP32 on the underside of the adapter board.

Let me know what you find.

CircuitSetup avatar Feb 21 '20 16:02 CircuitSetup

One other thing - Are you sending any data via MQTT? I just fixed a bug with that when it got disconnected.

CircuitSetup avatar Feb 21 '20 19:02 CircuitSetup

Yes - I am sending MQTT data to my OpenHab instance - should I stop this? TBH I'm not using it right now. I'm also happy to flash a Beta with the fix if you have it?

amil109 avatar Feb 22 '20 16:02 amil109

I fixed an issue in the latest release that would cause MQTT to try to reconnect for 14 min straight if the connection was lost. It was only supposed to do this for 5 seconds, then every 10 seconds. This could cause the ESP32 to lock up, which may be what you are experiencing. Please flash the latest bin to the ESP32 via the web interface, and let me know if this helps.

CircuitSetup avatar Feb 28 '20 16:02 CircuitSetup

I've flashed it now - I'll keep you posted!

Prior to this, it worked for ~15 days without issue, so it might be a hard one to detect. If I kill my MQTT server and wait, that should replicate it right? I can certainly try that. It's completely possible that the energy monitor failures were aligned with me messing around with the MQTT server, and I just didn't realise in the past.

amil109 avatar Feb 29 '20 00:02 amil109

Hi @CircuitSetup,

My device has been operating without issue for a week since flashing - I think if it lasts one more week without issue we can consider this topic closed! Thanks very much for your help. I'll post if I have any issues in the next 7 days.

amil109 avatar Mar 07 '20 11:03 amil109

Great! I'm glad to hear it's working better for you now.

CircuitSetup avatar Mar 09 '20 12:03 CircuitSetup

Hi @CircuitSetup I got my device and it's great. Works really well, besides the disconnects. I thought it was my wifi signal as my AP was very far.. I have since got another AP in the mix ( Unifi AP Setup, network is solid ) and I get a great -40 dBm connection. Yet at random.. It can happen more than once a day, but it's at minimum once a day I will need to unplug the power supply and plug it back in.. If I am running a consistent ping it comes back really fast, 3-5 seconds. Is that even a full boot? Any idea what could be going on? I am only running Emoncms also .. No MQTT from this device.

Free RAM: 213K Flash Size: 4096K Version: v2.5.7

Oh , power supply was included in your provided kit. I am using a full kit from you.

McGeaverBeaver avatar Mar 15 '20 02:03 McGeaverBeaver

@CircuitSetup John, Could you please advise.. It's really annoying this is not a stable device..

McGeaverBeaver avatar Mar 21 '20 14:03 McGeaverBeaver

Sorry @McGeaverBeaver, I overlooked your initial message.

Did you update the firmware from the provided bin or did you compile from source?

Also, when it randomly disconnects is there a browser window open anywhere?

It may help to collect data for available ram in emoncms to see if the esp is running out, then freezing. That would indicate a memory leak. I've seen this happen when the software isnt closing connections correctly.

CircuitSetup avatar Mar 21 '20 15:03 CircuitSetup

@CircuitSetup No problem.. I have updated the FW from the provided bins.. You have 2 , do I need both uploaded? I have done that src.spiffs & src.ino.esp32.. It's on v 2.5.7 and said successful after both uploads..

No browsers opened when this happens. I only have it logging to Emoncms..

I can also check the lights before I reboot it next time this happens but I think the red light is on and no blue light..

is a reboot normal to only take a few seconds? unplug power supply from wall and plug it back in..

McGeaverBeaver avatar Mar 21 '20 23:03 McGeaverBeaver

You shouldn't need to update the SPIFFS file, no. The latest just has some text changes in the admin interface.

It sounds like the esp32 is freezing. Are you using the transformer that came with the kit or another one?

If you can, I would monitor the available memory, which you should see as an input in emoncms. This could indicate a memory leak.

CircuitSetup avatar Mar 26 '20 19:03 CircuitSetup

Ok I have added RAM to start logging.. I will let you know the results .. Currently sitting at 225464

Oh and using the transformer that came with the kit.. Full kit, I have made no modifications to it.

McGeaverBeaver avatar Mar 26 '20 22:03 McGeaverBeaver

@CircuitSetup Hey , So it happened again, here is a short clip of showing the ram usage .. doesn't look to be memory related.. https://fastshare.space/33d36b3b6RS.mp4

McGeaverBeaver avatar Mar 31 '20 22:03 McGeaverBeaver

@McGeaverBeaver I'm sorry it's still giving you issues. If it's not memory, the ESP32 is either having issues with power, or staying connected to your AP. Is your network configured with any repeaters where 2 physical APs have the same name? Or 5ghz and 2.5ghz bands with the same name?

CircuitSetup avatar Apr 03 '20 12:04 CircuitSetup

Hey @CircuitSetup

So I don't think it will be my WiFi , I have a Unifi setup with 2 of the latest AP Pros. I also have a " IoT " SSID set for only 2.4ghz.. I have over 30 devices on my WiFi and none have issues.

As for power issues , you mean delivery? I can ensure you there are no power issues in my home when this happens..

Any way to get a daily reboot integrated, It seems that would just be the easy fix here as unplugging the transformer from the wall and plugging it back again right away, not waiting.. and if I have a ping up on another system the network comes back in just a few seconds..

so I ask again, how long does this normally take to reboot from full power loss? I really don't feel like to open my panel up and d/c the CT's so I can test this as there looks to be retained power from the CT's..

photos of the device / connection to power

https://fastshare.space/c55f83e1eRS.jpg

https://fastshare.space/4c7fd62e5RS.jpg

McGeaverBeaver avatar Apr 05 '20 12:04 McGeaverBeaver

Thanks for clarifying!

I meant to say, regarding power issues, is that the ESP32, for some reason, requires more power than what is available, or there is just something wrong with it. When this happens, it could just be losing the wifi connection over and over, then freezing when trying to reconnect to the AP.

Normally, it should take around 10 seconds to reboot and re-connect to the programmed AP.

The right way to do it, is to set up some kind of watchdog - if the meter doesn't send data for x amount of time. Power cycle the AC adapter via smart plug. But that's still a bandaid.

Either way, I want to try a new (programmed) ESP32. If you can provide your order number, I'll get it mailed out.

Also, nice homelab!

CircuitSetup avatar Apr 06 '20 19:04 CircuitSetup

@CircuitSetup Thanks John, Fair enough it would only be a bandaid and not identify root cause..

My Order 89095 from crowd supply. I didn't think about it until last night my Konnected.IO also runs on an ESP32 and it doesn't have the same disconnect issues so maybe you are right and onto something..

Thanks again John!

McGeaverBeaver avatar Apr 06 '20 22:04 McGeaverBeaver

Oh okay - you must not be in my order system. Please send me your mailing address via this form: https://circuitsetup.us/index.php/contact/

CircuitSetup avatar Apr 07 '20 18:04 CircuitSetup

@CircuitSetup My bad.. I sent it twice , Forgot the address the first message. Thanks Again John!

McGeaverBeaver avatar Apr 08 '20 22:04 McGeaverBeaver

Great, I'll send it out today!

CircuitSetup avatar Apr 09 '20 14:04 CircuitSetup

@CircuitSetup Thanks!! I'll give you an update after I swap it out and wait a couple days!

McGeaverBeaver avatar Apr 10 '20 14:04 McGeaverBeaver

@CircuitSetup Hey John, So I got the new ESP32 , swapped it out and I've def noticed that it's been more stable.. But I've had it disconnect on me twice without recovery.. Should it not remain to try to connect to AP and go back into AP mode every 5 minutes? I don't even see the ESP broadcasting it's SSID right now with the current failure and lights on it are : On your board - Green Light, Red / Blue Light flashing. On the ESP - solid red light, blue light flashing. not sure if that helps you.

McGeaverBeaver avatar May 09 '20 11:05 McGeaverBeaver

@McGeaverBeaver It should try to reconnect to your AP every 5 min if it fails, yes (assuming the ESP32 isn't frozen), and will be in AP mode otherwise.

The blue LED will indicate the status:

  • blinks every 2 seconds - the ESP32 is in AP mode, and it should be broadcasting.
  • blinks every 4 seconds - connected to your AP.
  • doesn't blink at all, then the ESP32 is likely locked up.

I just updated EmonESP to v2.6, which has a number of bug fixes, adds a watchdog timer, and replaces the MQTT library. See details here: https://github.com/CircuitSetup/Split-Single-Phase-Energy-Meter/releases/tag/v2.6

CircuitSetup avatar May 27 '20 20:05 CircuitSetup

Edit:

@CircuitSetup So I had it on a relay and was rebooting it if it stopped responding to ping.. Was working well, didn't have any gaps in data for days as the reboots take only a second.

I did as you suggested and updated to - v2.6-src.ino.esp32.bin and I lost Emoncms communcation.

Managed to downgrade back to 2.5.7 and Emoncms is working fine again.. Not sure why it wouldn't work on 2.6.

McGeaverBeaver avatar May 31 '20 19:05 McGeaverBeaver

I'm not sure why you wouldnt be able to access the interface with v2.6. Was the blue light blinking on the esp32? If so, it was either in AP mode or connected to your AP with a new IP. Maybe your AP thought it was a new device?

CircuitSetup avatar May 31 '20 20:05 CircuitSetup