Adafruit_Wippersnapper_Arduino
Adafruit_Wippersnapper_Arduino copied to clipboard
adafruit
PyPortal Titano - Wippersnapper gets disconnected from WiFi and never reconnects
Reference this forum thread: https://forums.adafruit.com/viewtopic.php?t=210269
In short, I have a PyPortal Titano that monitors the state of charge of a LiPo battery via an I2C interface. It reports the battery status every one minute. It is installed in an area that has weak/intermittent WiFi service.
At random intervals (ranging from hourly to weekly), the device's Adafruit IO status goes to Offline.
If I monitor the USB serial port connection, the code is still running and pinging, but I2C publishing is not successful.
The code never recovers from this condition, until I press the Reset button.
Logs are in the forum thread.
Arduino board PyPortal Titano: SW v1.0.0-alpha.82. NINA: 1.7.7
To Reproduce See description above and in the forum thread.
Expected behavior I expect the Wippersnapper software to automatically recover if an I2C or WiFi connection or publishing failure occurs.
Which components are connected to your device The issue occurs with either an Adafruit INA219, or with the Adafruit MAX17048 battery gauge via the I2C interface.
The connection failure is not related to the state of charge of the battery. The Titano is running from USB +5V power. The battery, charger, and a 60 mA load are only an experiment (so the feed has something useful to report).
After verifying that the issue occurs with either of those sensors alone, I recently added an HTU31D temperature sensor.
I also have the two discrete outputs from the Titano connnected to an Adafruit latching relay (for controlling a LiPo battery charger module).
Desktop (please complete the following information):
- OS: Windows 10
- Browser: Firefox
- Version: current release
Follow-ups: The NINA firmware has been updated.
I added some other sensors, they update at different intervals. The "WiFi won't reconnect" issue happens regardless of how many sensors are configured.
Here is one of the serial port logs recorded while the Titano SW is running but isn't reconnecting the WiFi:
PING! PING! PING! PING! Sensor 0x41 Temperature: 22.86 degrees C Sensor 0x41 Humidity: 32.68%RH PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent! PING! PING! PING! PING! PING! STATUS LED BLINK KAT Executing periodic event on A16 Publishing pinEvent...Published! Sensor 0x40 Voltage: 3.66 v Sensor 0x40 Current: 59.20 mA PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent! Sensor 0x36 Voltage: 3.65 v Sensor 0x36 Read: 5.74 % PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent! PING! PING! PING! PING! PING! PING! PING!
Very strange, and the PING messages in theory mean it's sending and receiving a ping/pong message from the MQTT broker (i.e. wifi still active).
@T-Mosher try this firmware (mostly untested so shout if it's broken) - taken from #584 PyPortal_Titano_beta82_PbErrors_585_g2fa39e88.zip
Also could you just confirm whether you have cut the 5V/3V jumper on the PyPortal Titano to select the i2c bus voltage?
I did not cut the trace, because the Titano tutorial page has this note in a big blue box: Unlike the original PyPortal, the I2C connector is level shifted to 3V so its safe to use without changing the power trace
I just loaded that test build, I'll post back if there are results.
Note that my wifi has been behaving well for the last few days, so I haven't had any recent connection issues with the previous build (but did going back a few days).
The "went offline" issue did happen with the test build. Here's what the serial port is doing after the issue (I was not logging when it happened).
Sensor 0x40 Voltage: 4.12 v Sensor 0x40 Current: 69.50 mA PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publ ish I2CDeviceEvent! PING! PING! PING! PING! PING! PING! PING! STATUS LED BLINK KAT Sensor 0x40 Voltage: 4.12 v Sensor 0x40 Current: 69.40 mA PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent! PING! PING! PING! PING! PING! PING! Sensor 0x40 Voltage: 4.12 v Sensor 0x40 Current: 69.40 mA PUBLISHING -> I2C Device Sensor Event Message...ERROR: Failed to encode and publish I2CDeviceEvent! PING! PING!
Thanks @T-Mosher, a new build ready for you... wippersnapper.pyportal_titano_tinyusb.1.0.0-beta.82-26-g9dda2fda.zip
You've helped identify some shortcomings in the logging that I was previously unaware of, win win for everybody!
This build is not expected to resolve the problem, but hopefully identify the issue in more detail.
@T-Mosher made a minor mistake in that one, will be printing i2c failures for each success 🙃
Corrected 📈 wippersnapper.pyportal_titano_tinyusb.1.0.0-beta.82-28-g7c55ad4d.zip
Seems to be working. Just to verify, is this what you expect to see on the serial port? Sensor 0x40 Voltage: 4.14 v Sensor 0x40 Current: 69.70 mA PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED! Sending PING: SUCCESS! Sending PING: SUCCESS! Sending PING: SUCCESS! Sending PING: SUCCESS! Sending PING: SUCCESS! Sending PING: SUCCESS!
Yes that's encouraging, as previously it was saying ping before it actually sent a ping, and not commenting on success or failures. Similarly the publish could fail and it would be ambiguous between two code sections, now it will separately report an mqtt publish failure (sadly not the reason) as opposed to a separate packet encoding error
On Wed, 22 May 2024, 03:24 T-Mosher, @.***> wrote:
Seems to be working. Just to verify, is this what you expect to see on the serial port? Sensor 0x40 Voltage: 4.14 v Sensor 0x40 Current: 69.70 mA PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED! Sending PING: SUCCESS! Sending PING: SUCCESS! Sending PING: SUCCESS! Sending PING: SUCCESS! Sending PING: SUCCESS! Sending PING: SUCCESS!
— Reply to this email directly, view it on GitHub https://github.com/adafruit/Adafruit_Wippersnapper_Arduino/issues/585#issuecomment-2123751799, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTBZ47U45JJANY5TGMB4DLZDP6UZAVCNFSM6AAAAABHN5ELS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTG42TCNZZHE . You are receiving this because you were assigned.Message ID: @.***>
Update on testing: No lockups have occurred in the last few days, so testing continues.
Just a question, when the terminal says PING, is that a real IP ping that goes over the WiFi? Or is it some other internal use of the word 'ping'?
Very good question 😅 It's an mqtt ping, a heartbeat or keep-alive packet if you will. It sends a PINGREQ and expects a PINGRESP from the MQTT broker.
The logging was just saying it was about to send it but not report what happened.
On Fri, 24 May 2024, 18:05 T-Mosher, @.***> wrote:
Just a question, when the terminal says PING, is that a real IP ping that goes over the WiFi? Or is it some other internal use of the word 'ping'?
— Reply to this email directly, view it on GitHub https://github.com/adafruit/Adafruit_Wippersnapper_Arduino/issues/585#issuecomment-2130008383, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTBZ46NMVUIU5GBD3BLYZTZD5XPJAVCNFSM6AAAAABHN5ELS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZQGAYDQMZYGM . You are receiving this because you were assigned.Message ID: @.***>
Okay @T-Mosher, third times a charm, this one should do what we've been hoping for.
It announces the RSSI in the serial monitor, and successfully restarts the PyPortal Titano's onboard ESP32, in theory we may fail to send a datapoint or two but will then reconnect at the next scheduled ping.
Give it a try and let us know how you get on, also thanks again for being a willing test pilot!
wippersnapper.pyportal_titano_tinyusb.1.0.0-beta.83-31-g9c0c151b.zip
OK, I've installed it. Will keep you posted on results.
FYI, I've been running the previous release you posted above (beta.82) since May 21 (9 days ago), and have not seen any lockups. I don't know if any of the changes in that build addressed the issue, or I've just been lucky with a stable WiFi connection lately. I have no way to tell.
Sadly I think just lucky, which probably indicates to us a long test period.
I tested a modified 82/83 and there was no reconnect when I disconnected my WiFi hotspot, whereas the new build definitely reconnected. Fingers crossed 🤞
On Fri, 31 May 2024, 05:19 T-Mosher, @.***> wrote:
OK, I've installed it. Will keep you posted on results.
FYI, I've been running the previous release you posted above (beta.82) since May 21 (9 days ago), and have not seen any lockups. I don't know if any of the changes in that build addressed the issue, or I've just been lucky with a stable WiFi connection lately. I have no way to tell.
— Reply to this email directly, view it on GitHub https://github.com/adafruit/Adafruit_Wippersnapper_Arduino/issues/585#issuecomment-2141205777, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTBZ42IYKOIXIZASPEX4P3ZE723PAVCNFSM6AAAAABHN5ELS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBRGIYDKNZXG4 . You are receiving this because you were assigned.Message ID: @.***>
To speed up the testing process, I'm trying to artificially degrade my WiFi performance (i.e. wrapping things in aluminum foil, etc), but it's not really effective (and I can't just disconnect the router at will because it's shared by other users).
Just had this happen (failed ping and recovered - but the terminal didn't reconnect, so I have to restart the log). Note that it did automatically recover the WiFi connection and continued supplying data to the feeds.
Checking back in that log file, there are a total of 20 ping failures - but only that last one caused the serial port logging to stop.
Sensor 0x40
Voltage: 4.12 v
Sensor 0x40
Current: 64.70 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING: SUCCESS!
WiFi RSSI: -63
Sending PING: SUCCESS!
WiFi RSSI: -62
Sending PING: SUCCESS!
WiFi RSSI: -76
Sending PING: SUCCESS!
WiFi RSSI: -64
Sending PING: SUCCESS!
WiFi RSSI: -63
Sending PING: SUCCESS!
WiFi RSSI: -78
STATUS LED BLINK KAT
Sensor 0x40
Voltage: 4.12 v
Sensor 0x40
Current: 64.40 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING: SUCCESS!
WiFi RSSI: -63
Sending PING: FAILURE! Running network FSM...
Connected to WiFi!
Connecting to AIO MQTT (attempt #0)
WiFi Status: 20
Adafruit.io WipperSnapper
-------Device Information-------
Fir
Adafruit.io User: T_Mo
WiFi Network: farklenet
MAC Address: C8:9D:06:86:E6:94
-------------------------------
Generating device's MQTT topics...
Running Network FSM...
Establishing network connection...
Performing a WiFi scan for SSID...SSID found! RSSI: -68
Connecting to WiFi (attempt #0)
Reset Pin: 7
Resetting ESP32...
ESP32 booted, version: 1.7.7
Connecting to farklenet
Had another one of these last night (a WiFi restart). It was associated with a feed alert that it had not updated for 10 minutes (normally reports every 30 seconds).
Note that when this WiFi restart happens, I also get a "ding" from Windows that the USB drive has re-connected. Then the serial port logging stops.
Sending PING: SUCCESS!
WiFi RSSI: -73
STATUS LED BLINK KAT
Sensor 0x40
Voltage: 4.11 v
Sensor 0x40
Current: 66.00 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING: SUCCESS!
WiFi RSSI: -58
Sending PING: SUCCESS!
WiFi RSSI: -74
Sending PING: SUCCESS!
WiFi RSSI: -59
Sending PING: SUCCESS!
WiFi RSSI: -75
Sending PING: SUCCESS!
WiFi RSSI: -59
Sending PING: FAILURE! Running network FSM...
Connected to WiFi!
Connecting to AIO MQTT (attempt #0)
WiFi Status: 20
Adafruit.io WipperSnapper
-------Device Information-------
Fir
Adafruit.io User: T_Mo
WiFi Network: farklenet
MAC Address: C8:9D:06:86:E6:94
-------------------------------
Generating device's MQTT topics...
Running Network FSM...
Establishing network connection...
Performing a WiFi scan for SSID...SSID found! RSSI: -63
Connecting to WiFi (attempt #0)
Reset Pin: 7
Resetting ESP32...
ESP32 booted, version: 1.7.7
Connecting to farklenet
Another WiFi error and restart.
I"m not sure if you want me to continue reporting these. Please advise.
Sending PING: SUCCESS!
WiFi RSSI: -64
Sensor 0x40
Voltage: 4.09 v
Sensor 0x40
Current: 62.60 mA
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sending PING: SUCCESS!
WiFi RSSI: -66
Sending PING: SUCCESS!
WiFi RSSI: -63
Sensor 0x41
Temperature: 25.34 degrees C
PUBLISHING -> I2C Device Sensor Event Message...ERROR: MQTT Publish failed!
ERROR: Failed to encode and publish I2CDeviceEvent!
Sending PING: SUCCESS!
WiFi RSSI: -63
Sending PING: SUCCESS!
WiFi RSSI: -62
Sending PING: FAILURE! Running network FSM...
Connected to WiFi!
Connecting to AIO MQTT (attempt #0)
WiFi Status: 20
Adafruit.io WipperSnapper
-------Device Information-------
Fir
Adafruit.io User: T_Mo
WiFi Network: farklenet
MAC Address: C8:9D:06:86:E6:94
-------------------------------
Generating device's MQTT topics...
Running Network FSM...
Establishing network connection...
Performing a WiFi scan for SSID...SSID found! RSSI: -63
Connecting to WiFi (attempt #0)
Reset Pin: 7
Resetting ESP32...
ESP32 booted, version: 1.7.7
Connecting to farklenet
Adafruit.io WipperSnapper
-------Device Information-------
Fir
Adafruit.io User: T_Mo
WiFi Network: farklenet
MAC Address: C8:9D:06:86:E6:94
Thanks @T-Mosher that's enough for now, really appreciate the extra feedback. Ideally it probably wouldn't reconnect serial/reboot the main device (only the esp32 co-processor). I'm going to be cleaning up the associated PR branch this week so may have a new version to test, but you can equally just ignore it and carry on now that you have a fix. It will probably make it into the next WipperSnapper release (or one after if not).
Forgot to mention this change has been released last week. Current latest version is v1.0.0.beta-88
Closing this for now. Thanks for the help testing, and feel free to reopen this issue if needed (or create a new one).