docker-wyze-bridge icon indicating copy to clipboard operation
docker-wyze-bridge copied to clipboard

Timed out connecting - ENR changing/expiring

Open krystiancharubin opened this issue 2 years ago • 16 comments

Keep getting a timed out connecting every few days. Need to delete cameras.pickle and restart for the bridge to connect. Camera still has the same ip address.

2022/08/16 22:44:14 [WyzeBridge] Starting rtsp-simple-server v0.19.3
2022/08/16 22:44:14 [WyzeBridge] [ON-DEMAND] ⌛️ WyzeCam Doorbell - front-door on 10.1.0.124 (1/3)
2022/08/16 22:44:16 [WyzeBridge] 127.0.0.1 - - [16/Aug/2022 22:44:16] "GET /events/start/front-door HTTP/0.9" 200 -
2022/08/16 22:44:36 [Front Door] IOTC_ER_TIMEOUT
2022/08/16 22:44:38 [WyzeBridge] ⏰ Timed out connecting to front-door (20s).
2022/08/16 22:44:46 [WyzeBridge] 127.0.0.1 - - [16/Aug/2022 22:44:46] "GET /events/start/front-door HTTP/0.9" 404 -

After deleting cameras.pickle

2022/08/16 22:49:02 [WyzeBridge] 🚀 STARTING DOCKER-WYZE-BRIDGE v1.8.3
 * Serving Flask app 'frontend'
2022/08/16 22:49:02 [WyzeBridge] 📚 Using 'user' from local cache...
 * Debug mode: off
2022/08/16 22:49:02 [WyzeBridge] 🔍 Could not find local cache for 'cameras'
2022/08/16 22:49:02 [WyzeBridge] WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:5000
 * Running on http://172.23.0.5:5000
2022/08/16 22:49:02 [WyzeBridge] Press CTRL+C to quit
2022/08/16 22:49:02 [WyzeBridge] 📚 Using 'auth' from local cache...
2022/08/16 22:49:02 [WyzeBridge] ☁️ Fetching 'cameras' from the Wyze API...
2022/08/16 22:49:03 [WyzeBridge] ⚠️ Error getting cameras - Expired token?
2022/08/16 22:49:03 [WyzeBridge] ♻️ Refreshing tokens
2022/08/16 22:49:03 [WyzeBridge] 💾 Saving 'auth' to local cache...
2022/08/16 22:49:03 [WyzeBridge] ☁️ Fetching 'cameras' from the Wyze API...
2022/08/16 22:49:04 [WyzeBridge] 💾 Saving 'cameras' to local cache...
2022/08/16 22:49:04 [WyzeBridge] 
🎬 STARTING ALL 1 CAMERAS
2022/08/16 22:49:04 [WyzeBridge] Starting rtsp-simple-server v0.19.3
2022/08/16 22:49:04 [WyzeBridge] [ON-DEMAND] ⌛️ WyzeCam Doorbell - front-door on 10.1.0.124 (1/3)
2022/08/16 22:49:06 [WyzeBridge] 127.0.0.1 - - [16/Aug/2022 22:49:06] "GET /events/start/front-door HTTP/0.9" 200 -
2022/08/16 22:49:09 [Front Door] 📡 Getting 180kb/s HD stream (20fps) via LAN mode (WiFi: 60%) FW: 4.25.1.060 🔒 (DTLS) (2/3)
2022/08/16 22:49:09 [Front Door] 🔊 Audio Enabled - PCM > AAC/8,000Hz
2022/08/16 22:49:09 [Front Door] WARNING: Skipping smaller frame at start of stream (frame_size=4)
2022/08/16 22:49:12 [RTSP][FRONT-DOOR] ✅ '/front-door' stream is UP! (3/3)
2022/08/16 22:49:17 [RTSP][FRONT-DOOR] 📖 New client reading

krystiancharubin avatar Aug 17 '22 02:08 krystiancharubin

This looks similar to #508. Not sure what is causing it though.

Could someone run python -m pickle cameras.pickle before/after clearing it to see if there are any major changes (other than the thumbnail update)?

mrlt8 avatar Aug 17 '22 03:08 mrlt8

I'll do the comparison next time it stops connecting.

krystiancharubin avatar Aug 17 '22 03:08 krystiancharubin

The only attribute that is different in the new cameras.pickle is the enr attribute

krystiancharubin avatar Aug 18 '22 00:08 krystiancharubin

Thank you @krystiancharubin! The ENR is used to generate the authkey to connect to the camera and is also used as the password to start the avclient.

Not sure why wyze is changing the ENR more often for some users since it used to remain pretty static for months.

It should also be giving the "ENR_AUTH_FAILED" message instead of "IOTC_ER_TIMEOUT", so I should probably add some kind of check to see if it's actually a real timeout or an auth failed before fetching the new ENR from the API.

mrlt8 avatar Aug 18 '22 02:08 mrlt8

I downgraded to 1.7.5 when I first started encountering this issue with 1.8.0 and the cameras were working fine. Is there anything in 1.8.0 that changed how auth works?

krystiancharubin avatar Aug 18 '22 02:08 krystiancharubin

The only thing I can think of is v1.8.0 is defaulting to the US tutk server to better match the official wyze app:

https://github.com/mrlt8/docker-wyze-bridge/blob/5c31becaef1d407cd1060ee0ba3171b9553518b4/app/wyzecam/iotc.py#L94

Still shouldn't trigger wyze to push a new authkey/ENR to the camera though...

mrlt8 avatar Aug 18 '22 02:08 mrlt8

FWIW this is happening to me as well. Let me know if I can provide any information that would help chase this down.

neilenns avatar Aug 18 '22 20:08 neilenns

v1.8.4 should pull new camera data and ENR from the wyze api when it encounters the timeout, but more datapoints will be useful to determine how often or what triggers wyze to push a new authkey/ENR.

mrlt8 avatar Aug 19 '22 02:08 mrlt8

Yep, I rolled to v1.8.4 yesterday. I'll let you know if the problem returns!

neilenns avatar Aug 19 '22 14:08 neilenns

With 1.8.4 the camera comes back, although there are long periods of dropouts. I think it hitting that 15 minute cool-off period between refreshes.

krystiancharubin avatar Aug 20 '22 14:08 krystiancharubin

I can try reducing that. Unfortunately, legitimate timeouts can also trigger the same error so I didn't want to trigger the cloud flare ratelimiter: https://github.com/mrlt8/docker-wyze-bridge/issues/453#issuecomment-1169453426

I think we should be more than safe with something like a 120 second cool down.

Edit: any chance you caught the time when the ENR change happened?

mrlt8 avatar Aug 20 '22 14:08 mrlt8

Seems to have happened twice today at 8:30 and then 9:30 also yesterday at 13:00 and 15:00.

krystiancharubin avatar Aug 20 '22 15:08 krystiancharubin

Looks like I had it happen once at 3:13am this morning.

neilenns avatar Aug 20 '22 17:08 neilenns

Seems like wyze is refreshing enr every hour now. Camera usually comes back up quickly but there are still times where it drops out for a few min after a refresh.

krystiancharubin avatar Aug 27 '22 18:08 krystiancharubin

I wonder if @cheme75 is seeing this on some of the different versions of the doorbell?

mrlt8 avatar Aug 29 '22 15:08 mrlt8

Honestly, I’m not watching very often, I take a peek every so often. It’s a nice convenience to have the web ui. I see little difference when I view the orig db or the old beta db. In general, I don’t get any connection timeouts, the beta db connects a bit faster, but it varies, I see roughly 5-20 sec to display in home app. The web ui usually pops fairly quickly - most open video immediately. If you want me do any specific comparisons, let me know.

cheme75 avatar Aug 29 '22 17:08 cheme75

update: I seem to be getting the IOTC_ER_TIMEOUT just recently - with resultant max retries failure connecting. I was away a few weeks and lost connection a while back but figured it was just server went down due to power blip since it is not on a ups. Had a few glitches restarting the docker pc due to a few updates and reboots, etc - so only got it fairly stable for a few days now and this connection max retries has become fairly regular. Sorry I'm not much help on troubleshooting.

cheme75 avatar Nov 12 '22 01:11 cheme75

Is the camera showing as offline in the app? Can you power cycle the camera to see if that helps with the timeout?

mrlt8 avatar Nov 12 '22 02:11 mrlt8

Is the camera showing as offline in the app? Can you power cycle the camera to see if that helps with the timeout?

Cam works fine in the app, although just for live stream, none of the settings can be changed and it doesn’t record events but occasionally detects motion. Took a break overnight. Just did a power cycle. Under #605, I found docker had some glitch restarting so reinstalled and will reboot - pc is old takes a while. Hoping after we relocate out of SoFlo, I can build a better setup, maybe get a rpi.

cheme75 avatar Nov 12 '22 04:11 cheme75