dahlb
AwsCrtError: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
Running X86-64. I have been following this project and noticed the newest update supports this version now.
I successfully login but then the integration shows this error:
Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 357, in async_setup result = await component.async_setup_entry(hass, self) File "/config/custom_components/ha_hatch/init.py", line 112, in async_setup_entry await setup_connection("initial setup") File "/config/custom_components/ha_hatch/init.py", line 88, in setup_connection _, mqtt_connection, rest_devices, expiration_time = await get_rest_devices( File "/usr/local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 55, in get_rest_devices mqtt_connection.connect().result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result raise self._exception awscrt.exceptions.AwsCrtError: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
is it possible you are trying to run this integration on two instances of HA? that seems the most likely cause based on upstream issues that match your logs' error https://github.com/aws/aws-iot-device-sdk-cpp-v2/issues/110
No, I only have one computer and it’s being ran natively on its own on that hardware by itself. No other computers involved.
I will try power cycling it fully and see what that does
Sent from Outlook for iOShttps://aka.ms/o0ukef
From: Brendan Dahl @.> Sent: Saturday, September 24, 2022 5:25:02 PM To: dahlb/ha_hatch @.> Cc: escapethewormhole @.>; Author @.> Subject: Re: [dahlb/ha_hatch] 1.7.0 - Error (Issue #10)
is it possible you are trying to run this integration on two instances of HA? that seems the most likely cause based on upstream issues that match your logs' error aws/aws-iot-device-sdk-cpp-v2#110https://github.com/aws/aws-iot-device-sdk-cpp-v2/issues/110
— Reply to this email directly, view it on GitHubhttps://github.com/dahlb/ha_hatch/issues/10#issuecomment-1257081827, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZRSSIFUMXZ2U6DZII7OYTTV76EU5ANCNFSM6AAAAAAQUYDXHI. You are receiving this because you authored the thread.Message ID: @.***>
I'm having the same issue, running HA in a Docker container.
I can see the package is definitely installed in the container, I can load it, and it's the latest version. This suggests it might be an issue with Hatch's certificates?
sorry but it seems unlikely to be caused by hatch's certificates as not everybody is experiencing this issue, maybe your router's firewall is blocking websockets?
https://github.com/dahlb/hatch_rest_api/blob/1745d9c97e477ae540f7ec3e21eaedde2c49dc8f/src/hatch_rest_api/util_bootstrap.py#L36-L40
Not sure if this is the cause, but if you're interested in eliminating periodical reconnecting to the MQTT endpoint in order to have the credentials unexpired, you can provide a callable delegation, which does it automatically, to the credential provider instead of feeding static credentials.
https://awslabs.github.io/aws-crt-python/api/auth.html#awscrt.auth.AwsCredentialsProvider.new_delegate
thanks for the improvement, I'll track it in https://github.com/dahlb/ha_hatch/issues/12 but I don't think it's the cause of this error since the described experience doesn't seem to have worked even for a little while
Just tried 1.8.0 with the same result.
One question, how does it pick what device? I only have the one but in the app when I login I then have to tell it what hatch I want to control. This integration never does that. Could that be where it is hanging up?
I expect from your error that it is caused by your internet connection. But this integration doesn't pick it just controls all the devices on your account
Any idea what one would look at for my internet connection should that be the culprit?
I have gigabit fiber, through a Unifi UDM-Pro. I don't have any of my ISP hardware in between either.
I don't know much about firewall settings. Though I also use a fiber to UDM pro router, but it doesn't cause me this error. I just don't know another reason your aws connection would be getting unexpectedly discounted.
@escapethewormhole, were you able to resolve this? I'm having the same issue and @dahlb marked my issue #27 as a duplicate of this one.
@nderrico1223 - you're the fourth person in this thread who has indicated they're using a Unifi router (I've got a Dream Machine). I feel like that's a pattern here.
@dahlb, since you have a UDM Pro and it's working for you - do you have Threat Management enabled by chance? I do, and I suspect that's what's blocking AWS here, but even when I disable it I don't see any change in the error message. Maybe I'll try that again now that there's more logging.
I just installed this integration (v1.9.7) via HACS in my Raspberry Pi Home Assistant installation. I also got this error immediately after setting up my credentials in the integration.
I pasted the error messages I saw below.
My Raspberry Pi is only running Home Assistant and is directly wired to my Ubiquiti EdgeRouter X using Verizon FiOS.
First error
This error originated from a custom integration.Logger: homeassistant.config_entries Source: custom_components/ha_hatch/init.py:111 Integration: Hatch Rest Mini/Plus (documentation, issues) First occurred: 3:21:47 PM (2 occurrences) Last logged: 3:46:51 PM
Error setting up entry (REDACTED)@gmail.com for ha_hatch Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 372, in async_setup result = await component.async_setup_entry(hass, self) File "/config/custom_components/ha_hatch/init.py", line 135, in async_setup_entry await setup_connection("initial setup") File "/config/custom_components/ha_hatch/init.py", line 111, in setup_connection _, mqtt_connection, rest_devices, expiration_time = await get_rest_devices( File "/usr/local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 64, in get_rest_devices raise e File "/usr/local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 60, in get_rest_devices mqtt_connection.connect().result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception awscrt.exceptions.AwsCrtError: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
Second error
This error originated from a custom integration.Logger: hatch_rest_api.util_bootstrap Source: custom_components/ha_hatch/init.py:111 Integration: Hatch Rest Mini/Plus (documentation, issues) First occurred: 3:21:47 PM (2 occurrences) Last logged: 3:46:51 PM
MQTT connection failed with exception AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
I believe I may be in a similar boat... I also have a UDM-Pro, threat management is turned on. I initially had some sort of permissions issue but have since set up HA to run as root (temporarily) which solved the permissions issue. Now this is the only remaining issue.
Error 1:
Logger: homeassistant.config_entries Source: .local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py:60 First occurred: 8:23:05 PM (1 occurrences) Last logged: 8:23:05 PM
Error setting up entry [email protected] for ha_hatch Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/homeassistant/config_entries.py", line 382, in async_setup result = await component.async_setup_entry(hass, self) File "/config/custom_components/ha_hatch/init.py", line 148, in async_setup_entry await setup_connection("initial setup") File "/config/custom_components/ha_hatch/init.py", line 120, in setup_connection _, mqtt_connection, rest_devices, expiration_time = await get_rest_devices( File "/config/.local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 64, in get_rest_devices raise e File "/config/.local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 60, in get_rest_devices mqtt_connection.connect().result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception awscrt.exceptions.AwsCrtError: AWS_ERROR_HTTP_WEBSOCKET_UPGRADE_FAILURE: Failed to upgrade HTTP connection to Websocket.
Error 2: Logger: hatch_rest_api.util_bootstrap Source: .local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py:63 First occurred: 8:23:05 PM (1 occurrences) Last logged: 8:23:05 PM
MQTT connection failed with exception AWS_ERROR_HTTP_WEBSOCKET_UPGRADE_FAILURE: Failed to upgrade HTTP connection to Websocket.
Well I'll join the list. USG-3P, threat management off. Same issue.
awscrt.exceptions.AwsCrtError: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
2023-01-18 16:40:18.857 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140201106029872] 2023-01-18 18:23:02.423 WARNING (SyncWorker_5) [urllib3.connectionpool] Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f830268d180>, 'Connection to
Me too. Had some internet issues this morning and now when I restart HASS i'm getting this error. I also can't seem to connect to the Hatch via the app now, so I'm not sure this is related to the integration? Has anybody found a workaround? Whether it's waiting/time, resetting the hatch, or anything else?
@FuzzyMistborn I also experienced this for the first time this morning after I unplugged/forced ha to restart from an apparent lockup last night. I am no longer seeing the error, are you still getting it or does it work for you now too?
Thanks! Just reloaded and it's working now. Maybe some kind of outage on Hatch's end then?
my guess is this is related to https://github.com/dahlb/hatch_rest_api/issues/4 and potentially a missing throttling when network connectivity is compromised or if multiple instances of this integration are configured to use the same hatch account.
hatch likely improved their response after filing the old bug above in cases where throttling causes a connection flood they are at least temporary banning ips to prevent paying the bill on connection storms.
unfortunately I lost all my logs due to my haos restarting, but I will try to experiment with connection retry attempts in a few weeks when I might have time.
That would make sense. I have an automation reload the integration at 6:30am because I ran into a few issues where the integration seemed to lose it's connection/stop working. I was debugging something this morning with an automation involving the sound machine's availability and reloaded the integration 1 more time and then also ended up rebooting HASS for an unrelated reason. Probably hit a limit so it's like a fail2ban type block.
I would like to mention my previous reply, which might not be the direct fix but provides a IMO pretty reliable reconnecting mechanism, and the MQTT client id is possibly the cause, where I use a random uuid for each new connection instead of a user-dependent pattern.
FYI, our integration, in which the connected server is also built on AWS iot infra, has stably worked for several months without any connection issue like this (MQTT hung up) on my side. I've been using a Unifi router for two years.
Hope this info helps.
I found some new documentation that helped me fix a few of the causes for this error, v1.12.4 might fix everybody in this threads issues.
one of the problems that likely affected a number of these users was that hatch uses multiple aws regions and until now the api this project uses was locked to us-west2 and if you are assigned a different region you would've gotten the HANGUP error
another problem was caused by resetting connections with overlapping ids so I've appended a uuid to the connection ids to allow both api ops easy identification while allowing multiple connections per user when needed.
another possible error that I am not sure anybody was experiencing but also could've caused this error was special characters in the username/email so those have been stripped from the client id if they exist.
the really annoying thing I've found in today's research is a lot of, if not all aws iot errors just through this catch all instead of providing better detail ...
That would make some sense. I am not in western US, I am western Canada and I believe Amazon has a data center near me that I would be assigned to most likely
Sent from Outlook for iOShttps://aka.ms/o0ukef
From: Brendan Dahl @.> Sent: Wednesday, January 25, 2023 6:44:28 PM To: dahlb/ha_hatch @.> Cc: escapethewormhole @.>; Mention @.> Subject: Re: [dahlb/ha_hatch] AwsCrtError: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly. (Issue #10)
I found some new documentation that helped me fix a few of the causes for this error, v1.12.4 might fix everybody in this threads issues.
one of the problems that likely affected a number of these users was that hatch uses multiple aws regions and until now the api this project uses was locked to us-west2 and if you are assigned a different region you would've gotten the HANGUP error
another problem was caused by resetting connections with overlapping ids so I've appended a uuid to the connection ids to allow both api ops easy identification while allowing multiple connections per user when needed.
another possible error that I am not sure anybody was experiencing but also could've caused this error was special characters in the username/email so those have been stripped from the client id if they exist.
the really annoying thing I've found in today's research is a lot of, if not all aws iot errors just through this catch all instead of providing better detail ...
— Reply to this email directly, view it on GitHubhttps://github.com/dahlb/ha_hatch/issues/10#issuecomment-1404459710, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZRSSICL2XVBFXCCDONEV6LWUHJHZANCNFSM6AAAAAAQUYDXHI. You are receiving this because you were mentioned.Message ID: @.***>
Unfortunately still no joy for me on 1.12.4 - I'm also in Canada.
2023-01-25 22:07:29.238 ERROR (MainThread) [hatch_rest_api.util_bootstrap] MQTT connection failed with exception AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly. 2023-01-25 21:50:13.588 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry [email protected] for ha_hatch Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 382, in async_setup result = await component.async_setup_entry(hass, self) File "/config/custom_components/ha_hatch/init.py", line 148, in async_setup_entry await setup_connection("initial setup") File "/config/custom_components/ha_hatch/init.py", line 120, in setup_connection _, mqtt_connection, rest_devices, expiration_time = await get_rest_devices( File "/usr/local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 67, in get_rest_devices raise e File "/usr/local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 63, in get_rest_devices mqtt_connection.connect().result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception
I found some new documentation that helped me fix a few of the causes for this error, v1.12.4 might fix everybody in this threads issues.
one of the problems that likely affected a number of these users was that hatch uses multiple aws regions and until now the api this project uses was locked to us-west2 and if you are assigned a different region you would've gotten the HANGUP error
another problem was caused by resetting connections with overlapping ids so I've appended a uuid to the connection ids to allow both api ops easy identification while allowing multiple connections per user when needed.
another possible error that I am not sure anybody was experiencing but also could've caused this error was special characters in the username/email so those have been stripped from the client id if they exist.
the really annoying thing I've found in today's research is a lot of, if not all aws iot errors just through this catch all instead of providing better detail ...
Mine is not working still, same errors.
Likewise. Just updated to v1.12.4 and got the same error I got in https://github.com/dahlb/ha_hatch/issues/10#issuecomment-1367604323. Full debug dump:
Debug logging
2023-01-25 21:50:19.833 DEBUG (MainThread) [custom_components.ha_hatch] async setup entry: {'email': '***@gmail.com', 'password': '***'}:{}
2023-01-25 21:50:20.702 DEBUG (MainThread) [custom_components.ha_hatch] updating credentials: initial setup
2023-01-25 21:50:20.713 DEBUG (MainThread) [hatch_rest_api.util_http] sending https://data.hatchbaby.com/public/v1/login requestsending https://data.hatchbaby.com/public/v1/login request with {'email': '***@gmail.com', 'password': '***'}
2023-01-25 21:50:21.150 DEBUG (MainThread) [hatch_rest_api.util_http] response headers:
2023-01-25 21:50:21.152 DEBUG (MainThread) [hatch_rest_api.util_http] response json: {'status': 'success', 'message': 'Successful login', 'payload': {'id': 420267, 'createDate': '2019-06-10 16:13:03', 'updateDate': '2019-12-25 17:16:20', 'active': True, 'email': '***@gmail.com', [REDACTED] 'appType': 'Android', 'firstName': '***', 'lastName': None, 'signupSource': 'app', 'timezone': 'America/New_York', 'timeZoneAdjust': -5, 'stage': 49}, 'sync': 1674701416018, 'token': 'AAA'}
2023-01-25 21:50:21.152 DEBUG (MainThread) [hatch_rest_api.util_http] sending https://data.hatchbaby.com/service/app/iotDevice/v2/fetch request
2023-01-25 21:50:21.246 DEBUG (MainThread) [hatch_rest_api.util_http] response headers:
2023-01-25 21:50:21.246 DEBUG (MainThread) [hatch_rest_api.util_http] response json: {'status': 'success', 'message': 'Fetched list of IoT Devices', 'payload': [], 'sync': 1674701416189}
2023-01-25 21:50:21.248 DEBUG (MainThread) [hatch_rest_api.util_http] sending https://data.hatchbaby.com/service/app/restPlus/token/v1/fetch request
2023-01-25 21:50:21.351 DEBUG (MainThread) [hatch_rest_api.util_http] response headers:
2023-01-25 21:50:21.352 DEBUG (MainThread) [hatch_rest_api.util_http] response json: {'status': 'success', 'message': 'Got Cognito OpenId token', 'payload': {'endpoint': 'https://aacsjje0gbbcc-ats.iot.us-west-2.amazonaws.com', 'identityId': 'us-west-2:06ae5904-7636-4564-9475-5645dfd8f59b', 'region': 'us-west-2', 'cognitoPoolId': 'us-west-2:5651fc22-f313-49d5-b379-fdab2c535d9c', 'token': 'BBB'}, 'sync': 1674701416282}
2023-01-25 21:50:21.352 DEBUG (MainThread) [hatch_rest_api.util_http] sending https://cognito-identity.us-west-2.amazonaws.com requestheaders: {'content-type': 'application/x-amz-json-1.1', 'X-Amz-Target': 'AWSCognitoIdentityService.GetCredentialsForIdentity'}sending https://cognito-identity.us-west-2.amazonaws.com request with {'IdentityId': 'us-west-2:06ae5904-7636-4564-9475-5645dfd8f59b', 'Logins': {'cognito-identity.amazonaws.com': 'BBB'}}
2023-01-25 21:50:21.745 DEBUG (MainThread) [hatch_rest_api.util_http] response headers:
2023-01-25 21:50:21.760 DEBUG (MainThread) [hatch_rest_api.util_http] response raw: {"Credentials":{"AccessKeyId":"CCC","Expiration":1.674705021E9,"SecretKey":"DDD","SessionToken":"EEE"},"IdentityId":"us-west-2:06ae5904-7636-4564-9475-5645dfd8f59b"}
2023-01-25 21:50:22.395 ERROR (MainThread) [hatch_rest_api.util_bootstrap] MQTT connection failed with exception AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
2023-01-25 21:50:22.434 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry ***@gmail.com for ha_hatch
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 382, in async_setup
result = await component.async_setup_entry(hass, self)
File "/config/custom_components/ha_hatch/__init__.py", line 148, in async_setup_entry
await setup_connection("initial setup")
File "/config/custom_components/ha_hatch/__init__.py", line 120, in setup_connection
_, mqtt_connection, rest_devices, expiration_time = await get_rest_devices(
File "/usr/local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 67, in get_rest_devices
raise e
File "/usr/local/lib/python3.10/site-packages/hatch_rest_api/util_bootstrap.py", line 63, in get_rest_devices
mqtt_connection.connect().result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
awscrt.exceptions.AwsCrtError: AWS_ERROR_MQTT_UNEXPECTED_HANGUP: The connection was closed unexpectedly.
I'm hitting this error also. My wife has the app and had to reset her password to figure out what it was, and when she did the device was gone from the app, so I feel like that could be a root cause/related. I feel like previously it might've only been on Bluetooth and not WiFi but I haven't looked into this yet.
My networking gear is Mikrotik, so not Ubiquiti, but perhaps similarly configured.
Looks like my problem was resolved.
I was troubleshooting an issue with the Tuya integration and realized the issue with it was that my time was incorrect as it was 7 minutes slow. I messed around trying to correct it in HA, but later realized that somehow the whole unRAID server was off by 7 minutes.
Long story short, manually setting my time to the correct time fixed the Tuya issue and seems to have resolved my Hatch integration issue as well!