SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch

[Phoenix-DH]

Good morning together,

since yesterday evening I receive the following errors in my HA installation and the aarlo plugin is not able to start:

Logger: pyaarlo Source: custom_components/aarlo/pyaarlo/init.py:712 Integration: aarlo (documentation, issues) First occurred: 12:31:53 AM (150255 occurrences) Last logged: 6:56:33 AM

•	general-error=TimeoutError Traceback (most recent call last): File "/config/custom_components/aarlo/pyaarlo/backend.py", line 485, in _mqtt_main self._event_client.connect(MQTT_HOST, port=443, keepalive=60) File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 914, in connect return self.reconnect() File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1044, in reconnect sock = self._create_socket_connection() File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3685, in _create_socket_connection return socket.create_connection(addr, timeout=self._connect_timeout, source_address=source) File "/usr/local/lib/python3.10/socket.py", line 845, in create_connection raise err File "/usr/local/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) TimeoutError: timed out
•	general-error=SSLCertVerificationError Traceback (most recent call last): File "/config/custom_components/aarlo/pyaarlo/backend.py", line 485, in _mqtt_main self._event_client.connect(MQTT_HOST, port=443, keepalive=60) File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 914, in connect return self.reconnect() File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1073, in reconnect sock.do_handshake() File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mqtt-cluster.arloxcld.com'. (_ssl.c:1007)
•	general-error=SSLEOFError Traceback (most recent call last): File "/config/custom_components/aarlo/pyaarlo/backend.py", line 485, in _mqtt_main self._event_client.connect(MQTT_HOST, port=443, keepalive=60) File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 914, in connect return self.reconnect() File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1073, in reconnect sock.do_handshake() File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake self._sslobj.do_handshake() ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

Is there some additional information I can provide?

Thanks!

[Geoff-Hz]

Hi, Same problem for me.

Thanks

[medivb]

Some for me as well:

Deze fout is ontstaan door een aangepaste integratie.

Logger: pyaarlo Source: custom_components/aarlo/pyaarlo/init.py:712 Integration: Arlo Camera Support (documentation, issues) First occurred: 09:45:09 (7740 occurrences) Last logged: 10:56:16

general-error=SSLCertVerificationError Traceback (most recent call last): File "/config/custom_components/aarlo/pyaarlo/backend.py", line 485, in _mqtt_main self._event_client.connect(MQTT_HOST, port=443, keepalive=60) File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 914, in connect return self.reconnect() File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1073, in reconnect sock.do_handshake() File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mqtt-cluster.arloxcld.com'. (_ssl.c:1007)

[vinnyspb]

A workaround that seems to fix it for me is described here.

Apparently the issue is caused by Arlo MQTT backend changing certificate CN:

$ openssl s_client -showcerts -connect mqtt-cluster.arloxcld.com:443 | openssl x509 -noout -dates
....
depth=0 C = US, ST = California, L = Milpitas, O = "Arlo Technologies, Inc.", CN = mqtt-cluster-z1.arloxcld.com
....
notBefore=Oct 12 01:13:50 2022 GMT
notAfter=Oct 12 01:13:50 2023 GMT

As you can see CN now has -z1 suffix. However I tried modifying MQTT_HOST in pyaarlo, then I get past certificate verification but instead I get Websocket handshake error which is probably caused by something on the backend side.

[Phoenix-DH]

Will try the solution provided, thanks for the hint.

[Phoenix-DH]

Approved and working by adding the following to the config:

aarlo:
  #..your current config..
  #..use either sse or mqtt for this
  backend: sse

[medivb]

Thanks! That solved it for me as well.

[Geoff-Hz]

Thanks, Ok for me too

[twrecked]

@vinnyspb thanks for looking into this, and thanks for trying to change the MQTT_HOST. Maybe I'll have to make that configurable or see if I can pick it up from the information I get back from Arlo.

Did this just start happening or was it after a Home Assistant upgrade? (My money is on Arlo changing something...)

[Phoenix-DH]

For me it happened after a restart of Home Assistant as it seems, because my instance crashed.

[vinnyspb]

Mine did not crash, I restarted for other reasons, but it started happening immediately after restart as well. Guessing that's because until you restart, the connection was still established and working. So no certificate validation was performed until then.

[ismarslomic]

Same issue here! Started to happen after I restarted HA earlier today. The CPU run to 100% on my host machine. Resolved by adding backend: sse to the config.

[rpitera]

I am still at 2023.4.6 and it happened to me after a restart. The solution given fixed it for me. Thanks @vinnyspb for the fix!

[JigSawFr]

Thanks for the fix too !

[dannyb2100]

Approved and working by adding the following to the config:

aarlo:
  #..your current config..
  #..use either sse or mqtt for this
  backend: sse

Worked for me. Thanks for the fix.

How did this become an issue? What's changed?

[ismarslomic]

Not sure what changed, but I assume Arlo is doing some technical changes to their services. I recently received e-mail from them where they informed that they are working on new mobile app. Not sure if that caused this issue though.

[twrecked]

I think it's safe to say they've changed their back end, these are the most obvious changes:

• They've changed the header requirement for the CloudFlare login piece. I couldn't even connect to have the MQTT fail...
• They appear to have broken the MQTT certificates or we're now heading to the wrong host. (Thanks @vinnyspb for finding that.)
• They've changed MQTT transport from websockets to tcp.

I'll play around some more but the following diff should work around the issue. If somebody could try it that would be great, and I'll think about pushing an update later.

• Changes the headers.
• Relaxes the certificate name checking.
• Changes the transport to tcp.

Obviously, relaxing the name check isn't a proper fix. But we need to either:

• Wait for Arlo to fix it.
• Find out the new host name we should be using.
• Make both this configurable.

diff --git a/custom_components/aarlo/pyaarlo/backend.py b/custom_components/aarlo/pyaarlo/backend.py
--- a/custom_components/aarlo/pyaarlo/backend.py	(revision 00dcba913874258984cce167332572c8f922a379)
+++ b/custom_components/aarlo/pyaarlo/backend.py	(date 1683675617572)
@@ -472,12 +472,14 @@
 
             # Create and set up the MQTT client.
             self._event_client = mqtt.Client(
-                client_id=self._event_client_id, transport="websockets"
+                client_id=self._event_client_id, transport="tcp"
             )
             self._event_client.on_log = self._mqtt_on_log
             self._event_client.on_connect = self._mqtt_on_connect
             self._event_client.on_message = self._mqtt_on_message
-            self._event_client.tls_set_context(ssl.create_default_context())
+            ssl_context = ssl.create_default_context()
+            ssl_context.check_hostname = False
+            self._event_client.tls_set_context(ssl_context)
             self._event_client.username_pw_set(f"{self._user_id}", self._token)
             self._event_client.ws_set_options(path=MQTT_PATH, headers=headers)
 
@@ -639,7 +641,7 @@
             "Source": "arloCamWeb",
             "User-Agent": self._user_agent,
             "x-user-device-id": self._user_id,
-            "x-user-device-name": "QlJPV1NFUg==",
+            "x-user-device-automation-name": "QlJPV1NFUg==",
             "x-user-device-type": "BROWSER",
         }
 
@@ -777,7 +779,7 @@
             "User-Agent": self._user_agent,
             "Source": "arloCamWeb",
             "x-user-device-id": self._user_id,
-            "x-user-device-name": "QlJPV1NFUg==",
+            "x-user-device-automation-name": "QlJPV1NFUg==",
             "x-user-device-type": "BROWSER",
         }
 

[townsmcp]

Approved and working by adding the following to the config:

aarlo:
  #..your current config..
  #..use either sse or mqtt for this
  backend: sse

This has worked for me to. Thanks for the info

[rmettier]

Had the same issue, same solution as mentioned multiple times in this thread worked for me. Just wanted to say thanks, this little issue and the speed at which a fix was found and communicated is an example of why this project is so awesome. You guys rock.

[rgon10]

I think it's safe to say they've changed their back end, these are the most obvious changes:

* They've changed the header requirement for the CloudFlare login piece. I couldn't even connect to have the _MQTT_ fail...

* They appear to have broken the _MQTT_ certificates or we're now heading to the wrong host. (Thanks @vinnyspb for finding that.)

* They've changed _MQTT_ transport from `websockets` to `tcp`.

I'll play around some more but the following diff should work around the issue. If somebody could try it that would be great, and I'll think about pushing an update later.

* Changes the headers.

* Relaxes the certificate name checking.

* Changes the transport to `tcp`.

Obviously, relaxing the name check isn't a proper fix. But we need to either:

* Wait for _Arlo_ to fix it.

* Find out the new host name we should be using.

* Make both this configurable.

diff --git a/custom_components/aarlo/pyaarlo/backend.py b/custom_components/aarlo/pyaarlo/backend.py
--- a/custom_components/aarlo/pyaarlo/backend.py	(revision 00dcba913874258984cce167332572c8f922a379)
+++ b/custom_components/aarlo/pyaarlo/backend.py	(date 1683675617572)
@@ -472,12 +472,14 @@
 
             # Create and set up the MQTT client.
             self._event_client = mqtt.Client(
-                client_id=self._event_client_id, transport="websockets"
+                client_id=self._event_client_id, transport="tcp"
             )
             self._event_client.on_log = self._mqtt_on_log
             self._event_client.on_connect = self._mqtt_on_connect
             self._event_client.on_message = self._mqtt_on_message
-            self._event_client.tls_set_context(ssl.create_default_context())
+            ssl_context = ssl.create_default_context()
+            ssl_context.check_hostname = False
+            self._event_client.tls_set_context(ssl_context)
             self._event_client.username_pw_set(f"{self._user_id}", self._token)
             self._event_client.ws_set_options(path=MQTT_PATH, headers=headers)
 
@@ -639,7 +641,7 @@
             "Source": "arloCamWeb",
             "User-Agent": self._user_agent,
             "x-user-device-id": self._user_id,
-            "x-user-device-name": "QlJPV1NFUg==",
+            "x-user-device-automation-name": "QlJPV1NFUg==",
             "x-user-device-type": "BROWSER",
         }
 
@@ -777,7 +779,7 @@
             "User-Agent": self._user_agent,
             "Source": "arloCamWeb",
             "x-user-device-id": self._user_id,
-            "x-user-device-name": "QlJPV1NFUg==",
+            "x-user-device-automation-name": "QlJPV1NFUg==",
             "x-user-device-type": "BROWSER",
         }
 

I experienced the issue after a restart of HA. I applied this patch to the custom components directly and seems to have resolved the issue. From some limited testing, it seems the integration works as expected with this change.

The certificate I get has the common name of mqtt-cluster-z1.arloxcld.com. There's also a SAN for mqtt-cluster-z2.arloxcld.com. It looks like both of these names are valid in DNS and work, but I'm not sure how their high availability is setup and what the implications of using one of those clusters directly is.

From a DNS propagation checker, this seems to be the result globally. The authoritative DNS server for arlo seems to be responding with a new AWS CNAME (mqtt-cluster-z1-new-ext-944ca53bc9be2a3c.elb.eu-west-1.amazonaws.com), I'm not 100% sure what it was previously though.

I'm guessing disabling the cert checking for now will be fine and that Arlo will eventually re-issue the certificates with the main domain listed as a SAN.

Let me know if you need additional assistance with testing. As always, your efforts on this project are greatly appreciated!

[dw1562]

I have installed the latest update and all is working fine. I had put in the workaround mentioned above. Should that now be removed or should I just leave it alone (if it isn't broken don't fix it)?

[twrecked]

@dw1562 If it's working you can leave it as-is. But if you find you are missing events you could change the back end to mqtt.

[Geoff-Hz]

Hi, After updating and backend in mqtt this is the error:

Error: general-error=SSLCertVerificationError Traceback (most recent call last): File "/config/custom_components/aarlo/pyaarlo/backend.py", line 490, in _mqtt_main self._event_client.connect(self._arlo.cfg.mqtt_host, port=443, keepalive=60) File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 914, in connect return self.reconnect() File "/usr/local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1073, in reconnect sock.do_handshake() File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mqtt-cluster.arloxcld.com'. (_ssl.c:997)

with backend = sse -> OK (but I miss some events)

[twrecked]

@Geoff-Hz ; interesting, which of the new mqtt options did you try this with?

[ajfisher]

Same issue today once I updated to 2023.5.x this fix worked:

aarlo:
  #..your current config..
  #..use either sse or mqtt for this
  backend: sse

[Serpe-Verde]

it works properly. And the processor is back to running below 10% Thank you