Android app not respecting user-installed CA certificates

[tsndr]

Hey, today I noticed that my BlueBubbles app can't connect to the server anymore.

My Use Case

• Hosted on local Mac mini
• Using local DNS that resolves the public domain to the local IP of my Mac mini (same domain and port interally and externally)
• Using self signed SSL certificate

Things I've tried

• Check if my URL is reachable (internal and external) ✅
• Make sure the correct SSL certificate is used by the server (internal and external) ✅
• Make sure the CA certificate is installed on my phone ✅
• Check if the webapp is working through bluebubbles.app ✅

Observations

• The android app can't connect either from within the network nor externally.
• The phone app just say's that it can't connect to the Websocket.

Is there any way I can debug this further?

[SpaceSaver]

Does the url shown on the Android client match the url shown on the server?

[tsndr]

Yes it does

[SpaceSaver]

Have you tried restarting the server app?

[tsndr]

Multiple times, also checked for updates (which there weren't any) and even rebooted the Mac mini.

[SpaceSaver]

Try disabling "Detect localhost Address"

[tsndr]

Never had it enabled.

• Using local DNS that resolves the public domain to the local IP of my Mac mini (same domain and port interally and externally)

Wasn't necessary, because of my local DNS server, that points my domain to the local address. Everything worked up until recently, and since I didn't make any changes to my configuration, I'd imagine that the recent update to the app could have something to do with it?

[jjoelj]

Make sure the certificate on the server includes the entire certificate chain. Other people have experienced similar issues on 1.13.0. See #2673

[tsndr]

There is no chain, just the CA certificate and the certificate for my domain.

The CA certificate usualy isn't included in the certificate chain since it is installed on all target devices, only intermediate certificates should be (if present).

[jjoelj]

It would be helpful if you could collect logs from the Android app using Settings > Troubleshooting > start logging/end logging.

[tsndr]

Thanks for the hint, was looking for a log but didn't see the option.

Unfortunately the log isn't really helpful:

[BlueBubblesApp][2024-02-27 18:22:05.388529][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:22:05.433431][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:22:06.761570][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:22:06.762308][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:22:06.762956][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:22:06.767638][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:22:07.759693][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:22:07.829685][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:22:08.575974][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:22:08.576718][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:22:08.576964][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:22:08.582472][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:22:09.483964][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:22:09.613110][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:22:10.437295][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:22:10.438031][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:22:10.439126][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:22:10.445130][INFO] Sending method get-server-url to Kotlin

[jjoelj]

Try starting logging, then going to Connection & Server settings and clicking 'fetch latest url'. Then end logging. It should log an error.

[tsndr]

Here we have some errors:

[[BlueBubblesApp][2024-02-27 18:34:27.909874][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:27.958314][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:29.226711][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:29.226945][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:29.227045][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:29.232638][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:34:30.553920][INFO] (REQUEST[GET]) -> PATH: https://imessage.mydomain.com:54321/api/v1/ping
[BlueBubblesApp][2024-02-27 18:34:30.604116][ERROR] (ERROR[null]) -> PATH: https://imessage.mydomain.com:54321/api/v1/ping
[BlueBubblesApp][2024-02-27 18:34:30.604418][ERROR] (ERROR[null]) -> HandshakeException: Handshake error in client (OS Error: 
	CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:393))
[BlueBubblesApp][2024-02-27 18:34:30.604556][ERROR] (ERROR[null]) -> null
[BlueBubblesApp][2024-02-27 18:34:30.604595][ERROR] (ERROR[null]) -> null
[BlueBubblesApp][2024-02-27 18:34:30.688345][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:30.740924][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:32.648183][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:32.717029][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:32.961218][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:32.961789][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:32.961923][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:32.965123][INFO] Sending method get-server-url to Kotlin

....

[BlueBubblesApp][2024-02-27 18:34:40.774652][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:40.818161][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:41.835292][INFO] (REQUEST[GET]) -> PATH: https://imessage.mydomain.com:54321/api/v1/fcm/client
[BlueBubblesApp][2024-02-27 18:34:41.910744][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:41.911118][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:41.911236][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:41.915888][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:34:41.962439][ERROR] (ERROR[null]) -> PATH: https://imessage.mydomain.com:54321/api/v1/fcm/client
[BlueBubblesApp][2024-02-27 18:34:41.962565][ERROR] (ERROR[null]) -> HandshakeException: Handshake error in client (OS Error: 
	CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate(handshake.cc:393))
[BlueBubblesApp][2024-02-27 18:34:41.962640][ERROR] (ERROR[null]) -> null
[BlueBubblesApp][2024-02-27 18:34:41.962663][ERROR] (ERROR[null]) -> null
[BlueBubblesApp][2024-02-27 18:34:41.962821][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:41.962849][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:41.962886][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:41.964463][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:34:43.113867][INFO] Connecting to socket...
[BlueBubblesApp][2024-02-27 18:34:43.198279][INFO] Socket connect error, fetching new URL...
[BlueBubblesApp][2024-02-27 18:34:43.844429][INFO] Fetching new server URL from Firebase
[BlueBubblesApp][2024-02-27 18:34:43.844724][INFO] (FCM-Auth) -> Authenticating with FCM
[BlueBubblesApp][2024-02-27 18:34:43.844943][INFO] Sending method firebase-auth to Kotlin
[BlueBubblesApp][2024-02-27 18:34:43.848563][INFO] Sending method get-server-url to Kotlin
[BlueBubblesApp][2024-02-27 18:34:45.277396][INFO] Connecting to socket...

So it is an SSL error. I do have my CA certificate installed on my device, and double checked through my browser that it works, no certificate issues there.

[jjoelj]

Well, I guess that confirms that it's a certificate issue. And all sources I can find say that it's some issue with the certificate chain. You're right that the CA certificate should not be included in the chain, but double check that your self-signed certificate has all the correct intermediate certificates. Browsers tend to automatically complete the chain to account for misconfigured webservers, which is most likely why there are no issues when going through a browser.

You can view the completed certificate used by your browser, but steps vary based on what browser you're using. I'd recommend exporting whatever certificate the browser completes and comparing it to the one you have added to the server.

[tsndr]

There are no intermediate certificates in my case, my certificate chain looks like this:

CA Certificate
└─ imessage.mydomain.com

This matches the output of openssl s_client and my browsers output.

Only the imessage.mydomain.com certificate has to be send by the server, since the device has the CA certificate installed onto it.

[tsndr]

Could it be possible, that the app isn't respecting the systems CA certificates?

[jjoelj]

https://github.com/flutter/flutter/issues/41781#issuecomment-702948726

I did some more research, and it seems like we have to specifically allow user-installed CA Certs in the network security config on Android. I have no idea why this wasn't an issue before.

[tsndr]

Awesome find, can someone push an update?

Until now, I've just had a red certificate warning on the Connection & Server server page, but everything worked so I didn't really care.

[jjoelj]

Unfortunately, it seems like this is still an open issue with the dart HTTP client which is what gets used in BB https://github.com/dart-lang/sdk/issues/50435

The comment I referenced earlier fixed the issue for a person who wrote their own native code for HTTP, so it won't apply to us.

Again, I don't know why it was working before, but we will investigate what changed.

[tsndr]

Let me know if I can do anything to help.

[duronald]

Also having this issue, rolling back to the last stable version, v1.12.7 temporarily fixes it for now.

[duronald]

Upon further testing, it appears that previous versions of the app did not check the user certificate store at all. I removed my certificate and it still worked fine on previous versions. The new one must autoreject a connection if it can't find the cert and doesn't check the user certificate store.

[SpaceSaver]

Upon further testing, it appears that previous versions of the app did not check the user certificate store at all. I removed my certificate and it still worked fine on previous versions. The new one must autoreject a connection if it can't find the cert and doesn't check the user certificate store.

Make sure that the whole cert chain is installed.

[duronald]

I checked and unless I did something wrong all intermediates and root self signed certs are installed.