Let’s Encrypt certificates for TURNS are rejected

[rajil]

I tried dev 0.23 build from https://buildkite.com/matrix-dot-org/riotx-android/builds/2964

It has the same TURNS error reported previously, here and here. The coturn is checked it with is 4.5.1.2.

The coturn config looks like so,

listening-port=3478
tls-listening-port=5349
alt-listening-port=3479
alt-tls-listening-port=5350
listening-ip=192.168.1.10
external-ip=xx.xx.xx.xx
min-port=49152
max-port=65535
verbose
fingerprint
use-auth-secret
static-auth-secret=xx
realm=turn.mydomain.com
userdb=/var/lib/turnserver/turndb
stale-nonce=600
cert=/etc/letsencrypt/live/mydomain.com/fullchain.pem
pkey=/etc/letsencrypt/live/mydomain.com/privkey.pem
cipher-list="ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AES:RSA+3DES:!ADH:!AECDH:!MD5"
no-multicast-peers
mobility
cli-password=xx
no-tlsv1
no-tlsv1_1

[532910]

I believe this is a security issue!

[hex-m]

Our friends over a Jitsi seem to have a similar problem. (issue, discussion)

[a1ad]

yeah, same here: unknown CA

[dhaminitinAltran]

facing the same issue has anyone got a solution yet?

[buzzdx]

i'm having the same problem. synapse and coturn are both set up to use turns. when i try to make a call from element android app to another android app on a different network the turnserver logs look like this:

turnserver[14012]: 860: IPv4. tcp or tls connected to: a.b.c.d:58145 turnserver[14012]: 860: session 001000000000000028: TLS/TCP socket disconnected: a.b.c.d.5:58145 turnserver[14012]: 860: session 001000000000000028: usage: realm=<my_realms>, username=<>, rp=0, rb=0, sp=0, sb=0 turnserver[14012]: 860: session 001000000000000028: closed (2nd stage), user <> realm <my_realms> origin <>, local h.i.j.k:5349, remote a.b.c.d, reason: TLS/TCP socket buffer operation error (callback)

when i switch the turn_uris to begin with turn: instead of turns: it works flawlessly from any device to any device (phone to app, app to phone, all over different networks).

as soon as tls is used with the android app it does not work anymore. this seems to be a real problem as there are more than one discussions about it on github and other sites.

coturn has access to the certs, which are let's encrypt certs, as many others described.

this may be off topic, but there's also a problem with the android app where you cannot verify anything if your first login after account creation is made with the android app. had to use desktop/web app to do the first login, after that all works as expected.

would be really great if somebody could solve this, or point to a solution or potential upcoming fix. thanks.

edit: is it possible that this is the same problem we are seeing here? https://community.jitsi.org/t/android-app-rejecting-lets-encrypt-certificates-for-turns/29468

quote of the final post on that page:

The problem – as I understand it – is on the client side:

If the Android app connects to the webserver (nginx) via https it verifies the server certificate using the list of trusted root CAs that is built into the Android OS. Let’s Encrypt certificates are verified successfully. However, if the app cannot establish a connection to the videobridge directly and needs to fallback to the turnserver a “different part” of the Jitsi Meet Android app is used to establish the turns:// connection: This connection is apparently handled by a webrtc library (which seems to be included in the Jitsi App as a native library via react-webrtc). And the webrtc library does not use the list of trusted CAs which is provided by the Android OS. Instead, another list of trusted CAs is compiled into this library and only the CAs contained in this hardcoded list are used to verify the certificate that the app receives from the turn server. Unfortunately, this hardcoded list of CAs does not include the trust anchor that’s needed to trust a certificate issued by Let’s Encrypt.

[buzzdx]

It seems that what was mentioned in my previous post is correct. I switched to ZeroSSL certificates as replacements for Let's Encrypt certificates, and without any other change, Element Android was immediately able to connect voice and video calls using turns across different networks. So the problem is indeed that for some reason, maybe that webrtc library having an incomplete list of trusted CAs as mentioned above, Element Android does not accept Let's Encrypt certificates. The Jitsi devs have an open ticket where they stated that for now they will add Let's Encrypt to the webrtc-lib's trusted ca list. Maybe someone could do the same for Element Android?

[asimons04]

Can confirm using ZeroSSL certs with coturn is a workaround to avoid having to fallback to no SSL. Just make sure you combine your cert + ca bundle into a fullchain for coturn, though.

[buzzdx]

thanks for confirming the workaround works asimons04. now is there anything we can do to make the devs aware of the problem and possible solution? i tried email contact but they couldn't help, from what i understand they do not give support for self hosting. they told me: Please also submit debug logs from Element Android after you have created your issue and link them together. I'm not really sure how to do that, pretty new to github. if anybody here knows , maybe you could submit the needed data? thanks

[tleydxdy]

fwiw the issue eventually comes down to this https://bugs.chromium.org/p/webrtc/issues/detail?id=11710 which is part of the webrtc dependency.

[AndrewFerr]

Note that this issue affects matrix.org's TURN server too, which makes sense given that it uses a Let's Encrypt certificate.

From my testing, this issue prevents calls to connect if all of the following conditions are met:

• at least one client is Element Android (I haven't tested with the iOS client)
• call traffic is required to go through the TURN server (which I tested by having one client behind a corporate VPN that apparently blocks direct p2p traffic)
• only turns connections are available (which I tested by blocking traffic on ports 3478-3479)

Under these conditions, not only did calls between users on my own homeserver fail (which uses a Let's Encrypt cert for its TURN server), but calls between matrix.org accounts failed to connect, as well.

[progserega]

Switch to ZeroSSL is not allow for all. Some countries is banned:

[xuhdev]

fwiw the issue eventually comes down to this https://bugs.chromium.org/p/webrtc/issues/detail?id=11710 which is part of the webrtc dependency.

Nice catch, and a comment is suggesting that users of the library should not use their truststore at all: https://bugs.chromium.org/p/webrtc/issues/detail?id=11710#c11

[mrPjer]

An alternative to ZeroSSL is also the Namecheap PositiveSSL certificate - https://www.namecheap.com/security/ssl-certificates/comodo/positivessl/.

Tried it out and it works.

[karolyi]

I just had a WebRTC call via matrix and by looking at the remote candidates (my buddy was using element android) it seems that letsencrypt certificates work. My own coturn server which offers TURNS, started to show up as a remote candidate with both IPv4 and IPv6 addresses.

The call was built in the end with my Librewolf using my TURN server, that talked to directly his IPv6 address. Call quality was good.

Can someone confirm?

[bessw]

I can confirm. Just tried it and first one mobile could start calls, the other not, but after signing out (make sure you have a key backup) and logging in again it worked on that mobile, too.

We tested with or without VPN and mobile data or WIFI:

• Element Android - Element Android
• Element Android - Element Desktop

[karolyi]

Sounds good.

Now the only question remains, what's the underlying cause of it suddenly working? Is something in LetsEncrypt's cross-signers changed or is this a security hole?

[theAeon]

Could just mean that google finally got around to fixing the WebRTC inbuilt certs.

[maximushugus]

I can confirm the problem is still here for me. I self host synapse and coturn, and I use Letsencrypt certificate. Using TURN (not encrypted) working without any problem, I can make call behind nat from Android Element app, from Firefox or Google Chrome. But as soon as I use TURNS (encrypted) with letsencrypt certificate, I'm not able to make call behind NAT

I do not want to use another certificate provider. Is there any solution ? Why Element doesn't trust Letsencrypt ?

Thanks

[Josue-T]

Hello,

Tested between element web and element android with this following synapse config and it looks like working correctly

turn_uris: [ "turns:matrix.domain.tld:<port>", "turns:matrix.domain.tld:<alt_port>" ]

Checked with wireshark and I saw a DTLS negotiation so I think it's working with let's encrypt.

So I think the issue is fixed.

[532910]

@Josue-T: im.vector.app or io.element.android.x and which version?

[532910]

can confirm im.vector.app works with turns tcp

[Josue-T]

@Josue-T: im.vector.app or io.element.android.x and which version?

With im.vector.app from fdroid on /e/ OS.

[maximushugus]

I still have the problem with im.vector.app installed directly from this github repository (v1.6.12) (on a degoogle lineageOS phone) : if one (or both) of the 2 devices involved in a call is behind NAT, and using TURNS with a Let's Encrypt certificate, the call doesn't work. If I switch to TURN (not encrypted), with the same condition, it does work. My TURN server is working correctly even with TURNS with a certificate from Let's Encrypt, because i also use it for other programs without any problem.

For people who managed to make it work, are you sure you are using TURNS with Let's Encrypt, behind a NAT ?

[ketan-vijayvargiya]

I too think this issue is now resolved.

This is what I have in my NixOS config:

    coturn = {
      enable = true;
      use-auth-secret = true;
      static-auth-secret-file = config.age.secrets.coturnStaticAuthSecret.path;

      realm = coturnRealm;

      no-cli = true;
      secure-stun = true;
      cert = "${config.security.acme.certs."${coturnRealm}".directory}/fullchain.pem";
      pkey = "${config.security.acme.certs."${coturnRealm}".directory}/key.pem";
    };

    matrix-synapse = {
      enable = true;

      extraConfigFiles = [ config.age.secrets.coturnStaticAuthSecretForSynapse.path ];

      settings = {
        // Bunch of other stuff.

        turn_uris = [ "turns:${coturnRealm}?transport=udp" "turns:${coturnRealm}?transport=tcp" ];
        turn_allow_guests = false;
      };
    };

I and spouse use Element for Android and iOS respectively, and we are able to call each other.