Wincrypto does not like Firefox

[xnorpx]

Error handling UDP input: Dtls(CryptoError(WinCrypto(Hresult(HRESULT(0x80090320))))) Source: CryptoError(WinCrypto(Hresult(HRESULT(0x80090320)))) Error handling UDP input: Dtls(CryptoError(WinCrypto(Generic("Handshake failed")))) Source: CryptoError(WinCrypto(Generic("Handshake failed")))

fyi @efer-ms

0x80090320 is a Windows error code that means SEC_E_NO_CREDENTIALS - "No credentials are available in the security package."

[xnorpx]

let's see what copilot can do :)

[efer-ms]

The issue does not appear to be with the certificate generated by wincrypto, but rather SChannel is rejecting the ClientHello from Firefox.

[efer-ms]

Setting media.peerconnection.dtls.version.max to 771 (DTLS 1.2), seems to get things working. This was set to 772 by default. So something in DTLS 1.3 in Firefox and SChannel. SChannel should support 1.3 on Win11 that I'm testing on.

[xnorpx]

Set media.peerconnection.dtls.version.max to 771

For people that want to work around it for now

[algesten]

DTLS 1.3 and WebRTC, is that really a thing?

[efer-ms]

DTLS 1.3 and WebRTC, is that really a thing?

Apparently it is in Firefox (https://bugzilla.mozilla.org/show_bug.cgi?id=1884140)

My guess is that SChannel is mishandling the ClientHello, and probably does TLS 1.3 related things and breaks. Like my suspicion is on SChannel being wrong over FF. AFAICT nothing in the ClientHello should cause an issue for a DTLS 1.2 implementation.

My guess is that SChannel sees some of the extensions or cipher changes in the ClientHello and gets confused. The added extensions in the ClientHello are delegated_credentials, key_share, and supported_versions. Other changes are the addition of a few ECs in supported groups, and the TLS 1.3 cipher suites

[efer-ms]

Ok. I've narrowed down the issue to the presentation of the 'cookie' extension in the DTLS Client Hello. When this is presented, SChannel fails. If we strip that extension out of the ClientHello, SChannel will send a ServerHello, but then fail when verifying the Encrypted Client Handshake, since we modified the hello.

Chrome/Edge is also doign DTLS 1.3, but importantly do NOT provide the cookie extension.

[efer-ms]

From my investigation, this isn't fixable from the SChannel application side.

Firefox should probably not send the cookie extension in the second client hello if the server sent a HelloVerifyRequest (which indicates the server is NOT DTLS 1.3). This is what Chrome/Edge does.

[efer-ms]

Firefox has a bug for this issue... https://bugzilla.mozilla.org/show_bug.cgi?id=1314849

[Frosne]

Oh, hi. I am Anna, Firefox.

I am curious if there is a simple way to test the patch when we implement it? I am not very familiar with SChannel.

[xnorpx]

@Frosne Thanks for quick reply!

We can probably write up a simple example or you can share a private build of FF and we can verify it for you.

Let us know what you prefer.

[Frosne]

As I mention in the bug, this week I am away, but I will try my best to look at the problem the next week.

I think it's an NSS bug, so we don't need the full firefox :) NSS is significantly smaller. What about this - I write a nss patch, then you will have to apply the patch and re-build NSS. This should be enough. Would it work for you?

[xnorpx]

If we can just swap out the nss dll on Windows and use it firefox that will be fine. I guess we can also just write out some tests with it. Thoughts @efer-ms ?

[Frosne]

I will also definitely write DTLS tests on our side :)

[efer-ms]

I'm not an expert at SChannel either.

I'll see if I can write something simple where you can just run the DTLS server part of str0m, and then you can try to connect to it as a client. Seems usful to have for other cases where we're debugging our dtls impls.

[algesten]

@efer-ms if you look at dimpls integration test how they rig openssl it maybe is useful as inspiration.

[Frosne]

Thanks to a great work of a colleague of mine, we have a patch - https://phabricator.services.mozilla.com/D272447 (you need only lib/ssl/tls13exthandle.c bit). It would be so nice if you can check that now it works!

Thank you!

[efer-ms]

I have a simple DTLS Server example setup over in this fork.. cargo run --example dtls_server --manifest-path wincrypto/Cargo.toml

You can run it like so: cargo run --example dtls_server --manifest-path wincrypto/Cargo.toml

I was attempting to get it to talk to tstclnt from the nss directory. I managed to get to the point that NSS rejects cause the cert is self-signed, and it doesn't find a client cert to present. I didn't see if there was a way to get tstclnt to generate a cert, or a way to provide one from a file.

[efer-ms]

for reference: openssl s_client -dtls -cert ecc_cert.crt -key ecc_private.key -connect 127.0.0.1:30000 -mtu 1200 -use_srtp SRTP_AEAD_AES_128_GCM will complete the handshake.

Here is the log from the test server: test_str0m_dtls_server.log

The log form openssl is less exciting:

Connecting to 127.0.0.1
CONNECTED(00000004)
Can't use SSL_get_servername
depth=0 CN=localhost
verify error:num=18:self-signed certificate
verify return:1
depth=0 CN=localhost
verify return:1

@Frosne

[efer-ms]

It looks line -o on tstclnt did the trick. So with the patched code ./tstclnt.exe -h 127.0.0.1 -P client -p 30000 -I P256,P384 -D -o .. completes the DTLS handshake, albeit without a Client Certificate presented back and a failure looking for the use_srtp extension which isn't present.

test_str0m_dtls_server_tstclnt.log

@Frosne

For reference, here is the log when using a tstclnt without the code change: test_str0m_dtls_server_tstclnt_unpatched.log

[xnorpx]

@Frosne any update from your side?

[Frosne]

Hi! The patch has just landed to NSS: https://hg-edge.mozilla.org/projects/nss/rev/fa9fa5503a876406b113b2eacf7de239f711d550

[xnorpx]

@Frosne do you have rough eta when this will hit Firefox nightly?

[Frosne]

I think it should already be in Nightly

[xnorpx]

Did not work. @Frosne can you verify it should be in:

Dtls error which is not recoverable disconnecting dtls_err=CryptoError

[efer-ms]

@Frosne , it looks like the original issue regarding the Cookie is resolved

However, it now appears that the client hello is fragmented. I think SChannel has issues with that. It preivously wasn't fragmented. Any idea why the hello has blown up to be fragmented now? Looks like it's mostly the "key_share" extension which shars a 1216 byte key for group X25519MLKEM768. I don't think that was there before.

....

<UPDATE> If I set security.tls.enable_kyber to false to disable PQC everything works! </UPDATE>

@Frosne , do you know when it's intended to enable kyber in stable releases? It looks like we need to contact the schannel folks to investigate why fragmented client hellos fail, since I assume as PQC becomes the norm.. if a large key like that needs to be exchanged we'll be hitting the fragmentation limit in other browsers too.

[efer-ms]

Confirmed with the Schannel team that they have a bug regarding fragmented client hellos and are wokring on a fix. @algesten , I think we should close out this issue though, since the initial issue is solved. We can track the large hello issue separately if we'd like, or not worry about it for now.

When SChannel has a fix, it should roll out to supported windows versions, so unsupported windows will still be broken. I don't think that should be an issue, as if you're using wincrypto, it's cause you want to use supported OS APIs for crypto. If you're ok not using OS APIs you can just use openssl or dimpl.

[xnorpx]

Let's close it, not a str0m issue. Firefox is fixed once it rolled out. (unless they add kyber) which will be fixed by Microsoft in relevant Windows versions.