moq icon indicating copy to clipboard operation
moq copied to clipboard
verified publisher icon kixelated

Browser support / possible certs issue (local dev demo not working)

Open wormoworm opened this issue 2 years ago • 9 comments

I'm working on a proof-of-concept that aims to demonstrate video streaming from one machine to another using QUIC.

My first goal in this project is to demonstrate streaming "via localhost" - i.e. running the relay and publisher on the same machine as I view the stream on. To do this have been following the Local Development guide. The machine in question is running Windows 11, and the moq-rs binaries are all running inside a WSL2 Ubuntu 20.04 container.

Errors I am seeing I am running into problems when trying to view the stream in a browser. This is what I see in various browsers I have tried:

  • Firefox prod v119.0.1 results in the error "NotSupportedError: WebTransport constructor: No support for serverCertificateHashes yet". This might be expected, as the Issues pages does mention no Firefox support for WebCodecs yet - however this is not the sort of error message I would expect in this case!
  • Firefox nightly v121.0a1, with the dom.media.webcodecs.enabled flag set to true results in the same error as above.
  • Chrome prod v119.0.6045.124 results in the error "WebTransportError: Opening handshake failed."

The steps I am following are:

  • Installed go via apt and cargo via the install helper script they provide.
  • Installed ffmpeg v4.4.4-0ubuntu1~20.04.sav1.1
  • Downloaded the sample video to dev/source.mp4.
  • Run ./dev/cert to generate the certificate.
  • Run ./dev/relay to start the relay. It compiled without errors initially, and now runs without needing to compile anything. When I run this, I see the relay start up and it says it is listening on port 4443.
  • In a separate terminal (leaving relay running), run ./dev/pub. It appears to run correctly, and the bottom line of output is updated once per second and shows an increasing frame counter.
  • I then try and view the stream by visiting the URL the publisher prints out: https://quic.video/watch/Hefring?server=localhost:4443. Hefring is the name of my machine, which happens to be set in the $NAME env var. At this point I see the errors described above.

Other things I have tried

  • Adding the WSL2 network adapter's address to the cert command, so the cert includes it in the call to mkcert, just like it does for 127.0.0.1. No change, the same errors are shown.

  • Running the same setup on my home server, which runs dedicated Ubuntu 20.04. However, on this machine, the pub script does not successfully manage to start ffmpeg (no frames are sent). However, running pub-file does produce an output file on disk!

Questions

  • Have I understood the basic setup correctly? Should what I am doing (run the relay in the background, then run the publisher, then view the stream) work out of the box?
  • It looks to me to be a certificate issue - but I'm not sure why, as I was able to run the customised mkcert tool just fine (I think).
  • What browser (and version) is known to work to stream data from the relay?

Any help with this would be greatly appreciated. I'm not really sure what's going on here - I'm a bit of a newbie to this sort of thing! And if you are able to share the details of a known working setup, perhaps I can use that to help diagnose my issue.

I have attached logs from a run of the relay and publisher, in case they help: moq-rs-logs-1.txt

Thanks for any help in advance!

wormoworm avatar Nov 15 '23 13:11 wormoworm

It sounds like serverCertificateHashes isn't supported on Firefox yet. We use this hash to serve thr self-signed cert for local development because of a Chrome bug; it doesn't use the local CA for WebTransport only.

  1. Can you try Chrome? Firefox is untested.
  2. Can you comment out serverCertificateHashes in moq-js? It might work in Firefox depending on the cert manager.

When you search for where to comment out, you might notice that it only uses the fingerprint if the hostname starts with "localhost". You can use a different hostname to avoid this behavior if you don't want to run your own copy of moq-js.

Something like: echo dev.moq 127.0.0.1 >>> /etc/hosts And then: https://quic.video/watch/Hefring?server=dev.moq:4443

kixelated avatar Nov 15 '23 13:11 kixelated

  • I have tried Chrome already (v119.0.6045.124), but I get "WebTransportError: Opening handshake failed". I am now using Chrome for my testing, as FF seems to add more issues.
  • I realised I wasn't following the recommend method fully. I was not using moq-js - instead I was simply opening the link that the pub script was echoing out.
  • I have now installed moq-js via npm, but I get the same result in Chrome: "WebTransportError: Opening handshake failed".
  • Is there a difference between:
    • Opening the link from pub in a browser: https://quic.video/watch/Hefring?server=localhost:4443
    • Running moq-js and using the page it auto-opens for me https://quic.video/watch/Hefring?server=localhost:4443
  • I did search for serverCertificateHashes in the moq-js codebase, and I only saw it in one place: client.ts, line 37. I tried commenting this, but I'm not sure I'm seeing what you think I should be! I also don't see anything to do with localhost in that file.
  • I edited my hosts file to point dev.moq to 127.0.0.1 and then tried the UI at https://quic.video/watch/Hefring?server=dev.moq:4443, but I still got the same "WebTransportError: Opening handshake failed".

wormoworm avatar Nov 15 '23 15:11 wormoworm

I think I made some progress - instead of passing a hostname in the server GET param, I tried passing the IP address of the network adapter that WSL is using, giving me this URL: https://quic.video/watch/Hefring?server=172.30.137.205:4443.

Now when I hit that URL, I see more activity in the relay - it gets further through the handshake before failing. Below is the output I see when hitting that URL in both Chrome and FF nightly:

---------------------------------
CHROME
---------------------------------

[2023-11-15T15:13:03Z DEBUG rustls::server::hs] decided upon suite TLS13_AES_128_GCM_SHA256
[2023-11-15T15:13:03Z DEBUG rustls::server::hs] Chosen ALPN protocol [104, 51]
[2023-11-15T15:13:03Z DEBUG moq_relay::session] received QUIC handshake: ip=[::ffff:172.30.128.1]:58199
[2023-11-15T15:13:03Z WARN  moq_relay::quic] connection terminated: failed to establish QUIC connection

    Caused by:
        aborted by peer: the cryptographic handshake failed: error 46: 199:TLS handshake failure (ENCRYPTION_HANDSHAKE) 46: certificate unknown


---------------------------------
FIREFOX NIGHTLY
---------------------------------

[2023-11-15T15:13:25Z DEBUG rustls::server::hs] decided upon suite TLS13_AES_128_GCM_SHA256
[2023-11-15T15:13:25Z DEBUG rustls::server::hs] Chosen ALPN protocol [104, 51]
[2023-11-15T15:13:25Z DEBUG moq_relay::session] received QUIC handshake: ip=[::ffff:172.30.128.1]:62107
[2023-11-15T15:13:25Z WARN  moq_relay::quic] connection terminated: failed to establish QUIC connection

    Caused by:
        aborted by peer: the cryptographic handshake failed: error 48

wormoworm avatar Nov 15 '23 15:11 wormoworm

I think you stumbled upon the fact that the QUIC server is binding to a different address than the browser is trying to connect to. Ignore Firefox for now, it's a red herring.

You can't use an IP address in the ?server parameter unless the TLS certificate is valid for that IP. Try adding the IP to dev/cert or adding the equivalent of /etc/hosts to have localhost.moq point to 172.30.137.205.

I believe that Quiche binds to a single interface instead of all interfaces like intended with 0.0.0.0. I've never tested this code on Windows but yeah, the split network interfaces sounds like an issue.

kixelated avatar Nov 16 '23 06:11 kixelated

I made some more progress today by running everything (relay, publisher and moq-js client) on a Pi4. This worked pretty much out-of-the-box, which is great as it proves that most of the setup is fine.

What I'm trying to get working now is running the client on another machine on the same LAN (so relay and publisher on the Pi, client on another machine). My process is as follows:

  • Edit dev/cert on the Pi and add the Pi's IP address to the mkcert cert generation call on line 18.
  • Run ./dev/cert on the Pi.
  • Run ./dev/relay on the Pi and leave it running.
  • In a new shell, run ./dev/pub and leave it running.
  • On a machine on the same LAN as the Pi, visit https://quic.video/watch/dev?server=10.0.1.199:4443. 10.0.1.199 is the Pi's IP address, and it matches the address I added to the mkcert call earlier. At this point, I see the same error (shown below) printed in the output of relay:
[2023-11-17T17:06:42Z DEBUG rustls::server::hs] decided upon suite TLS13_AES_128_GCM_SHA256
[2023-11-17T17:06:42Z DEBUG rustls::server::hs] Chosen ALPN protocol [104, 51]
[2023-11-17T17:06:42Z DEBUG moq_relay::session] received QUIC handshake: ip=[::ffff:10.0.1.50]:54252
[2023-11-17T17:06:42Z WARN  moq_relay::quic] connection terminated: failed to establish QUIC connection

    Caused by:
        aborted by peer: the cryptographic handshake failed: error 46: 199:TLS handshake failure (ENCRYPTION_HANDSHAKE) 46: certificate unknown

The error displayed in in the MoQ UI in Chrome is "WebTransportError: Opening handshake failed." Any idea on what I'm missing?

One thing I have noticed is that the readme for the development scripts mention that the serverFingerprints options must be used - but I don't see any occurence of this string in this repo, or in moq-js. Since this section of that readme talks about WebTransport and certificates, I feel like it might be relevant!

wormoworm avatar Nov 17 '23 17:11 wormoworm

mkcert needs to be run on the host that will accept the cert. That means mkcert on the client and copy the generated cert to the server (Pi).

serverCertificatesHash not serverFingerprints

kixelated avatar Nov 18 '23 13:11 kixelated

I tried running mkcert on the client machine (the machine that will recieved the video stream), but I'm still getting a TLS error 46 during handshake. Below are the steps I'm taking:

  • Edit dev/cert on the client machine and add the Pi's IP address (10.0.1.199 static) to the mkcert cert generation call on line 18. Also set the HOST on line 7 to be the IP of the Pi.
  • Run ./dev/cert on the client machine.
  • Copy the generated cert (/usr/local/share/ca-certificates/mkcert_development_CA_123def) from the client machine to the same directory (/usr/local/share/ca-certificates/) on the Pi.
  • Run update-ca-certificates on the Pi to refresh the CAs.
  • Copied the generated 10.0.1.199.crt|key files to the Pi and move them to the moq-rs/dev directory.
  • Run ./dev/relay on the Pi and leave it running.
  • In a new shell, run ./dev/pub and leave it running.
  • Access the stream in Chrome on the client machine by navigating to https://quic.video/watch/dev?server=10.0.1.199:4443.

I'm still getting the same TLS error 46 - below are all the logs that result from attempting to load the stream URL in Chrome on the client machine:

[2023-11-20T17:58:48Z DEBUG rustls::server::hs] decided upon suite TLS13_AES_128_GCM_SHA256
[2023-11-20T17:58:48Z DEBUG rustls::server::hs] Chosen ALPN protocol [104, 51]
[2023-11-20T17:58:48Z DEBUG moq_relay::session] received QUIC handshake: ip=[::ffff:10.0.1.50]:49530
2023-11-20T17:58:48.936467Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2023-11-20T17:58:48.936798Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: [::ffff:10.0.1.50]:49530, src_ip: Some(::ffff:10.0.1.199), enc: Some(Ect0), len: 1305, segment_size: Some(1200) }
[2023-11-20T17:58:49Z WARN  moq_relay::quic] connection terminated: failed to establish QUIC connection

    Caused by:
        aborted by peer: the cryptographic handshake failed: error 46: 199:TLS handshake failure (ENCRYPTION_HANDSHAKE) 46: certificate unknown

wormoworm avatar Nov 20 '23 18:11 wormoworm

I had some (limited) success by using Firefox. In FF, you can make exception for individual certificates, so I visited https://<pi_ip>:4443 whilst the relay was running, and accepted the certificate. I was then able to visit https://quic.video/watch/dev?server=<pi_ip>:4443 and view the video stream. However, the stream itself is very unstable (I'm guessing this is caused by FF's WebTransport implementation):

  • The video needs to be low bitrate (the highest that I could get working was a 240p sample).
  • If the Pi is connected via Wi-Fi, the stream will halt after roughly 1s. On ethernet, it is able to continue indefinitely.
  • The WebCodecs option flashes and flickers a lot (not an issue with MoQ I guess, more to do with FF).
  • The Media Source option is what seems to work more reliably.
  • In general, to get the client to work, I need to reload the page a few times, and sometimes I'll need to restart the relay and / or publisher too.

I believe my problems boil down to the following two statements:

  1. I am unable to get Chrome to trust the certificate that is being served by relay. As per the posts above, I've tried generating certificates on both the server and the client machine, and I've tried every method I know of importing the generated CA into the Windows / Ubuntu certificate stores. The error I see in Chrome's console is net::ERR_QUIC_PROTOCOL_ERROR.QUIC_TLS_CERTIFICATE_UNKNOWN (TLS handshake failure (ENCRYPTION_HANDSHAKE) 46: certificate unknown)
  2. The only browser that will allow me to trust the certificate is Firefox, which currently has a very buggy implementation of WebTransport / WebCodecs.

Are you able to share how you import certificates into the certificate store in such a way that Chrome will trust them? mkcert reports that it has installed the CA in FF's and Chrome's trust stores: The local CA is now installed in the Firefox and/or Chrome/Chromium trust store (requires browser restart)!, but Chrome is still not accepting the cert.

wormoworm avatar Nov 21 '23 17:11 wormoworm

Hi, I'm still not able to get Chrome to accept the self-signed certificate, despite mkcert reporting The local CA is now installed in the Firefox and/or Chrome/Chromium trust store (requires browser restart)!.

Any ideas on what I might be missing?

wormoworm avatar Nov 27 '23 15:11 wormoworm