Add WebRTC (WHIP) output support

[kc5nra]

Co-authored-by: Sean DuBois [email protected] Co-authored-by: DDRBoxman [email protected] Co-authored-by: tt2468 [email protected]

First and foremost, thanks to the original author of the draft PR for webrtc Colin Edwards (DDRBoxman).

Description

This PR adds WebRTC support to OBS. WebRTC is an alternative to RTMP that is being adopted by more services. Adding WebRTC support to OBS would benefit OBS and its users greatly. These are some of the highlights. Following this I also have a FAQ for questions we have already received regarding this PR.

This initial version is intentionally simplified to only support output. Later PRs will add source support.

Specifically, this adds a plugin that extends support for:

• WebRTC Service type with recommended settings for maximum compatibility with webrtc peers
• WebRTC Output
• Rust implementation of the WebRTC stack with FFI bindings

Remaining tasks:

• [x] Fix-up remaining CI build environments to correctly install and build rust
• [x] Ensure that the rust static libraries are built against the correct OSX compatability version
• [ ] Emit the correct dropped frames amount from peer stats
• [ ] ...

Motivation and Context

Subsecond Latency

With WebRTC we are able to achieve sub second latency. With our initial measurements we see ~120 Milliseconds from Broadcaster to Playback. We believe we can continue to bring down this number as well.

Interactive latency lets broadcasters create new experiences they couldn’t before. OBS users can now create interactive experiences with their viewers. This would allow talk shows and other productions that require conversational latency to use OBS.

Remove FTL from OBS

FTL is a protocol that was created for Mixer. It provided sub-second latency and was used by a few Open Source projects and streaming services. The protocol has been abandoned and it would be nice to remove it from OBS. Adding WebRTC would make that possible since it provides the latency required in addition to the many benefits inherent in the WebRTC stack like encryption, network topology strategies, robust congestion control, etc.

Latest in Video Compression

WebRTC will let OBS users experiment with newer codecs. WebRTC is flexible in what codecs it supports so users could use H265 and AV1.

AV1 users are reporting a 50% reduction in bandwidth required.

Broadcast all angles

WebRTC supports multiple video streams in the same session. OBS could experiment with broadcasting multiple scenes at once! Viewers could then switch between scenes or different views in the same game.

Simulcast

WebRTC allows broadcasters to upload multiple streams of different quality. OBS users could upload ‘high’, ‘med’, and ‘low’ streams themselves. This would reduce the server costs greatly, allowing more

Peer-to-Peer (if you need it)

With WebRTC OBS users could share their video without needing a video server at all! WebRTC can establish a P2P connection so users could send video from OBS directly to their users. No more worrying about setting up special servers to ingest and transcode.

In the future we could add a WebRTC source. Then broadcasters could add WebRTC sources to their scenes. This means low latency collaboration with no backend required!

How Has This Been Tested?

This has currently been tested on Linux and Mac with locally run broadcast-box as well as remotely hosted WHIP endpoints.

Configurations tested:

• Mac OSX M1 Monterey
• Debian Sid

Original draft PR: https://github.com/obsproject/obs-studio/pull/7192 Closes #7192

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• [x] My code has been run through clang-format.
• [x] I have read the contributing document.
• [x] My code is not on the master branch.
• [x] The code has been tested.
• [x] All commit messages are properly formatted and commits squashed where appropriate.
• [x] I have included updates to all appropriate documentation.

[kc5nra]

FAQ

How can I use this today?

Multiple companies provide WebRTC Broadcast/Playback today. Cloudflare and Millicast are two popular options.

Gilmesh has created and is hosting a public WebRTC server at https://b.siobud.com/. You can run this locally for developing and testing as well https://github.com/glimesh/broadcast-box

Why libdatachannel as the WebRTC library?

WebRTC has many libraries that we could have used. The first one we evaluated was Google's implementation known as libwebrtc. The time required to fetch+download would have been too much of a burden for the OBS code base. These metrics come from a Macbook M1 with a 400 Mb/s connection.

Fetching the source of libwebrtc: 11 minutes 20 seconds Installing the dependencies: 3 minutes and 30 seconds Building libwebrtc: 7 minutes Size added by libwebrtc: 35 megabytes

This caused us to evaluate other options. We arrived at https://github.com/paullouisageneau/libdatachannel and it had everything we needed! It is a C++ WebRTC implementation and uses CMake. It adds very litle impact to the project.

Additional Build time: 9 seconds Size Added: 11 Megabytes

What is WHIP?

WHIP is the process that WebRTC uses to establish a session between two WebRTC Agents (Client/Server in our case). The actual definition of it can be found at https://www.ietf.org/archive/id/draft-ietf-wish-whip-05.txt

[phillipmiller0044]

my name phillip miller playing ghost recon breakpoint hi thank you for your support

[renandincer]

Hi all - we at Cloudflare have been a part of the WHIP/WHEP community since the launch of our public server earlier this year. In addition to our WebRTC based video streaming server, we run public RTMP and SRT servers as well.

WHIP is a great alternative to RTMP because WebRTC protocols and browser APIs are IETF and W3C standards with broad adoption. WebRTC developer community is large. Many have already written WHIP client and server implementations and participate in the protocol discussions.

WHIP and OBS communities both leverage the power of open source and open standards.

OBS adding support for WHIP would benefit creators in the short term with features like enabling low latency broadcasting and output in modern codecs. These features driven by robust WebRTC standards.

In the long term, we are very excited by the creativity developers on the Web platform will bring to the OBS community. Creators that use OBS, and the community in general would greatly benefit from new uses of OBS enabled by this PR.

[tytan652]

flatpak doesn't provide a way to resolve FetchContent ahead of time

Find another way than a git submodule.

[eminence]

Testing report:

(Building this PR was trivial using the automated ci\build-windows.ps1 script. Thanks to the OBS team for creating this script)

Running the latest commit (993f104fe0f02ec7ad651e1ba2ff5f9e244086b0) on Windows 10. Streaming to https://b.siobud.com/ worked out-of-the-box with no tweaking needed. Streaming to cloudflare seems to half-work: It seemed like the stream was properly ingested by Cloudflare, but playback wasn't working. This could just be some issue with Cloudflare's beta webRTC support

[tt2468]

Could you describe what you're using as your encoder settings and WHEP player?

[eminence]

Could you describe what you're using as your encoder settings and WHEP player?

Encoder is nvenc h264, CBR ~600kbps "ultra low latency". For playback I was mostly relying on the built-in player in the Cloudflare Dashboard. I admit I'm not sure if that was using WHEP or not. Twice I was able to see something in this player, but none of my other ~15 attempts were successful. In the coming days I am going to try some other WHEP clients that Cloudflare reports as being supported.

[vpalmisano]

Streaming to cloudflare seems to half-work: It seemed like the stream was properly ingested by Cloudflare, but playback wasn't working. This would just be some issue with Cloudflare's beta webRTC support

I've tested the plugin with Mediasoup SFU. Audio and video are correctly negotiated and ingested after filtering the dtls fingerprints to sha-256. At client side, audio stream playback is ok, instead video doesn't work, maybe a H.264 packetization issue?

[kc5nra]

Could you describe what you're using as your encoder settings and WHEP player?

Encoder is nvenc h264, CBR ~600kbps "ultra low latency". For playback I was mostly relying on the built-in player in the Cloudflare Dashboard. I admit I'm not sure if that was using WHEP or not. Twice I was able to see something in this player, but none of my other ~15 attempts were successful. In the coming days I am going to try some other WHEP clients that Cloudflare reports as being supported.

Ensure you aren't checking disable suggested encoder settings as nvenc will not send the required sps/pps each idr otherwise

[PatTheMav]

I'll look into adding corrosion to Windows and macOS at obs-deps, it was simple enough for the latter, the former might be a bit of a wildcard.

[kevmo314]

Inspired by this PR, I've been working on adding WHIP support to FFmpeg here: https://github.com/kevmo314/ffmpeg-webrtc

It uses the same Rust module, would it make sense to share the library between FFmpeg and OBS if it were published somewhere independent? Also, the FFmpeg implementation runs into the issue of how to package the shared library along with the binary. I'm curious if this PR also runs into that/how to work around it? I couldn't get the Rust library to statically compile.

[tmatth]

Inspired by this PR, I've been working on adding WHIP support to FFmpeg here: https://github.com/kevmo314/ffmpeg-webrtc

It uses the same Rust module, would it make sense to share the library between FFmpeg and OBS if it were published somewhere independent? Also, the FFmpeg implementation runs into the issue of how to package the shared library along with the binary. I'm curious if this PR also runs into that/how to work around it? I couldn't get the Rust library to statically compile.

This is very cool. Ideally if such a library had C bindings (like rav1e does) the chance of adoption and integration in other projects would be a lot higher.

[murillo128]

Just tested it with Millicast/Dolby Streaming WHIP endpoint and it is working perfectly, nice work!

[lminiero]

It works nicely with Janus too, although with some caveats:

First of all, it looks like negotiation may be broken: in fact, OBS seems to be offering multiple codecs, but will then fail if the server chooses anything except H.264. I guess this is intended since H.264 is what's configured in the settings, but in that case I think H.264 is all that OBS should negotiate. When a different codec is negotiated, the UI just shows a "Failed to connect" modal dialog, and the logs contain a misleading error like this:

11:22:44.388: [webrtc_rs_lib::output:260]: Failed connecting to: unable to start track, codec is not supported by remote
11:22:44.388: [webrtc_rs_lib::output:144]: Worker encountered error: ConnectFailed

This is misleading since negotiation was successful, but the codec the server chose was different from the one OBS wanted to use.

Apart from this, everything worked nicely, which was a pleasant surprise. I just had to enforce "42e01f" as the profile to use to get video to work in Firefox, since the other advertised profile resulted in Firefox rejecting video (but that's not a problem in this integration, I just specified this as an FYI).

I'm definitely looking forward to, and supporting. this effort proceeding!

[ASchneiderBR]

This also works great with VDO.Ninja, here's a video from Steve - https://youtu.be/ynSOE2d4Z9Y

[Sean-Der]

@lminiero thanks for testing! I will get that fixed. I had hardcoded the test server to be H264 only and forgot to follow up on that.

In the future I want to get multiple codecs working (AV1 especially seems to excite people)

[tytan652]

Please avoid filling the PR of only feedbacks, this could make the PR getting locked because it makes less easy to check. PR are meant for mostly code review.

I had hardcoded the test server to be H264 only and forgot to follow up on that.

OBS GUI is already hardcoded to stream only H264 or HEVC.

In the future I want to get multiple codecs working (AV1 especially seems to excite people)

I'm already working on allowing more codecs depending on the streaming protocol capability (RFC 45).

[mpisat]

does anybody have any pre-built binaries (with WHIP support) that I can test? Merging remains to be blocked and I can't compile obs-studio in mac os (m1 pro) for some reason -- autobuild script fails at some point.

[Fenrirthviti]

does anybody have any pre-built binaries (with WHIP support) that I can test? Merging remains to be blocked and I can't compile obs-studio in mac os (m1 pro) for some reason -- autobuild script fails at some point.

Artifacts are available here: https://github.com/obsproject/obs-studio/pull/7926/checks

[mpisat]

does anybody have any pre-built binaries (with WHIP support) that I can test? Merging remains to be blocked and I can't compile obs-studio in mac os (m1 pro) for some reason -- autobuild script fails at some point.

Artifacts are available here: https://github.com/obsproject/obs-studio/pull/7926/checks

Thanks a lot! Downloading it right now for a WHIP test :)

[Sean-Der]

@lminiero @tmatth The Offer should only contain H264+Opus now!

Fixed with 76897211cbab854f398209c623a0b3033f6d5c6a

[PatTheMav]

While adding Corrosion to obs-deps I've ran into several issues not addressed by the PR and also found an architectural issue:

• Windows build scripts have not been updated
• Pre-installed rust environments are not taken into account
• Issues with rust environments installed via winget or brew (which do not ship with rustup) are not taken into account

On CI one can pollute the environment without much care, as the images are ephemeral, but that's not true for local development and we usually aim to change/modify local environments as little as possible. On Windows, whichever rust environment comes first in the system path will be picked up corrosion, which in my case lead to the arm64 environment being used for x64 compilation (which broke obviously).

The same happened with my local macOS environment when I tried to build for x86_64. I solved this by using rustup-init (provided by Homebrew) to install a local rust environment where I could change/add targets needed. On Windows I ensure the currently required toolchain comes first in $Env:Path.

Architecturally I think it's not desirable to build a dependency as part of the main code-tree (that's why we have obs-deps in the first place): webrtc-rs-lib should be pre-compiled by obs-deps (where it would be built as a native Rust project) and then the static C++ library generated from that as well. The archives would contain libwebrtc_rs_lib.a (just like we ship the AJA NTV2 SDK as a static library) and bindings.h (renamed to libwebrtc_rs_lib.h to avoid collisions with a too generic name).

On obs-studio we'd just ship a FindLibWebrtc.cmake file which discovers both and allows the plugin to just link to webrtc-rs-lib (as it does already).

This would:

• Reduce complexity of the plugin's CMake file tremendously
• Allows the Rust library to be easily pinned to required dependencies in the more "stable" obs-deps repository
• Reduces the need to re-compile the rust library for every obs-studio project generated
• Reduces compilation times for obs-studio
• Removes the need for corrosion (which lacks support for macOS-specific CMake features and also generates CMake packages files that are not portable)

As webrtc-rs-lib is an external dependency (just like FFmpeg, Qt, and others) it should be implemented as such.

[PatTheMav]

FWIW I just extracted the rust directory, built the library standalone with cargo, copied the static library into my obs-deps directory, renamed bindings.h to webrtc_rs_lib.h and copied it there as well (I had to use the one generated prior because the Rust project does not generate the header without environment variables set) and quickly created FindLibWebrtc.cmake.

I replaced all corrosion code from the CMakeLists.txt with find_package(LibWebrtc REQUIRED) and changed the header include to #include <webrtc_rs_lib.h>. Plugin compiled fine and was able to stream using WHIP (just got lots of lines with this warning: warning: [webrtc_mdns::conn:356]: Failed to parse mDNS packet parsing/packing of this type isn't available yet).

[Be-ing]

Pre-installed rust environments are not taken into account Issues with rust environments installed via winget or brew (which do not ship with rustup) are not taken into account

Have you reported these bugs to Corrosion?

webrtc-rs-lib should be pre-compiled by obs-deps (where it would be built as a native Rust project) and then the static C++ library generated from that as well. The archives would contain libwebrtc_rs_lib.a (just like we ship the AJA NTV2 SDK as a static library) and bindings.h (renamed to libwebrtc_rs_lib.h to avoid collisions with a too generic name).

This seems like it could create a hassle for downstream packagers.

[tytan652]

This seems like it could create a hassle for downstream packagers.

I forgot about this, :facepalm:.

[tytan652]

~~Outside or the in-tree or out-of-tree library issue that needs to be discussed.~~

This feature needs to have a CMake option ENABLE_WEBRTC/ENABLE_WHIP (ON by default of course). Yes, only RTMPS have one for now but this is required.

Not all downstream packagers will package OBS Studio with WHIP.

[tt2468]

I don't think any discussion needs to be had regarding the out of tree thing. We're going to put the rust code into its own repository, let obs-deps build it for the platforms where required, and let package maintainers release the binaries on dedicated packages, to allow other applications to use it.

[tytan652]

and let package maintainers release the binaries on dedicated packages

They will never do that because it's meant only for OBS in the actual situation, they will just disable the feature (with or without toggle for that).

[Elf36]

Can I compile & build this into my current latest official OBS on Mac M1 ?

[tytan652]

@Be-ing, libwebrtc_rs_lib (not the best name) will normally became a standalone library meant to be packaged as shared library on Linux distribution and used by other software like FFmpeg. If I understood the situation.