H264 simulcast support

[Ryner01]

Hello,

Would it be possible to add simulcast support for H264 codec in webRTC calls? I understand that this would be very challenging by itself but since you already have an infrastructure capable of VP8 simulcast I figured that it should at least be possible.

From what I have seen h264 simulcast is already implemented in libwebrtc so almost all clients should support it by default. Including Safari & Chrome (together with native apps). Still not sure about the FF support. It could fallback to VP8 simulcast in that case since from the subscriber perspective it's just a single VP8/H264 feed. Both have standard support for all clients.

Here are just some of the reasons why I think this feature could be worthwhile:

• Apple hardware has no support for VPX HW accelerated codecs
• Even new modern APUs like ryzen 4XXX have no hardware support for VP8. Just VP9 & H family codecs.
• According to apple: We found that, on an iPhone 7 Plus in laboratory conditions, the use of H.264 on a 720p video call increases the battery life by up to an hour compared to VP8 source
• I understand that Google is pushing VP9 SVC but there will probably not be any hardware support for that ever (at least from Apple). And it's not implemented in Twilio anyway.
• For our particular use case (https://www.getwelder.com/) freeing up CPU cycles by at least decoding on hardware is very very beneficial. Since we are running encoding in the background together with other intensive tasks like screen sharing.

I think that if Twilio apps are to stay competitive it should start with infrastructure & call quality (including the battery draw). Hangouts are already implementing VP9 SVC and FaceTime is already running on HW acceleration for a long time. From what I remember Zoom is also using H264 SVC which can at least be decoded in hardware (If I understood it correctly)

Sorry if this is not the right place to propose this feature. If that is the case I will try to find a feedback email address.

[Bumbolio]

This has been a limiting factor for our application which is a mix of web clients and iOS clients. The lack of H.264 simulcasting means that if any of the clients connected have a poor internet connection, all the video tracks will be reduced in quality to accommodate the slowest client. When producing live video content, it's expected that the video going out to the live audience maintains a high image quality, even if some of the room participants are unable to receive that many tracks at that high of a bit rate. Our product uses iOS devices to mix remote participants and local WiFi cameras to produce live content. While VP8 is possible in iOS, using a software encoder greatly limits what we are able to do, so we've opted to limit the video room to H.264 only.

[manjeshbhargav]

Hi @Ryner01 , @Bumbolio ,

Sorry for the delayed response. Unfortunately, H264 simulcast support is not in our near-term roadmap for 2021. However, I will make sure to bring this up in our future planning sessions so that we can get enough momentum for working on this.

Thanks,

Manjesh

[ceaglest]

I'm really happy there is an issue to track this. FWIW, it helps to have both internal and external requests for these kind of features.

At this point, both mobile and JS video SDKs do have support for H.264 simulcast but intentionally disable it due to lack of support on the media server (you might notice something to this effect in logs as you are negotiating media in a Room). One issue with plain H.264 simulcast is that temporal scalability with droppable P-slices (nal-ref-idc = 0) is "informal" rather than formally defined. So it would definitely be possible to achieve resolution switching with 2 or 3 spatial layers, but frame rate would be likely remain fixed regardless of the layer that is forwarded to the subscriber.

[JackKora]

We are having the same problem and would love H.264 Simulcast support for the same reasons. Better yet SVC support :)

[manjeshbhargav]

We have de-prioritized this feature.