Issue using OpenH264 version 2.6.0 in Firefox for MS Teams

[spwalmsley]

OS = Oracle Linux Server 7.9 (similar to Red Hat) browser = Firefox 128.5.1 (esr version)

ISSUE: When we initially started using the web version of Microsoft Teams (https://teams.microsoft.com) we had to enable the OpenH264 plugin to get video (e.g. camera images, screen capture) to work. At the time the OpenH264 version was 2.3.2, and once it was enabled video within Teams worked properly.

Starting yesterday (~24-Mar-2025) we stopped being able to view video in the online Teams app. Whilst checking that OpenH264 was still enabled we noticed that OpenH264 had been automatically updated to version 2.6.0 on 22-Mar-2025.

To see if 2.6.0 was the culprit we re-installed version 2.3.2 and verified that Teams video now worked properly again.

Has anyone else had similar issues with MS Teams video on Linux whilst using versions of the OpenH264 encoder > 2.3.2? Is it possible that there is an incompatibility with Teams in newer versions, or perhaps that we need to do some additional configuration for newer versions (e.g. additional security settings in Firefox, etc.)?

[aosmond]

Firefox media team checking in, I just tested on Firefox nightly and Firefox 128.5.1 ESR on Ubuntu 22.04 with OpenH264 2.6.0, and it seems to work just fine, encoding and decoding. You shouldn't need to do anything special to make it work.

Could you capture WebRTC + GMP (OpenH264) logs and share them with me so I can help diagnose? Thanks!

On Linux this is pretty easy, you can just set the MOZ_LOG environment variable, run Firefox and capture stdout/stderr to a file, i.e.: MOZ_LOG="jsep:5,sdp:5,signaling:5,mtransport:5,nicer:5,RTCRtpReceiver:5,RTCRtpSender:5,RTCDMTFSender:5,WebrtcTCPSocket:5,CamerasChild:5,CamerasParent:5,VideoEngine:5,ShmemPool:5,TabShare:5,MediaChild:5,MediaParent:5,MediaManager:5,MediaTrackGraph:5,cubeb:5,MediaStream:5,MediaStreamTrack:5,DriftCompensator:5,ForwardInputTrack:5,MediaRecorder:5,MediaEncoder:5,TrackEncoder:5,VP8TrackEncoder:5,Muxer:5,GetUserMedia:5,MediaPipeline:5,WebAudioAPI:5,webrtc_trace:5,RTCRtpTransceiver:5,ForwardedInputTrack:5,HTMLMediaElement:5,HTMLMediaElementEvents:5,GMP:5,GMPLibrary:5" /path/to/firefox 2>&1 | tee teams.log

[aosmond]

Really even just the GMP logs will probably give enough, e.g., MOZ_LOG="GMP:5,GMPLibrary:5" /path/to/firefox 2>&1 | tee teams.log

[spwalmsley]

Thanks for your assistance with this.

Unfortunately I don't have a video call scheduled this evening test this, but I will give it a shot in the next couple of days (hopefully tomorrow).

On the off chance that the logs show something diagnostic even without a video stream active I started Teams from two separate profiles, one using OpenH264 2.3.2 and one using 2.6.0, and captured the logs which I have attached as a .zip archive.

sean

-----Original Message----- Subject: Re: [cisco/openh264] Issue using OpenH264 version 2.6.0 in Firefox for MS Teams (Issue #3870) To: Cisco/openh264 @.> Cc: Spwalmsley @.>, Author @.> From: Andrew Osmond @.> Date: Thu, 27 Mar 2025 16:38:34 -0400

Really even just the GMP logs will probably give enough, e.g., |MOZ_LOG="GMP:5,GMPLibrary:5" /path/to/firefox 2>&1 | tee teams.log|

— Reply to this email directly, view it on GitHub https://github.com/cisco/openh264/issues/3870#issuecomment-2759422259, or unsubscribe https://github.com/notifications/unsubscribe-auth/BQ5ZIVXYJQDRLSRKR4Q3F6L2WROUVAVCNFSM6AAAAABZ54ECSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONJZGQZDEMRVHE. You are receiving this because you authored the thread.Message ID: @.***>

aosmondaosmond left a comment (cisco/openh264#3870) https://github.com/cisco/openh264/issues/3870#issuecomment-2759422259

Really even just the GMP logs will probably give enough, e.g., |MOZ_LOG="GMP:5,GMPLibrary:5" /path/to/firefox 2>&1 | tee teams.log|

— Reply to this email directly, view it on GitHub https://github.com/cisco/openh264/issues/3870#issuecomment-2759422259, or unsubscribe https://github.com/notifications/unsubscribe-auth/BQ5ZIVXYJQDRLSRKR4Q3F6L2WROUVAVCNFSM6AAAAABZ54ECSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONJZGQZDEMRVHE. You are receiving this because you authored the thread.Message ID: @.***>

================================================================= Sean Walmsley @.***

[aosmond]

I don't see any attachments to the bug. Maybe github rejected them.

[aosmond]

Feel free to send them directly to me at my username at mozilla.com.

[aosmond]

I reviewed the logs, and the plugins are downloaded/detected correctly, so I'll need a log from the success/failure Teams cases to debug further.

[spwalmsley]

I logged 4 separate sessions in MS Teams, this time with a video feed to exercise the OpenH264 plugin, as follows:

OpenH264 version 2.3.2 with minimal logging (session worked) OpenH264 version 2.6.0 with minimal logging (session failed) OpenH264 version 2.3.2 with maximum logging (session worked) OpenH264 version 2.6.0 with maximum logging (session failed)

where "minimal logging" was with MOZ_LOG="GMP:5,GMPLibrary:5", "maximum logging" was with MOZ_LOG set to the more extensive string you provided above.

Note that none of these sessions had sound enabled as I had a telephone connection to maintain communication as I started/stopped the Teams session.

I had a quick comparison of the minimal logs and the first sign that I can see of a significant difference between the 2.3.2 and 2.6.0 log is:

FROM 2.3.2: [Parent 19355: GMPThread]: D/GMP GMPParent[7f070b349f00|childPid=20261] LoadProcess: Sent StartPlugin to child process

FROM 2.6.0: [Parent 20644: GMPThread]: D/GMP GMPParent[7f26d8ac3500|childPid=23207] LoadProcess: Failed to send start to child process

I will send the logs as before, but note that the second batch is ~8+ MB

Thanks for your assistance with this.

[aosmond]

You are correct, it looks like it is crashing in the RecvStartPlugin method. This typically happens due to: https://searchfox.org/mozilla-central/rev/5bea6ede57be43d450ecc24af7a535288c9a9f7d/dom/media/gmp/GMPLoader.cpp#122

Now I am not seeing the crash reason above getting logged, which I would have thought that would cause, but sometimes these get lost when a process crashes. I'm very excited to dig deeper into this, since 1) we have some crash reports about this, but haven't had a means to reproduce, and 2) your logs also show how/why we might be missing more crash reports, which we also have observed but not consistently.

I'd like to ask for a bit more information if you are willing, and can provide a simpler test case for you to try. This is failing so early, we only need the GMP logs going forward.

Hopefully easier steps to reproduce:

1. Run LD_DEBUG="all" NSPR_LOG_MODULES="all:5" MOZ_LOG="GMP:5" /path/to/firefox 2>&1 >gmp.log
2. Browse to https://mozilla.github.io/webrtc-landing/pc_test.html
3. Check "Require H.264 video".
4. Press Start.
5. GMP process should crash, close Firefox.

Be warned, this is going to generate a huge log file. Over 1GB for me. I don't need the whole thing, only everything starting from:

[Parent 84014: GMPThread]: D/GMP GMPParent[7d0b525df600|childPid=84396] LoadProcess: Launched new child process
[Parent 84014: GMPThread]: D/GMP GMPParent[7d0b525df600|childPid=84396] LoadProcess: Opened channel to new child process
[Parent 84014: GMPThread]: D/GMP GMPParent[7d0b525df600|childPid=84396] LoadProcess: Sent storage id to child process
[Parent 84014: GMPThread]: D/GMP GMPParent[7d0b525df600|childPid=84396] LoadProcess: Sent preload-libs ('libdl.so.2, libpthread.so.0, librt.so.1') to child process
[GMP 84396: Main Thread]: D/GMP GMPChild[pid=84396] Init pluginPath=/firefox/profile/path/gmp-gmpopenh264/2.6.0 useXpcom=1, useNativeEvent=1
[GMP 84396: Main Thread]: D/GMP GMPChild[pid=84396] RecvProvideStorageId
[GMP 84396: Main Thread]: D/GMP GMPChild[pid=84396] RecvStartPlugin

to this in the success case:

-1878558848[78d28f84d3a0]: Loaded library /firefox/orofile/path/gmp-gmpopenh264/2.6.0/libgmpopenh264.so (load lib)
[Parent 84014: GMPThread]: D/GMP GMPParent[7d0b3d566300|childPid=84407] LoadProcess: Sent StartPlugin to child process

and to this in the failure case:

[Parent 20644: GMPThread]: D/GMP GMPParent[7f26d8ac3500|childPid=23207] LoadProcess: Failed to send start to child process

This will provide additional context from our platform abstraction code, and from the OS shared library loader.

Thank you very much! The logs you have already provided give some great hints on how to fix not getting crash reports in some cases, so I am hoping the rest will solve the other mystery :). Hopefully this won't require getting symbols and attaching to the debugger on the GMP process, but I can walk you through that as well if needs be. This may also help us understand why we have Widevine loading failures sometimes on Linux.

[aosmond]

Additionally, could you provide me the contents of your about:support just in case?

Since this is probably a load library failure, you can also try launching Firefox in the failure case by disabling the GMP sandbox. Follow the steps above, but with the additonal environment variable MOZ_DISABLE_GMP_SANDBOX=1, e.g.: MOZ_DISABLE_GMP_SANDBOX=1 LD_DEBUG="all" NSPR_LOG_MODULES="all:5" MOZ_LOG="GMP:5" /path/to/firefox 2>&1 >gmp.log

If that somehow works, I'd love the logs from the 2.6.0 and 2.3.2 cases there as well.

[spwalmsley]

I attempted the debugging steps you detailed using the "https://mozilla.github.io/webrtc-landing/pc_test.html" page but I am not getting the same type of GMP output that you were expecting and that I saw when logging a Teams call containing video.

I note that when I follow the "simplified steps" using EITHER H.264 version 2.3.2 (works with MS Teams) or 2.6.0 (doesn't work with Teams):

• navigate to that page
• select H.264 video
• press "Start"

that I don't see any video activity on the screen, and that I get the following error message:

Parsing JSON: Failure callback: "The object can not be found here."

I'm starting to think that this page is not successfully launching an H.264 session in my browser for some (unrelated?) reason and thus the test output isn't actually covering what we're expecting. That is, I'm not seeing any instances of:

LoadProcess: Launched new child process LoadProcess: Opened channel to new child process LoadProcess: Sent storage id to child process etc.

I'll retry this process with an actual MS Teams session, but I'll have to wait until I can organize this (likely next week).

[aosmond]

Interesting, I only see that specific message when there is no camera detected because I left it unplugged. If it is detected but I don't give permission to access it, I get a different message, so I'm guessing it is the former. Strange that MS teams manages but the test page does not.

[spwalmsley]

I'm using Teams to display other attendees video (e.g. screen shares) on my Linux server to a remote machine via VNC, but my machine doesn't actually have a camera, i.e. I'm using a one-way video feed. Since I don't have a camera the error I'm seeing is consistent with your experience.

I hope to complete the additional logging tomorrow.

[spwalmsley]

I launched MS Teams sessions using both versions of H.264 and the logging string you suggested (LD_DEBUG="all" NSPR_LOG_MODULES="all:5" MOZ_LOG="GMP:5") and extracted the sub-blocks of output you requested. Note that in the 2.3.2 case there were two large blocks of log output, and that in the 2.6.0 case there were 5 much smaller blocks. I will forward these blocks as a .zip archive as before.

Taking a very quick look at the log blocks from the non-functional 2.6.0 case, it appears that the load of libgmpopenh264.so is failing as follows:

8873: checking for version CXXABI_1.3.9' in file /lib64/libstdc++.so.6 [0] required by file /home/sean/.mozilla/firefox/ehdetb44.TEST/gmp-gmpopenh264/2.6.0/libgmpopenh264.so [0] 8873: /lib64/libstdc++.so.6: error: version lookup error: version CXXABI_1.3.9' not found (required by /home/sean/.mozilla/firefox/ehdetb44.TEST/gmp-gmpopenh264/2.6.0/libgmpopenh264.so) (fatal)

Checking online it appears that this may be because the h.264 library was compiled using a newer C++ ABI than is supported on our Oracle Linux 7.9 platform?

The following page:

https://gist.github.com/nchaigne/ad06bc867f911a3c0d32939f1e930a11

states that it may be possible to use "the compiler flag -D_GLIBXX_USE_CXX11_ABI=0 to set a define that is needed in order to interop with system libraries on centos7 that were built using the pre-c++11 ABI". I assume that this would need to be incorporated into the build process for libgmpopenh264.so, is that correct? Is there any chance of this, or is there any other way to make the existing library compatible with our platform?

Thanks!

[matkoniecz]

In my case I was getting "The gmpopenh264 plugin has crashed" popups and broken video in MS Teams (outgoing one was not reaching others, incoming one was fine)

sudo dmesg was filled with apparmor logs complaining about firefox, I reported it at https://bugzilla.mozilla.org/show_bug.cgi?id=1956549

examples included [ 9149.120185] audit: type=1400 audit(1741179598.366:5202): apparmor="DENIED" operation="open" class="file" profile="firefox" name="/proc/12151/oom_score_adj" pid=6192 comm="firefox" requested_mask="w" denied_mask="w" fsuid=1000 ouid=1000

the advice they gave me was to delete /etc/apparmor.d/usr.bin.firefox created by buggy upgrade to get actual apparmor profile

it caused this warning to go away, crash also went away, MS Teams started working

[icolquhoun]

In my case I was getting "The gmpopenh264 plugin has crashed" popups and broken video in MS Teams (outgoing one was not reaching others, incoming one was fine)

I am experiencing the same problem.

sudo dmesg was filled with apparmor logs complaining about firefox, I reported it at https://bugzilla.mozilla.org/show_bug.cgi?id=1956549

examples included [ 9149.120185] audit: type=1400 audit(1741179598.366:5202): apparmor="DENIED" operation="open" class="file" profile="firefox" name="/proc/12151/oom_score_adj" pid=6192 comm="firefox" requested_mask="w" denied_mask="w" fsuid=1000 ouid=1000

the advice they gave me was to delete /etc/apparmor.d/usr.bin.firefox created by buggy upgrade to get actual apparmor profile

it caused this warning to go away, crash also went away, MS Teams started working

I disabled the usr.bin.firefox apparmor profile as well and the DENIED errors in the logs have cleared up, however, I'm still getting gmpopenh264 plugin crashes and outgoing video and screen sharing are broken. Incoming, and local video are fine.

Firefox 138.0 (was also happening on 137.0) (dpkg version from Mozilla's PPA) Openh264 plugin 2.6.0 Ubuntu 25.04