Green and purple tint seen on Chrome when using hardware H.264 encoder

[dat0106]

Describe the bug I use video call on Samsung S22(android 13) and OnePlus 10T(android 12), then view them on the web(https://example.livekit.io/#/) has video error(color incorrect), but some devices as Samsung A53(android 13) worked fine.

Screenshots

Device Info:

• Device: [e.g. Samsung s22]
• OS: [e.g. Android 13]
• LiveKit SDK version: [e.g. 1.1.5]

[davidliu]

@dat0106 What are you using to start the camera capture? Any options passed to the Room?

[dat0106]

@davidliu, I used code default in client-sdk-android testing with my server Url and my token, I also tested on flutter for android also got the same error, and also some devices suck as Samsung s22 and OnePlus 10T, We got the error on chrome (client-sdk-react-native, and Firefox works fine.

update: URL cloud: wss://ledat.livekit.cloud

token : eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NzU3MDM3MjksImlzcyI6IkFQSUtjOUY4REVVazZiMyIsIm5iZiI6MTY3NTcwMjgyOSwic3ViIjoid2ViIiwidmlkZW8iOnsiY2FuUHVibGlzaCI6dHJ1ZSwiY2FuUHVibGlzaERhdGEiOnRydWUsImNhblN1YnNjcmliZSI6dHJ1ZSwicm9vbSI6ImRhdCIsInJvb21Kb2luIjp0cnVlfX0.05SKjEHnyOvtMNY8W94mlCkvIcgPQ9xBl5BWR1c0Qzk

ERROR

[andrewdittmer]

Also getting this color issue with Samsung Galaxy S22. We are only passing in adaptiveStream = true as a parameter for RoomOptions. But removing that parameter didn't have an effect. The video header is showing RGB color space but WebRTC signal is showing YUV. Hope that helps.

[davidliu]

Hmm, I don't have direct access to a Galaxy S22; can debug on a cloud device but progress with that will be slow.

@andrewdittmer Can you point me where you're getting the video header info? Also, would you happen to know which codec this is happening with?

[khmak3]

the issue happen on AVC codec. If you egress the video stream and use mediainfo to extract header information. You can get it color.txt

[khmak3]

Firefox don't have playback issue. The decoder should use the video header to render instead follow the Webrtc signal. Moreover, the egress file can be playback without issue on chrome and edge

[davidzhao]

Looks like the video is published at High profile level 3.1. WebRTC only supports H.264 constrained mainline 3.1

@davidliu seems like we'll need to force Android SDK to negotiate only constrained mainline?

[davidzhao]

@khmak3 thank you for sharing the codec info there. that's super helpful. 🙏

[dat0106]

when with only one decode stream at web it will work fine

I think we have a temporary fix that is config server:

   enabled_codecs:
     - mime: audio/opus
     # - mime: video/vp8
     - mime: video/vp9
     # - mime: video/h264

[davidzhao]

@dat0106 that's helpful to know.. it looks like Chrome decodes it incorrectly when it's also encoding on windows.

By the way, you do not want to disable vp8 there. LiveKit does not support vp9. Leaving h.264 disabled would workaround this.

[dat0106]

yah, vp9 and vp8 work fine, at my service i want to check more about video quality which remove h264. As you said VP8 is currently the default video codec in WebRTC.

[davidliu]

@dat0106 hey, to clarify, you said client-sdk-react-native works fine? That sdk has high profile disabled, so would make sense here.

[dat0106]

yes, when I disable h.264 in service, client-sdk-react-native and client-sdk-js work well with multiple video streams coming in at the same time. If h.264 has, they just work as I mentioned above. if only Samsung s22 video stream (browser disable video), Chrome or Edge decode correctly. When participants are added, the video stream from Samsung s22 is decoded incorrectly on the browser. It's just a hotfix for my business. I'm still trying to figure out the problem. If you know or have any ideas about it, please let me know!

[davidliu]

@andrewdittmer @dat0106 can you try testing with overriding the videoEncoderFactory to turn off H264 high profile?

    LiveKit.create(
        appContext = application,
        overrides = LiveKitOverrides(
            videoEncoderFactory = SimulcastVideoEncoderFactoryWrapper(
                sharedContext = EglBase.create().eglBaseContext,
                enableIntelVp8Encoder = true,
                enableH264HighProfile = false,
            )
        )
    )

[dat0106]

@andrewdittmer @dat0106 can you try testing with overriding the videoEncoderFactory to turn off H264 high profile?

    LiveKit.create(
        appContext = application,
        overrides = LiveKitOverrides(
            videoEncoderFactory = SimulcastVideoEncoderFactoryWrapper(
                sharedContext = EglBase.create().eglBaseContext,
                enableIntelVp8Encoder = true,
                enableH264HighProfile = false,
            )
        )
    )

I tried it, but it didn't work

[davidliu]

@dat0106 can you check if your Galaxy S22 has an Exynos encoder? I'm not able to repro on Galaxy S22, but it seems like there's different hardware depending on region.

val list = MediaCodecList(MediaCodecList.REGULAR_CODECS).codecInfos
for (codec in list) {
    Log.e("Decoders", "name: ${codec.name}")
}

It should look like omx.exynos...

[dat0106]

@davidliu Samsung s22 (Qualcomm SM8450 Snapdragon 8 Gen 1 (4 nm) ) code.txt

[davidliu]

Does this issue appear on other platforms, or only web?

[dat0106]

Does this issue appear on other platforms, or only web?

@davidliu only web (chrome and edge), firefox working

[davidliu]

@andrewdittmer @dat0106 do you have a intel PC by chance?

Did some more testing, I'm able to repro on Intel PCs. AMD PCs seem to work fine, as does an Intel Mac loaded into bootcamp. Not really sure what the cause is yet, but I'll continue to investigate.

[davidliu]

A quirky finding is that the "picture in picture" mode (found in the right click context menu of the video element) and fullscreen mode seems to temporarily remedy the issue.

[dat0106]

A quirky finding is that the "picture in picture" mode (found in the right click context menu of the video element) and fullscreen mode seems to temporarily remedy the issue.

that's correct, in addition, you can zoom in and this error will go. I'm currently using AMD Pc.(Note: Ubuntu is this bug)

[andrewdittmer]

@andrewdittmer @dat0106 do you have a intel PC by chance?

Did some more testing, I'm able to repro on Intel PCs. AMD PCs seem to work fine, as does an Intel Mac loaded into bootcamp. Not really sure what the cause is yet, but I'll continue to investigate.

Intel PC for me.

[davidliu]

Okay, tested H264 on our web sample client with the Samsung S22 against PC on web, still reproing there (though turning off hardware acceleration on the PC's web client also fixes). Not really sure what can be done to fix this, other than to start blacklisting these devices from using H264.

Filed an issue on Chromium here: ~~https://bugs.chromium.org/p/webrtc/issues/detail?id=14929~~

New bug at: https://bugs.chromium.org/p/chromium/issues/detail?id=1418596