Support for transparent video

[jam-le]

When filing a feature request:

Fill out the sections below, leaving the headers but replacing the content. If you're unable to provide certain information, please explain why in the relevant section. We may close issues if they do not include sufficient information.

Playing Transparent Videos (with the transparency intact)

iPhones allow videos with transparent backgrounds to play. I would like ExoPlayer (on Android devices) to support videos which have an alpha channel. Currently, it is impossible to play high quality transparent videos that retain their transparent properties (i.e. their alpha channel properties).

Proposed solution

None, unfortunately.

Alternatives considered

Chromakey is one alternative solution, but tends to create grainy resolution around edges where transparency meets solids, along with other subtle issues.

[AquilesCanta]

I think for this to work you'll need to use a decoder that supports alpha channel, plus a texture view.

I am not sure whether any of the platform decoders supports alpha channel though, but it's possible to know this through CodecCapabilities: https://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities#colorFormats. If you cannot find a video codecs with the desired color format, you can bundle your own software decoder in your app. ExoPlayer already supports VP9 and AV1 and the ffmpeg video extension is in progress.

With a texture view and a alpha-channel-supporting decoder, this should work. Note however, that this is not something we'll be able to include in our roadmap to support out of the box any time soon, soo you'll need to do some research on your own.

@andrewlewis do you have any input on this?

[AquilesCanta]

Aside, a sample video would be welcome. It will help whoever tries to explore this line of work.

[andrewlewis]

To add to @AquilesCanta's suggestion about decoding via extensions, #2509 discusses rendering videos with an alpha channel using the vp9 extension. I think it's still the case that Android platform decoders don't support this.

[md84419]

+1 for chromakey support please

[md84419]

Aside, a sample video would be welcome. It will help whoever tries to explore this line of work.

See http://test.bsl.nrw/obv-client-demo/boy-test.html

The webpage uses the seriously.js JavaScript library from https://github.com/brianchirls/Seriously.js/ which in turn uses WebGL. The code is distributed under the MIT Open Source license. The chromakey implementation is at https://github.com/brianchirls/Seriously.js/blob/master/effects/seriously.chroma.js

Both videos are Copyright 3rd parties. The URL for the video is http://test.bsl.nrw/videos/signer.mp4

[lilinxiong]

@jam-le Hi～ I encountered the same problem as you, have you solved it now? In other words, is there a solution?

[md84419]

No solution yet. This issue is assigned to @andrewlewis.

[ColtonIdle]

Here is a video with transparency to use as a sample. https://static.videezy.com/system/resources/previews/000/048/733/original/200515-AlphaBluePlanet.mp4

[mirswith]

That alphaBluePlanet.mp4 video does not seem to have transparency (tested on iOS). This sample provided by Apple however does include a properly encoded video with alpha: https://developer.apple.com/documentation/avfoundation/media_playback_and_selection/using_hevc_video_with_alpha , included in the download is the file puppets_with_alpha_hevc.mov, you can convert this to an mp4 while retaining alpha with ffmpeg -i puppets_with_alpha_hevc.mov -codec copy puppets_with_alpha_hevc.mp4. I have tested this and confirmed the alpha is retained when playing on iOS.

[ColtonIdle]

@mirswith thanks! Now I just need to find an android player that can play it.

[mirswith]

Is the issue with ExoPlayer or the MediaCodec used for HEVC on android? I'm also trying to get alpha working with ExoPlayer but so far no luck.

[ColtonIdle]

I have no idea. Video codecs, formats, encoding, etc all go over my head. It's not my forte. I was hoping that there would be something by now that would support it though. @andrewlewis any chance there's some low hanging fruit around this to get this working with exoplayer? Thanks to @mirswith we do have a video sample that is confirmed to have alpha support.

[mirswith]

I would love to be wrong about this but it appears this is an issue with android.media.MediaCodec at least in my case. I am using a android.graphics.SurfaceTexture which ultimately gets set to the MediaCodec during its configuration which means the MediaCodec is responsible for rendering the video to that texture (not ExoPlayer). These surfaces use OES_EGL_image_external internally which does support the alpha channel. My shader is using the alpha component during rendering of that texture but the alpha is not being preserved. It would appear we have to wait for Android to support this directly and maybe they are with newer devices/android versions (I'm using a Pixel 4a for testing but this using the latest Android Version 11).

I have not tested this as it does not meet my needs but looking at the ExoPlayer VideoDecoderGLSurfaceView class the internal GLSL shader is not setup to handle alpha so that path would not work either but my guess is even if it did we would still need to wait for an update to the underlying video system on Android.

Again, I'd love to be wrong about this so if anyone else has some better news please set me straight! :)

[andrewlewis]

Just to check we are seeing the same thing, I tried playing the stream with alpha on a Pixel 4a and saw jagged green edges around the edges of the puppets, and a log line E QC2V4l2Codec: [hevcD_20] ? is not a supported pixel format! (this is using c2.qti.hevc.decoder). I don't have an iOS device here to test with but I assume the jagged green edges are not the expected behavior! Based on this, I agree it's likely the limitation is with android.media.MediaCodec. I think OES_EGL_image_external is the only efficient way to get MediaCodec output frames to GL. So there isn't much we can do on the ExoPlayer side when using platform decoding. Filed [internal: b/193006872] to ask about the limitations.

Using a bundled video decoder and VideoDecoderGLSurfaceView means we aren't relying on MediaCodec for video at all, so it should be possible to make this work (assuming EGL/GLES supports it). But we only have official extensions wrapping libvpx and libgav1 currently, so it wouldn't help with HEVC content like this. An FfmpegVideoRenderer could work though (#2159).

[andrewlewis]

It seems that on an iOS device there are still green pixels on the boundary but the background UI does show through the video beyond them (thanks to @claincly for testing this out).

[ColtonIdle]

@andrewlewis ah. So you were able to see this on iOS with the background UI showing, but on Android you just got a solid color? I'm assuming your comment still holds true then (An FfmpegVideoRenderer could work though)?

All in all, it seems like there's no "easy" way to get this to work so I will have to get back to my design team and tell them this is currently possible on iOS but not Android. Thank you all for helping.

[andrewlewis]

@andrewlewis ah. So you were able to see this on iOS with the background UI showing, but on Android you just got a solid color?

Right, I wasn't able to get the background UI to show through the SurfaceView (also tried with TextureView). I suspect the problem is with the decoder, but will report back here if we learn more on the internal bug.

I'm assuming your comment still holds true then (An FfmpegVideoRenderer could work though)?

In principle it should be possible to make this work with extension renderers that don't rely on the platform MediaCodec, but we don't have any plans to work on that currently.

[mugika9]

But we only have official extensions wrapping libvpx and libgav1 currently, so it wouldn't help with HEVC content like this. An FfmpegVideoRenderer could work though (#2159).

Is it currently possible to properly play vp8/vp9 video that have transparency with libvpx extension? I've try it but did not work and wondering if the problem lies on my implementation or it just currently not possible.

[whopavan]

Hey, I was trying to use a WebM file with transparent background in my app but instead of the background being transparent it's grey, is there no Android support for it?

[mirswith]

Unfortunately no, it does not yet support it. (tested with 2.14.0 and later release notes do not mention it).

[damikdk]

Any updates guys?

[Leonidos]

We want this feature )

[lucianobargmann]

I am very frustrated to learn that Android (which has its brand in a F1 car) does not support transparent video playback. I am having to rework all my videos into animated gifs, with HUGE loss in quality and INSANE increase in size.

Why isn't this a relevant feature for the OS?

[arturorey91]

ping

[Shivamdhuria]

any update on this?

[ankit-gupta2]

Any update ???

[yangcheng]

we want this feature too! is there any update?

[Shivamdhuria]

@andrewlewis any update on this? it's almost been 3 years since this issue was raised? Is this even on the roadmap?

[ankit-gupta2]

If this helps: https://github.com/google/ExoPlayer/issues/10957#issuecomment-1428047971

[andrewlewis]

I've pinged the internal bug again [Internal: b/190802665], but I'm afraid there aren't any substantive updates.

To give an idea about why it's taking a long time:

• Android device hardware accelerated codecs would need updates to support it, requiring involvement from chipset makers.
• It might be more feasible for software codecs (especially VP9/AV1) but this would still require changing how MediaCodec handles color formats internally.
• For on-screen playback, SurfaceView apparently doesn't not support translucency, and it's not clear if it's feasible to add it.

For now I'm trying to raise the importance of this feature with SoC vendors when we discuss the video editing use case, which we want to support better.