EricssonResearch
Support of hardware H.264 video on Android
I wonder what is the reason that Gstreamer androidmedia plugin (amc) is not used in OpenWebRTC under Android platform? openh264 could be used as a fallback.
For decoding that would be ok but the encoder API is too under specified to be able to be used reliably. The uncompressed input video format is not defined nor is it possible to specify and it is not the same on every device. And then even when Google fix it, the problem is that only new Android will support it.
What do mean by saying encoder API too under specified? The amc plugin uses at the end oficial http://developer.android.com/reference/android/media/MediaCodec.html API. That should hide most of the incompatibilities of various hardwares. It is definitly better then interface with the c-libriaries directly. I have tested AMC on two different Android devices with OpenWebRTC and it looks to work fine except the fact AMC itself needs minor improvements to use the MediaCodec API correctly.
@sdroege can answer better than I, but from what I have understood, the MediaCodec API does not allow enough information to be specified about the video format and memory layout of the input video frames. This means that you could end up with garbage output or segfaults unless you happen to get lucky.
If you have modifications to propose to the GStreamer AMC elements and can show that it works across different devices, we are absolutely interested in this and we will test on a few devices that we have too. :smile: It's not that we have anything against Android, just that the MediaCodec API has not been good enough so far and we have been tracking every Android update.
Maybe I get lucky ;), so this is why I am asking for more details if you can share. I bet you have not included AMC for a good reasons. Do you know if these garbage/segfaults applies to particular hardware or Android version? Which one? I would like to verify/reproduce it.
@marcin-sielski The main problem with the mediacodec API for encoding is that you don't know which color formats (layout, stride, plane offsets, etc) a specific encoder accepts. GStreamer could provide anything needed, we just don't know what is needed and it's different for every device.
I saw that there were some improvements in version 22 of the mediacodec API that improve some other aspects, but this still seems to be unsolved.
If you have any improvements to the androidmedia plugin, or ideas how things can be improved, please let us know :) GStreamer patch reviews, etc are happening at https://bugzilla.gnome.org (the GStreamer Bugzilla product there).
@sdroege Thank you for explanation. But still I have two different devices where I use encoder and it works just fine (Android v4.2.2, v5.1.1). I would like to find a device that is known cusing problems. I can imagine that there are devices based on the chipset family from the same manufacturer that will not work.
Which hardware is in those?
And you mentioned changes you had to do, what are those?
Some updates on that issue:
- I found currently one device where AMC ends up with strange colors (it is Exynos based).
- WebRTC.org added HW encoder support for H.264 for Androids starting from KitKat+ mainly because of possibility to change bitrate dynamically (MediaCodec.setParameters method).
- AMC works with JB with the limitation described in 2.
@marcin-sielski , could you please share your change about enabling hardware encoder on android? I'm also trying to verify it on android devices. thanks
