"Fastest" way to display h264 decoded frames on Android device

[Vinz1911]

Hi there,

I'm very happy with javacv which helped me a lot to decode a raw h264 video stream from my robot. The pure decoding into the javacv's frame works as fast es expected but I did not find a good way to display the frames on my android device. I used the FFmpegFrameGrabber with the pixelformat AV_PIX_FMT_RGBA and then AndroidFrameConverter to convert the frames into Bitmap and display it one a ImageView. That works but not perfect, I got some frame drops and screen tearing. So I need faster or better way to display it. I read that maybe it's a better way doing it with OpenGL ES but I didn't find a good starting point or example. Is the Frame rendering with OpenGL still the preferred way or are there better/faster solutions to display it on an Android device? Are there any helpful things you maybe can provide to help me out there?

This is the simple class I wrote which decodes and converts the Frame's

class Decoder(host: String, port: Int) {
    var completion: (Bitmap) -> Unit = {}
    private val scope: CoroutineScope
    private val grabber: FFmpegFrameGrabber
    private val converter: AndroidFrameConverter

    init {
        grabber = FFmpegFrameGrabber("tcp://$host:$port")
        scope = CoroutineScope(Dispatchers.Default)
        converter = AndroidFrameConverter()
    }

    fun start() {
        try {
            grabber.pixelFormat = AV_PIX_FMT_RGBA
            grabber.setVideoOption("threads", "1")
            grabber.start(); process()
        } catch (error: Exception) {
            Log.e("DECODER", "$error")
        }
    }

    fun stop() {
        try {
            grabber.stop()
        } catch (error: Exception) {
            Log.e("DECODER", "$error")
        }
    }

    private fun process() {
        while (!grabber.isCloseInputStream) {
            val frame = grabber.grabImage()
            val image = converter.convert(frame)
            scope.launch(Dispatchers.Main) { completion(image) }
        }
    }
}

Thank you very much and best Regards

[saudet]

What you're probably looking for is hardware acceleration. Try to call FFmpegFrameGrabber.setVideoCodecName("h264_mediacodec") and see what that gives.

[Vinz1911]

Thank you for you reply @saudet

But this does not work, that gives me the following error:

W/System.err: Error: [amediaformat @ 0x7681ee1480] No Java virtual machine has been registered
W/System.err: Error: [h264_mediacodec @ 0x7698d40c00] Failed to create media format

EDIT:

I'm not sure why I got this error, is there something I need to configure in Android Studio? In my gradle.build I have set up the following options:

compileOptions {
    sourceCompatibility JavaVersion.VERSION_1_8
    targetCompatibility JavaVersion.VERSION_1_8
}
kotlinOptions {
    jvmTarget = '1.8'
}

[saudet]

No, that's an error from FFmpeg. That should be done with the call to av_jni_set_java_vm(Loader.getJavaVM(), null), so I'm not sure what's going on here. If you try to call av_jni_get_java_vm() right after creating an instance of FFmpegFrameGrabber, but before calling start(), what does it return?

[Vinz1911]

Thank you, that worked. Now I see that is using MediaCodec but it throws infinite errors of:

W/System.err: Warning: [h264_mediacodec @ 0x7681f61400] Input packet is missing PTS

and it seems performance is worser then before

[saudet]

@tmm1 Any ideas?

[tmm1]

The way that mediacodec works requires PTS.

What kind of container is this h264 stream inside? Is it progressive or interlaced h264?

You may be able to use something like -fflags +genpts

[Vinz1911]

Thank you for your answer @tmm1 I'm not an Expert in this, I'm currently try to build my own control app for the DJI Robomaster. The only Information I got about the Video stream is the following:

OUT: H.264-encoded real-time video stream data with a resolution of 1280×720 and a refresh rate of 30 fps. 
The video stream data must be correctly decoded to display the video picture in real time.

So just tried to decode the Videostream from a tcp based connection and display it near realtime.

[tmm1]

What does ffprobe tcp://... show?

[Vinz1911]

This is what I got:

Input #0, h264, from 'tcp://192.168.2.1:40921':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1280x720, 30 fps, 30 tbr, 1200k tbn

[tmm1]

Okay so it's just raw h264 without a container. That explains missing PTS.

Try: grabber.setOption("fflags", "genpts")

[Vinz1911]

Sadly that didn't worked for me.

I adjusted my code like the following:

fun start() {
    try {
        FFmpegLogCallback.set()
        grabber.pixelFormat = AV_PIX_FMT_RGBA
        grabber.videoCodecName = "h264_mediacodec"
        grabber.setOption("fflags", "genpts")
        grabber.setVideoOption("threads", "1")
        av_jni_set_java_vm(Loader.getJavaVM(), null)
        grabber.start(); process()
    } catch (error: Exception) {
        Log.e("DECODER", "$error")
    }
}

but it's still throwing missing pts errors in a row

W/System.err: Info: Input #0, h264, from 'tcp://192.168.2.1:40921':
W/System.err: Info:   Duration: 
W/System.err: Info: N/A
W/System.err: Info: , bitrate: 
W/System.err: Info: N/A
W/System.err: Info: 
W/System.err: Info:   Stream #0:0
W/System.err: Info: : Video: h264 (Main), yuv420p(tv, bt709, progressive), 1280x720
W/System.err: Info: , 
W/System.err: Info: 30 fps, 
W/System.err: Info: 30 tbr, 
W/System.err: Info: 1200k tbn
W/System.err: Info: 
I/VideoCapabilities: Unsupported profile 4 for video/mp4v-es
D/ACodec: ACodec:m_framerate is 30!
I/MediaCodec: callingProcessName:de.weist.robomaster
I/OMXClient: MuxOMX ctor
I/ACodec: onAllocateComponent:7002 mSoftCodecPref:0 componentName:OMX.rk.video_decoder.avc
I/ACodec: onAllocateComponent:7033 mSoftCodecPref:0 componentName:OMX.rk.video_decoder.avc
D/MediaCodec: configure:in!
I/MediaCodec: msg kWhatConfigure in!
I/MediaCodec: msg kWhatConfigure out!
D/MediaCodec: configure:out!
D/MediaCodec: msg kWhatGetOutputFormat in !
D/MediaCodec: msg kWhatGetInputFormat/kWhatGetOutputFormat out !
W/System.err: Info: [h264_mediacodec @ 0x741d7fd400] Output crop parameters top=0 bottom=719 left=0 right=1279, resulting dimensions width=1280 height=720
W/System.err: Info: [h264_mediacodec @ 0x741d7fd400] MediaCodec started successfully: codec = OMX.rk.video_decoder.avc, ret = 0
W/System.err: Warning: [h264_mediacodec @ 0x741d7fd400] Input packet is missing PTS
W/System.err: Warning: [h264_mediacodec @ 0x741d7fd400] Input packet is missing PTS
W/System.err: Warning: [h264_mediacodec @ 0x741d7fd400] Input packet is missing PTS
...

[saudet]

Maybe it's setOption("genpts", "1")?

[Vinz1911]

still the same issue :(