avcodec_decode_video2 and got_picture_ptr

[sobotka]

Regarding frameFinished in https://github.com/mpenkov/ffmpeg-tutorial/blob/master/tutorial01.c#L150

It is worth noting here that frameFinished might result in missed frames. Consider the following frame format:

F1 F2 F3 F4
II BB BB PP
01 02 03 04 PTS (Presentation Time Stamp)
01 04 02 03 DTS (Decode Time Stamp)

In the above example, avcodec_decode_video2 would return got_frame with nonzero for the first frame, but what happens with the Bi-Predictive frame for frame 02? A Bi-Predictive frame requires both the previous frame and a frame somewhere after it. In this case, the frame it needs is in frame 04. So avcodec_decode_video2 will return got_frame with zero for frame 02 and 03, buffering the frames until it can properly decode them. At the point that it hits frame 04, it is now able to decode frame 02 and 03, and will return got_picture with a non-zero value.

01 02 03 04 05 06 Calls to avcodec_decode_video2
++ -- -- ++ ?? ?? got_picture_ptr returns zero on -- frames
01 -- -- 02 03 04 Order of returned frames

What does all this mean? By the time we reach the end of the file, although we have presented every frame we can decode in correct order, we will have ended up skipping a given number of frames for every B frame. The presentation will stop after decoding every frame, but not after processing every frame by our application!

As discussed, those frames are buffered internally. This means that at the end of the primary loop, every frame has been processed, and all of the information needed to process every B frame is now complete. For every time that got_picture returned zero, our buffer still grew by a frame, but avcodec_decode_video2 couldn't return it as it would be out of Presentation Time Stamp order. So where are those frames by the time we reach the end of this primary loop?

They are simply sitting buffered. To display all of the frames in the file, we must continue to call avcodec_decode_video2 after the primary loop until got_frame returns zero. This should be the number of times that got_picture returned zero.

[mpenkov]

Thank you for the detailed write-up! This definitely needs to be fixed and documented. Are you capable of fixing it yourself? If yes, I'll leave it up to you and wait for a pull request. Otherwise, I'll get around to it, hopefully sometime soon.

[sobotka]

Actually I am curious about the most fail-safe version of the code.

We clearly need a secondary loop after the primary. The question is how to appropriately grab all the frames?

if (got_picture) {} else skippedFrames++;
[...]
for (int i = skippedFrames; i > 0; i--) {}

Can work, but seems kludgy to me.

I suspect it might be more appropriate to perform a while loop something like...

while (retValue = avcodec_decode_video2(pCodecCtx, pFrame, got_picture, &packet), ((retValue >= 0) && (retValue != AVERROR_EOF) && (got_frame)) ) {}

Opinions?

[mpenkov]

Interestingly, here's how ffplay handles it:

if(avcodec_decode_video2(is->video_st->codec, frame, &got_picture, pkt)

< 0) return 0;

if (got_picture) {
    int ret = 1;
    ...
    return ret;
}

It doesn't seem to bother with it. I wonder why? Perhaps it may be good to ask this on the ffmpeg-users mailing list.

On 20 November 2012 01:53, Troy James Sobotka [email protected]:

Actually I am curious about the most fail-safe version of the code.

We clearly need a secondary loop after the primary. The question is how to appropriately grab all the frames?

if (got_picture) {} else skippedFrames++; [...] for (int i = skippedFrames; i > 0; i--) {}

Can work, but seems kludgy to me.

I suspect it might be more appropriate to perform a while loop something like...

while (retValue = avcodec_decode_video2(pCodecCtx, pFrame, got_picture, &packet), ((retValue >= 0) && (retValue != AVERROR_EOF) && (got_frame)) ) {}

Opinions?

— Reply to this email directly or view it on GitHubhttps://github.com/mpenkov/ffmpeg-tutorial/issues/7#issuecomment-10521164.

[sobotka]

My local tests here bear the issue out.

Perhaps it hasn't been noticed? If you have any content with B frames, I'd suggest a test at your end. Overall frame counts (both via nb_frame and ceil(m_pFormatCtx->duration * av_q2d(m_pFormatCtx->streams[m_stream]->r_frame_rate) / AV_TIME_BASE) estimations) should result in discrepancies between an incremented counter in the av_read_frame loop.

[mpenkov]

Could you throw this at the ffmpeg-user mailing list? You'll definitely get a definitive answer there.

I can ask them myself, but you seem to have a much better understanding of the issue than I do.

On 20 November 2012 10:03, Troy James Sobotka [email protected]:

My local tests here bear the issue out.

Perhaps it hasn't been noticed? If you have any content with B frames, I'd suggest a test at your end. Overall frame counts (both via nb_frame and ceil(m_pFormatCtx->duration * av_q2d(m_pFormatCtx->streams[m_stream]->r_frame_rate) / AV_TIME_BASE) estimations) should result in discrepancies between an incremented counter in the av_read_frame loop.

— Reply to this email directly or view it on GitHubhttps://github.com/mpenkov/ffmpeg-tutorial/issues/7#issuecomment-10538765.