cisco
Output frames from openh264 in chronological order by DecodeFrameNoDelay
Openh264 does NOT always output video frames in chronological order when calling by DecodeFrameNoDelay()
The output PTSs are as below; some of them are out-of-order. Should I reorder myself? If so, is there a sample code?
0 0
1 33333
2 66666
3 100000
4 133333
5 166666
6 200000
7 266666 *
8 233333 #
9 300000
10 333333
11 400000 *
12 366666 #
13 433333
14 466666
15 533333 *
16 500000 #
17 566666
18 600000
19 666666 *
20 633333 #
21 700000
22 733333
23 800000 *
24 766666 #
25 833333
26 900000 *
27 866666 #
28 933333
test file can be accessed here: https://drive.google.com/file/d/1B2-C-UL1QgMY8MRPec1mRgxQSKoO4yKq/view?usp=share_link
Openh264 decoder supports real-time decoding, so the output order might be the same as decoding order. @xiaotiansf Is that also true for B frame decoding?
That's not correct. If there is B-frame, the output order (which is called presentation order) will be different from decoding order. For example, the decoding order is I B B P, the output order has to be B B I ...
Thanks. So in conclusion, the output order may be different when B frames are involved. @WeiChungChang
@huili2 @xiaotiansf Thanks to the info.
So the conclusion is when I use DecodeFrameNoDelay() to decode movie with B frames, I need to implement a logic to reoder the decoded frames in chronological order?
If so, I had the simple implementation below(in pseudo code):
std::priority_queue<decoded_buffer> pq;
...
SBufferInfo buffer_info;
decoder_->DecodeFrameNoDelay(..., buffer_info);
if (buffer_info.iBufferStatus == 1) {
Receive current_frame;
if (!pq.empty() && pq.top().timestamp < current_frame's timestamp) {
output pq.top();
}
pq.push(current_frame);
}
I skip the logic in flush.
In this logic, when receive a new decoded frame it will not output it immediately. Instead, it inspect if
- the priority queue is not.
- the first(in priority queue, the buffer with smallest PTS(oldest)) is smaller than current frame's PTS.
If both 1 & 2 are ture, output the first item of priority queue.
For example, to
6 200000 (1st) -> pq = [200000]
7 266666 (2nd) -> output 200000, push 266666, pq = [266666]
8 233333 (3rd) -> no output, push 233333, pq = [233333, 2666666]
9 300000 (4th) -> output 2333333, pq = [2666666, 300000].
....
It output 200000, 233333, 266666, 300000... in chronological order.
[Q1]: could you please help to comment if this method covers openh264 DecodeFrameNoDelay() B frame handling logic and can reorder the decoded frames in chronological order?
[Q2]: if so, if it is possible (and suitable) to integrate the logic to help deocder always outputs frames in chronological order as most of the modern decoders?
Thanks
A1: Your logic is correct as long as you know "current_frame's timestamp", pts. A2. openh264 has already the mechanism to reorder frames in presentation order in welsDecoderExt.cpp (check it). The logic to reorder is not based on pts since decoder does not have pts available .
@xiaotiansf
As A2, but I still get the output not in presentation order. Does it mean I miss some settings? (I have checked but found nothing). It is why currently I rely on the logic myself.
I lieted the the sequence when sending to decoder and recerve from decoder below, F.Y.R.
Sent to decoder.
D1. 0
D2. 100000
D3. 33333
D4. 66666
D5. 233333
D6. 133333
D7. 166666
D8. 200000
...
Receive from decoder.
R0 0 = D1
R1 33333 = D3
R2 66666 = D4
R3 100000 = D2
R4 133333 = D6
R5 166666 = D7
R6 200000 = D8
R7 266666 * =
R8 233333 # = D5
@xiaotiansf
The detail information is as below, thanks~
Source movie: https://drive.google.com/file/d/1B2-C-UL1QgMY8MRPec1mRgxQSKoO4yKq/view?usp=share_link
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_time_base=1/60
codec_tag_string=avc1
codec_tag=0x31637661
width=1920
height=1080
coded_width=1920
coded_height=1088
closed_captions=0
has_b_frames=1
sample_aspect_ratio=1:1
display_aspect_ratio=16:9
pix_fmt=yuv420p
level=40
color_range=tv
color_space=bt709
color_transfer=bt709
color_primaries=bt709
chroma_location=left
field_order=progressive
timecode=N/A
refs=1
is_avc=true
nal_length_size=4
id=N/A
r_frame_rate=30/1
avg_frame_rate=30/1
time_base=1/15360
start_pts=0
start_time=0.000000
duration_ts=4688896
duration=305.266667
bit_rate=46367
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=1
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
TAG:creation_time=2022-07-26T21:54:09.000000Z
TAG:language=und
TAG:handler_name=ISO Media file produced by Google Inc.
[/STREAM]
@xiaotiansf
Source movie: https://drive.google.com/file/d/1B2-C-UL1QgMY8MRPec1mRgxQSKoO4yKq/view?usp=share_link
Are you able to access it? If not so, I can put it in other place and try again, thank you!
Yes. Are you sure that you send the pictures in decoding order to the decoder. I am seeing the following pictures in decoder order: fprobe ~/Downloads/videoplayback.mp4 -select_streams v -show_entries frame=coded_picture_number,pkt_dts_time,pict_type -of csv=p=0:nk=1 -v 0 | more 0.000000,I,0 0.033333,B,2 0.066667,B,3 0.100000,P,1 0.133333,B,5 0.166667,B,6 0.200000,B,7 0.233333,P,4 0.266667,B,9 0.300000,B,10 0.333333,B,11 0.366667,P,8
Yes. I have noticed that video output from the decoder is not pts order somehow. So it is a bug in openh264.
I tested if you use threaded decoding by setting threadcount=2 or 3, the output video order is correct. But non-threaded decoding gets incorrect output video order. I will fix the issue for non-threaded decoding.
@xiaotiansf 👍
It is a good catch, please update the commit so I can also give it a try, thanks a lot for your fast and detailed replies.
Hi @xiaotiansf
The issue starts from these 2 commits, for you reference, thank you!
https://github.com/cisco/openh264/commit/c96de068fcd435579eeb3c63c132d8b8d182964c
https://github.com/cisco/openh264/commit/0819f288dc5fc24fb34089113bc230aea65c8375