Timestamps incorrect on videos downloaded from playback server

[bog]

Which version are you using?

v1.5.0

Which operating system are you using?

• [ ] Linux amd64 standard
• [x] Linux amd64 Docker
• [ ] Linux arm64 standard
• [ ] Linux arm64 Docker
• [ ] Linux arm7 standard
• [ ] Linux arm7 Docker
• [ ] Linux arm6 standard
• [ ] Linux arm6 Docker
• [ ] Windows amd64 standard
• [ ] Windows amd64 Docker (WSL backend)
• [ ] macOS amd64 standard
• [ ] macOS amd64 Docker
• [ ] Other (please describe)

Describe the issue

Videos downloaded from playback server have incorrect timestamps

Describe how to replicate the issue

1. start the server with recordings enabled
2. publish an SRT stream (haven't tested other stream types)
3. download a video with curl (I downloaded a 10s sample as sample.mp4)
4. inspect the downloaded video with ffprobe: ffprobe -show_frames -show_entries frame=key_frame,pts,pkt_dts,best_effort_timestamp sample.mp4

Example output:

[FRAME]
key_frame=1
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=2946
best_effort_timestamp=2946
[/FRAME]
[FRAME]
key_frame=0
pts=2946
pkt_dts=14946
best_effort_timestamp=14946
[/FRAME]
[FRAME]
key_frame=0
pts=5946
pkt_dts=17945
best_effort_timestamp=17945
[/FRAME]
[FRAME]
key_frame=0
pts=17946
pkt_dts=20944
best_effort_timestamp=20944

Notice that pts remains the same for the first 26 frames and pkt_dts and best_effort_timestamp for the first 25. After that, timestamps appear to increase as expected. I would expect timestamps to increase at normal intervals from the start of the file. I have verified that the content of the frames appears correct, but the timestamps are not. I have noticed that the number of frames that report the same timestamp varies with different time ranges.

If you need sample videos or streams, I can provide.

Did you attach the server logs?

no (I saw no useful logs in the server)

Did you attach a network dump?

no, that seems less helpful here

Lastly, thanks for this amazing project.

[aler9]

Hello,

Notice that pts remains the same for the first 26 frames and pkt_dts and best_effort_timestamp for the first 25

Yes, your analysis is perfectly right, but this behavior is intentional. In most video codecs, a frame can be decoded only after a previous key frame has been decoded, therefore, when you request a frame corresponding to a certain absolute time, and this frame is not a key frame, the server searches for a previous key frame and prepends it to the file, with a duration equal to zero, together with all frames in between that and the desired frame, in order to decode and hide them instantly:

[key frame] [non-key frame] [non-key frame] [non-key frame] [non-key frame] .... [non-key frame, desired]

I've been looking for a better solution in order to avoid sending frames with a duration equal to zero, but for the moment i found none.

I'm certain that the key frame has to be sent to clients, but i agree with you that using a duration equal to zero may cause issues with some clients.

[bog]

Ahh, that makes sense (and part of why I included whether a frame was a key frame). I was expecting the timestamps to start incrementing when a key frame was found, but that was not it. It includes a key frame and doesn't increment timestamps until it's the time requested. Clever.

As you stated, I think the issue is that the frames still have pkt_duration=3000 and I'm not sure of the effect of setting that to zero.

Regardless, for our purposes, the previous keyframe would be an acceptable place to start from. Would you consider adding an option to request the video with a start time floored to the previous keyframe as the start (but with duration from the requested start time)?

[aler9]

I think the issue is that the frames still have pkt_duration=3000 and I'm not sure of the effect of setting that to zero.

in MP4 there are no separate fields for pkt_duration and pkt_dts, there is only duration - it's FFmpeg that has both fields, and DTS is computed automatically by summing durations of all previous frames. Therefore it's impossible to control them separately.

[bog]

Hi. I just noticed I have some other videos that start with negative pts. By some searching, it appears that's how other software handles this issue. That is, rather than giving all frames before the start time a pts of 0, it may be more correct for them to have a relatively correct pts that goes into the negative.