DASH Clipping mp4 breaks AV sync

[teplicko]

Hello,

I tried your module for DASH streaming with clipping input from MP4. I have vod_align_segments_to_key_frames on, because I use shaka player and it doesn't work with non-I-frame at the beginning.

I have video with I-frame every 2 seconds. When I use clipFrom, it seems like audio starts from exact clipFrom position, but video starts correctly from first I-frame position. I tried different clipFrom in 2s range and offset is exactly the time difference to the first key frame. http://127.0.0.1/dash/clipFrom/228000/,test,.mp4.urlset/manifest.mpd - perfect (even seconds are key-frames) http://127.0.0.1/dash/clipFrom/228500/,test,.mp4.urlset/manifest.mpd - 1.5s audio delay http://127.0.0.1/dash/clipFrom/229000/,test,.mp4.urlset/manifest.mpd - 1s audio delay http://127.0.0.1/dash/clipFrom/229500/,test,.mp4.urlset/manifest.mpd - 0.5s audio delay http://127.0.0.1/dash/clipFrom/230000/,test,.mp4.urlset/manifest.mpd - perfect again

I may do something wrong, because I'd suspect this kind of bug would be revealed long time ago. I checked, if there is any initial pts/dts offset, and the video doesn't have any (checked by ffprobe).

# for s in a v; do echo -n "$s: "; ffprobe -i /var/www/test.mp4 -show_frames -select_streams $s 2>/dev/null | grep pts_time | head -1; done
a: pkt_pts_time=0.000000
v: pkt_pts_time=0.040000

I'll send you examples with links (of my config, used url and test files) to your e-mail soon.

[erankor]

Yes, I'm familiar with this issue. We are using 2 second GOP as well, and we're making sure to send only clipFrom values which are multiples of 2 seconds. This happens because the module loads the minimum amount of data required to serve the request. So, when it gets a request for an audio segment, it doesn't even parse the video frames. And therefore, it can't align the audio according to the video frames.

[teplicko]

Well, will the issue be fixed soon/sometime/ever? I'd use your workaround, but I don't have all videos with the 2s GOP.

[erankor]

No plans to change this behavior any time soon

[teplicko]

Well, I'll leave the issue opened. I guess, if somebody will find the same problem, they'd like to know it's known.

[mdale]

Hitting this issue as well. How is this handled differently with HLS vs DASH ? We don't see the same issue with HLS.

[erankor]

It's because the default in HLS is to return the audio + video muxed together, while in DASH they are always separate. If you enable vod_hls_force_unmuxed_segments, HLS will probably show the same out-of-sync as DASH.

[mdale]

This happens because the module loads the minimum amount of data required to serve the request. So, when it gets a request for an audio segment, it doesn't even parse the video frames. And therefore, it can't align the audio according to the video frames.

Would it be possible for the module to load the video data to ensure the audio segment response is aligned based on the clipping points ? I guess you still end up with some audio samples miss-alignment relative to demuxed stream ... But would be much much better than the gap to the next keyframe.

[erankor]

Sorry, I'm not going to change this behavior.

[zylux1]

Hi, any update on this thread? I have the same problem and it occours even in hls with vod_hls_force_unmuxed_segments enabled. Thanks