Regression Bug: Playback does not Start in 1.0.x Versions with A/V Offset in First Segment

[softworkz]

What version of Hls.js are you using?

1.0.5

What browser and OS (including versions) are you using?

Win 10, Chrome latest

Test stream:

https://softworkz.github.io/hlstestfiles/regression/A276B7.m3u8

Checklist

• [x] The issue occurs in the stable client on https://hls-js.netlify.com/demo and not just on my page
• [x] The issue occurs in the latest client on https://hls-js-dev.netlify.com/demo and not just on my page
• [x] The stream has correct Access-Control-Allow-Origin headers (CORS)
• [x] There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

Current Version

• Paste the test stream (https://softworkz.github.io/hlstestfiles/regression/A276B7.m3u8) into latest demo (https://hls-js.netlify.com/demo)
• Press Play
• Hangs forever

Previous Versions (0.14.x)

• Repeat the same with v0.14.17 demo
• Plays without issues

[robwalch]

Here's the part of the gap-controller that jumps start gaps. It's limited to jumping 2 second VOD start gaps in v1.0.

export const MAX_START_GAP_JUMP = 2.0;

    // Skip start gaps if we haven't played, but the last poll detected the start of a stall
    // The addition poll gives the browser a chance to jump the gap for us
    if (!this.moved && this.stalled !== null) {
      // Jump start gaps within jump threshold
      const startJump =
        Math.max(nextStart, bufferInfo.start || 0) - currentTime;

      // When joining a live stream with audio tracks, account for live playlist window sliding by allowing
      // a larger jump over start gaps caused by the audio-stream-controller buffering a start fragment
      // that begins over 1 target duration after the video start position.
      const level = this.hls.levels
        ? this.hls.levels[this.hls.currentLevel]
        : null;
      const isLive = level?.details?.live;
      const maxStartGapJump = isLive
        ? level!.details!.targetduration * 2
        : MAX_START_GAP_JUMP;
      if (startJump > 0 && startJump <= maxStartGapJump) {
        this._trySkipBufferHole(null);
        return;
      }
    }

[robwalch]

Relates to #4064 which has a short enough start gap for hls.js to call _trySkipBufferHole, but will not autoplay because the gap is larger than the browser will skip on its own and we do not seek over gaps until after buffered and play is called.

[softworkz]

The point is, that there is no gap that needs to be jumped over. It's just that the audio starts a few hundred milliseconds before the video in the first segment. But at the cut between end of first and start of second segment, both, video and audio continue without a gap (just with an offset between audio and video).

To me it rather seems that the gap calculation is incorrect in some way, could that be possible?

[softworkz]

BTW, we have experienced this issue double times from two different and separate development processes: once for the new low latency live stream task and at the same time for the VOD-type streaming that we always had and where we were forced to revert all 1.0 upgrades and go back to 0.14.

[robwalch]

Not a Contribution

Both 0.14.x and 1.x have the video buffer appends beginning at 2.87s while audio starts at 0s. Older versions jump this gap. Newer releases do not for the reasons stated above. You can see this in the Timeline on the demo page or by querying video.buffered.start(0). The total duration is 60s in all versions of HLS.js.

Native playback in Safari by comparison has the buffer starting at 0 with a total duration of 59s.

This along with the log errors related to 608 captions makes me suspect that the root cause for this issue has something to do with the ts-demuxer and/or mp4-remuxer fumbling on certain TS packets in the media.

[softworkz]

Hi Rob, thanks for re-visiting!

Not a Contribution

I'm not sure what that is supposed to tell...?

Both 0.14.x and 1.x have the video buffer appends beginning at 2.87s while audio starts at 0s. Older versions jump this gap. Newer releases do not for the reasons stated above. You can see this in the Timeline on the demo page or by querying video.buffered.start(0). The total duration is 60s in all versions of HLS.js.

Native playback in Safari by comparison has the buffer starting at 0 with a total duration of 59s.

This along with the log errors related to 608 captions makes me suspect that the root cause for this issue has something to do with the ts-demuxer and/or mp4-remuxer fumbling on certain TS packets in the media.

What do you suggest us to do about it? It is blocking us from updating to 1.x.

Is there any way to achieve the 0.14 behavior with 1,x in this specific regard?

[softworkz]

I should add that this is not about a single/specific broken/faulty media file. I wouldn't have raised this issue in that case. We had upgraded our beta to 1.0.5 and got reports from many users that playback is no longer working. The common denominator of those issues was a large start offset between audio and video.

[robwalch]

Is there any way to achieve the 0.14 behavior in this specific regard?

You can seek to video.buffered.start(0) on "play" or increase MAX_START_GAP_JUMP if you are building from source.

I checked the demuxer. The audio samples PTS starts at 900000 and video PTS at 1158490. So as best as I can tell, we are honoring the timecode found in the media. The audio track is MP3 audio. I don't know if the first seconds of MP3 samples are priming data that is OK to drop. Unfortunately I don't see a clear path to correcting this from the muxer.

[robwalch]

I should add that this is not about a single/specific broken/faulty media file.

Can you share any information about how the media is encoded (what service or tools are used to encode the mp4 or package HLS)?

[softworkz]

I don't know if the first seconds of MP3 samples are priming data that is OK to drop

If that helps, then I'd say yes.

I think the reason why that happens is that the source is often TV recordings. Those recordings start at an arbitrary point in time. To satisfy the requirement that HLS segments need to start with a (video) keyframe, all video frames are dropped until the next key frame, while audio samples are taken immediately, which causes the situation that there can be a sequence of audio without video in the first segment until video frames are starting (with a key frame).

You can see this with hardware TVs when you switch channels: often you get audio before you can see a picture.

[softworkz]

Can you share any information about how the media is encoded (what service or tools are used to encode the mp4 or package HLS)?

Usually it's ffmpeg where we are using the segment muxer to create the HLS segments.

[robwalch]

Related to #5099

[robwalch]

With #5816 v1.5 jumps large start gaps. The player won't autostart as that would require a seek. It will however seek on play:

• #5816

[softworkz]

With #5816 v1.5 jumps large start gaps. The player won't autostart as that would require a seek. It will however seek on play:

• Skip large video start gaps using GAP flag #5816

Thanks @robwalch, that's awesome, just last week we got the latest user report, due to this. I'll test and report back shortly.