hls.js icon indicating copy to clipboard operation
hls.js copied to clipboard
verified publisher icon video-dev

Level Controller - Captions Not Available

Open wildg-ensemble opened this issue 1 year ago • 6 comments

What version of Hls.js are you using?

1.5.19

What browser (including version) are you using?

Version 131.0.6778.266 (Official Build) (arm64)

What OS (including version) are you using?

Mac OS Ventura (13.5)

Test stream

No response

Configuration

{
  debug: true,
  autoStartLoad: false,
  maxMaxBufferLength: 10,
  capLevelToPlayerSize: true,
  ignoreDevicePixelRatio: true,
  capLevelOnFPSDrop: true,
  backBufferLength: 2,
  liveSyncDurationCount: 3,
  liveMaxLatencyDurationCount: 10,
}

Additional player setup steps

this.hls = new Hls(options);
this.hls.loadSource(url);
this.hls.attachMedia(this.videoElement);

Checklist

  • [x] The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
  • [x] The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
  • [x] The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
  • [x] The stream has correct Access-Control-Allow-Origin headers (CORS)
  • [x] There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

Subtitle tracks are missing for playlists with captions.

HLS.js parses video playlists in m3u8-parser.parseMasterPlaylistMedia(). This creates separate CLOSED_CAPTIONS and SUBTITLES properties as part of the ManifestLoadedData payload. Then, the level-controller.filterAndSortMediaOptions() parses the payload data, but only checks if data.subtitles is available, not data.captions. Thus, I suggest adding the following:

// level-controller.ts, line 280
if (data.subtitles) {
  subtitleTracks = data.subtitles;
  assignTrackIdsByGroup(subtitleTracks);
} else if (data.captions) {
  subtitleTracks = data.captions;
  assignTrackIdsByGroup(subtitleTracks);
}

I believe this is a viable solution, but I'm not sure if there are side-effects I might not be considering. While I cannot provide a stream URL directly, here's some sample playlist data:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:TYPE=CLOSED-CAPTIONS,GROUP-ID="CC",LANGUAGE="eng",NAME="English",INSTREAM-ID="CC1"
#EXT-X-STREAM-INF:BANDWIDTH=4909291,AVERAGE-BANDWIDTH=4725600,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=1920x1080,FRAME-RATE=29.970,CLOSED-CAPTIONS="CC"
playlist_1920x1080.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3193687,AVERAGE-BANDWIDTH=3075600,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1280x720,FRAME-RATE=29.970,CLOSED-CAPTIONS="CC"
playlist_1280x720.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1363709,AVERAGE-BANDWIDTH=1315600,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1024x576,FRAME-RATE=29.970,CLOSED-CAPTIONS="CC"
playlist_1024x576.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=791841,AVERAGE-BANDWIDTH=765600,CODECS="avc1.64001e,mp4a.40.2",RESOLUTION=640x360,FRAME-RATE=29.970,CLOSED-CAPTIONS="CC"
playlist_640x360.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=448720,AVERAGE-BANDWIDTH=435600,CODECS="avc1.64000d,mp4a.40.2",RESOLUTION=384x216,FRAME-RATE=29.970,CLOSED-CAPTIONS="CC"
playlist_384x216.m3u8

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:7
#EXT-X-MEDIA-SEQUENCE:152154
#EXT-X-DISCONTINUITY-SEQUENCE:1567
#EXT-X-PROGRAM-DATE-TIME:2025-01-21T23:42:50.512Z
#EXTINF:6.00600,
playlist_1920x1080_234251__152154.ts
#EXTINF:6.00600,
playlist_1920x1080_234257__152155.ts
#EXTINF:6.00600,
playlist_1920x1080_234303__152156.ts
#EXTINF:6.00600,
playlist_1920x1080_234309__152157.ts
#EXTINF:6.00600,
playlist_1920x1080_234315__152158.ts
#EXTINF:6.00600,
playlist_1920x1080_234321__152159.ts
#EXTINF:6.00600,
playlist_1920x1080_234327__152160.ts
#EXTINF:6.00600,
playlist_1920x1080_234333__152161.ts
#EXTINF:6.00600,
playlist_1920x1080_234339__152162.ts
#EXTINF:6.00600,
playlist_1920x1080_234345__152163.ts

Expected behaviour

  1. Play a live stream playlist with CLOSED_CAPTIONS subtitles
  2. this.hls.subtitleTracks has the text tracks available

What actually happened?

  1. Play a live stream playlist with CLOSED_CAPTIONS subtitles
  2. this.hls.subtitleTracks has no text tracks available

Console output

HlsPlayerInstance.ts:20 [log] > Debug logs enabled for "Hls instance" in hls.js version 1.5.19
HlsPlayerInstance.ts:50 [log] > stopLoad
HlsPlayerInstance.ts:50 [log] > loadSource:<URL>
HlsPlayerInstance.ts:50 [log] > [stream-controller]: Trigger BUFFER_RESET
HlsPlayerInstance.ts:50 [log] > resume buffering
HlsPlayerInstance.ts:51 [log] > attachMedia
HlsPlayerInstance.ts:51 [log] > [buffer-controller] created media source: MediaSource
hls.mjs:174 [log] > [buffer-controller] Media source opened
hls.mjs:174 [log] > [level-controller]: manifest loaded, 5 level(s) found, first bitrate: 4909291
hls.mjs:174 [log] > setting initial bwe to 4909291
hls.mjs:174 [log] > [buffer-controller] 1 bufferCodec event(s) expected
hls.mjs:174 [log] > Setting autoLevelCapping to 1: 360p@791841 for media 562x313
hls.mjs:174 [log] > set autoLevelCapping:1
HlsPlayerInstance.ts:33 [log] > startLoad(0)
HlsPlayerInstance.ts:33 [log] > resume buffering
HlsPlayerInstance.ts:33 [log] > [abr] picked start tier {"codecSet":"avc1,mp4a","videoRanges":["SDR"],"preferHDR":false,"minFramerate":29.97,"minBitrate":448720}
HlsPlayerInstance.ts:33 [info] > [abr] switch candidate:4->1 adjustedbw(4909291)-bitrate=4117450 ttfb:0.1 avgDuration:0.0 maxFetchDuration:4.0 fetchDuration:0.1 firstSelection:true codecSet:avc1,mp4a videoRange:SDR hls.loadLevel:-1
HlsPlayerInstance.ts:33 [log] > [level-controller]: Switching to level 1 (360p SDR avc1,mp4a @791841) from level -1
HlsPlayerInstance.ts:33 [log] > [level-controller]: Loading level index 1 with <URL>
HlsPlayerInstance.ts:33 [log] > [stream-controller]: STOPPED->IDLE
HlsPlayerInstance.ts:33 [log] > [subtitle-stream-controller]: STOPPED->IDLE
hls.mjs:174 [log] > [level-controller]: reload live playlist 1 in 6957 ms
hls.mjs:174 [log] > [stream-controller]: Level 1 loaded [152582,152591][part-152591--1], cc [1571, 1571] duration:60.06
hls.mjs:174 [log] > [stream-controller]: Live playlist sliding: 0.00 start-sn: na->152582 prev-sn: na fragments: 10
hls.mjs:174 [log] > [buffer-controller] Updating Media Source duration to 60.060
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152582 cc: 1571 of [152582-152591] level: 1, target: 0
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [transmuxer-interface, main]: Starting new transmux session for sn: 152582 p: -1 level: 1 id: 1
        discontinuity: true
        trackSwitch: true
        contiguous: false
        accurateTimeOffset: false
        timeOffset: 0
        initSegmentChange: true
hls.mjs:174 [log] > [mp4-remuxer]: ISGenerated flag reset
hls.mjs:174 [log] > [mp4-remuxer]: initPTS & initDTS reset
hls.mjs:174 [log] > [mp4-remuxer]: reset next timestamp
hls.mjs:174 [log] > manifest codec:mp4a.40.2, ADTS type:2, samplingIndex:3
hls.mjs:174 [log] > parsed codec:mp4a.40.2, rate:48000, channels:2
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Init audio buffer, container:audio/mp4, codecs[selected/level/parsed]=[mp4a.40.2/mp4a.40.2/mp4a.40.2]
hls.mjs:174 [log] > [stream-controller]: Init video buffer, container:video/mp4, codecs[level/parsed]=[avc1.64001e/avc1.64001e]
hls.mjs:174 [log] > [buffer-controller] 0 bufferCodec event(s) expected audio,video
hls.mjs:174 [log] > [buffer-controller] creating sourceBuffer(audio/mp4;codecs=mp4a.40.2)
hls.mjs:174 [log] > [buffer-controller] creating sourceBuffer(video/mp4;codecs=avc1.64001e)
hls.mjs:174 [log] > [audio-stream-controller]: InitPTS for cc: 1571 found from main: 4137200712
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152582 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152582 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152582 of level 1 (frag:[0.000-6.006] > buffer:[0.000-5.995])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152583 cc: 1571 of [152582-152591] level: 1, target: 5.995
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152583 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152583 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152583 of level 1 (frag:[5.995-12.012] > buffer:[0.000-12.011])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152584 cc: 1571 of [152582-152591] level: 1, target: 12.011
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152584 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152584 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152584 of level 1 (frag:[12.011-18.018] > buffer:[0.000-18.005])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152585 cc: 1571 of [152582-152591] level: 1, target: 18.005
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152585 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152585 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152585 of level 1 (frag:[18.005-24.024] > buffer:[0.000-24.021])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152586 cc: 1571 of [152582-152591] level: 1, target: 24.021
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152586 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152586 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152586 of level 1 (frag:[24.021-30.030] > buffer:[0.000-30.016])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152587 cc: 1571 of [152582-152591] level: 1, target: 30.016
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152587 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152587 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152587 of level 1 (frag:[30.016-36.036] > buffer:[0.000-36.032])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152588 cc: 1571 of [152582-152591] level: 1, target: 36.032
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152588 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152588 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152588 of level 1 (frag:[36.032-42.042] > buffer:[0.000-42.027])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152589 cc: 1571 of [152582-152591] level: 1, target: 42.027
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152589 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152589 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152589 of level 1 (frag:[42.027-48.048] > buffer:[0.000-48.043])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152590 cc: 1571 of [152582-152591] level: 1, target: 48.043
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152590 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152590 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152590 of level 1 (frag:[48.043-54.054] > buffer:[0.000-54.037])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE
hls.mjs:174 [log] > [stream-controller]: Loading fragment 152591 cc: 1571 of [152582-152591] level: 1, target: 54.037
hls.mjs:174 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.mjs:174 [log] > [stream-controller]: FRAG_LOADING->PARSING
hls.mjs:174 [log] > [stream-controller]: Loaded fragment 152591 of level 1
hls.mjs:174 [log] > [transmuxer.ts]: Flushed fragment 152591 of level 1
hls.mjs:174 [log] > [stream-controller]: PARSING->PARSED
hls.mjs:174 [log] > [stream-controller]: Buffered main sn: 152591 of level 1 (frag:[54.037-60.060] > buffer:[0.000-60.053])
hls.mjs:174 [log] > [stream-controller]: PARSED->IDLE

wildg-ensemble avatar Jan 22 '25 00:01 wildg-ensemble

Hls.js does not expose 608 captions as HLS subtitle tracks. It does however add a TextTrack of kind “captions” to the media element so that they can be selected.

robwalch avatar Jan 22 '25 02:01 robwalch

Hi @robwalch, thank you for your response! Apologies for my lack of knowledge on this subject, but why can't 608 captions be exposed via HLS subtitle tracks? Asking for my own understanding, I don't doubt there is a valid reason for this

Adding captions as media element TextTracks seems to cause issues with the expired tracks. After reading other issue posts, I know we can't remove text tracks once they are added to a video element, and that the suggested workaround is to tear the player down. However, destroying and creating a new player would be extremely difficult to do with the framework I am using

wildg-ensemble avatar Jan 22 '25 20:01 wildg-ensemble

why can't 608 captions be exposed via HLS subtitle tracks? Asking for my own understanding, I don't doubt there is a valid reason for this

hls.subtitleTracks is for managing which HLS subtitle playlists and segments are loaded by HLS.js. The modules that manage these API endpoints are only concerned with HLS subtitle playlists and segments.

The subtitle tracks are known when the MVP is parsed, but the captions may not be unless signaled with #EXT-X-MEDIA:TYPE=CLOSED-CAPTIONS as in your example above (this is because the captions are carried in the media segments). Having the subtitle-track-controller work with the timeline-controller to also manage captions would be a new feature and require additional API. Such a change could impact current applications that use HLS.js v1.x.

Adding captions as media element TextTracks seems to cause issues with the expired tracks. After reading other issue posts, I know we can't remove text tracks once they are added to a video element, and that the suggested workaround is to tear the player down. However, destroying and creating a new player would be extremely difficult to do with the framework I am using

TextTracks are added for captions, subtitles, and metadata. Setting a textTrack mode to "show" is the preferred way of selecting the active track. This aligns with the browser's media controls and native HLS playback in Safari.

You can disable the use of TextTracks completely by setting renderTextTracksNatively to false in the config. In this case you need to manage parsed 608 cues and subtitle segments and the tracks that they belong to.

robwalch avatar Jan 22 '25 20:01 robwalch

Thus, I suggest adding the following

Calling assignTrackIdsByGroup on captions would not add a captions subtitle track. The grouping is irrelevant as the captions are in the media, always parsed with the media, and only enabled or disabled rather than selected like subtitle tracks which require separate loading and parsing. The solution is not so simple. Adding captions to subtitle tracks requires and feature request and would be a v2 change.

robwalch avatar Jan 22 '25 20:01 robwalch

Thank you for your detailed explanation @robwalch! This makes more sense to me now, captions being an extension of media while subtitle tracks require separate loading and managing. I still want to leverage HLS' text track rendering, so I was trying to avoid setting renderTextTracksNatively: false, but I may need to consider going that route for now

Is it alright if I add a feature request to manage captions & subtitles together in v2?

wildg-ensemble avatar Jan 22 '25 21:01 wildg-ensemble

Is it alright if I add a feature request to manage captions & subtitles together in v2?

Absolutely. Please do.

robwalch avatar Jan 23 '25 01:01 robwalch