IMSC/TTML support

[ArthurKnoep]

Description

Hi guys,

At the moment only WebVTT subtitles are supported for both DASH and HLS media source. However WebVTT is a very limited file format in terms of functionality, that is why a new standard for the web tends to emerge: IMSC (Internet Media Subtitles and Captions). To explain very briefly the concept, IMSC is pretty much HTML but with time tags so that only some XML elements appear based on time. This allows for custom text styling, custom text placement and pretty much anything you can do with basic HTML/CSS. (There is a demo on the MDN for those interested: https://developer.mozilla.org/en-US/docs/Related/IMSC)

IMSC can be presented under two distinct format:

• it can either be a text file under the following content-type: application/ttml+xml
• or it can be segmented as an fMP4 file where the xml content is stored under an ISOBMFF encapsulation using the following content-type: application/mp4

DASH has already official support for it, per the section 6.4.4 of the specification: https://dashif.org/docs/DASH-IF-IOP-v4.3.pdf. For HLS, Apple has announced support for it in 2017 for their hardware, however the updated RFC is still in draft: https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis/

For a personal project of mine I need to playback video files with IMSC subtitles, my videos are available both with HLS and DASH (for DRM purpose, Safari only supports HLS for DRMs). My initial plan was to create a custom plugin that would somehow extend VHS so that it can handle IMSC in addition to WebVTT. However it appeared to me that subtitles handling was too much intricate within VHS code base to be expanded by a VideoJS plugin. At the moment, the only solution I have is to remove subtitles from my HLS and DASH manifest, and add them as external text track for VideoJS. Then, I use a plugin I wrote that provides a custom source handler so that it can play them. This is not an optimal behavior as it requires to add another endpoint to my API to list the available subtitles and adds additional latency and frictions for initializing a playback.

My question is the following, would there be any motivation for having support for IMSC subtitles within VHS ?

Note that I could be up for the development but as I'm not familiar with VHS this will probably take me a lot of time without guidance and also I don't want to work on something that will not be accepted.

Sources

I can't share a source publicly available as I'm only conducting tests on my local laptop at the moment, however I can share a DASH AdaptationSet that loads an IMSC subtitle (the IMSC subtitle in question can be found here: basic-expanded.ttml)

...
<AdaptationSet id="2" contentType="text" mimeType="application/ttml+xml" lang="eng">
	<Role schemeIdUri="urn:mpeg:dash:role:2011" value="subtitle"/>
	<Representation id="2" bandwidth="1000">
		<BaseURL>basic-expanded.ttml</BaseURL>
	</Representation>
</AdaptationSet>
...

Results

Expected

When playing a DASH or HLS playlist with IMSC subtitles, I expect to see my subtitles displayed on the player.

Error output

Subtitles are handled by vttjs, so it does nothing with this warning message: video.js:210 VIDEOJS: WARN: Error encountered when parsing cues: Malformed WebVTT signature.

videojs-http-streaming version

videojs-http-streaming 3.6.0

videojs version

video.js 8.6.0

Browsers

• all

Platforms

• all

Other Plugins

none

Other JavaScript

none