Support SSA/ASS styling

[jakobkukla]

Add support for SSA/ASS styling. Currently there is only "minimal" support for ASS subtitles since styling was removed from the initial PR. (See https://github.com/google/ExoPlayer/pull/2281)

It would be great to have full support of the spec. Many subtitles use these styling features to work directly with the video (e.g. to substitute text in a video). SSA/ASS styling is also very common in certain genres of media like Cartoons, Anime or Karaoke.

[andrewlewis]

@icbaker Tentatively marking as an enhancement, but please dupe or close as needed. Thanks!

[icbaker]

Thanks for the request.

Unfortunately we don't currently have any capacity to improve ExoPlayer's SSA/ASS styling support. However we would welcome high quality PRs in this area.

I'm also happy to discuss various approaches with PR authors if a new feature doesn't fit easily into the existing code structure (either before the PR, or in the review, whichever is easier).

[13steinj]

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

(I'm not familiar with what exactly even is supported currently).

[icbaker]

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

That sounds like a good idea - the best way to do that is probably to compare the SsaDecoder to a copy of the spec. I've found the SSA/ASS spec to be fairly loose, and I'm not sure there's a canonical definition, but I've generally referred to this when working with these files in the past: http://moodub.free.fr/video/ass-specs.doc (linked from Matroska's page on SSA).

In case it's helpful, we would probably prioritise things roughly in this order:

• Features that look like they're implemented but don't work properly (basically bugs).
• Features that are similar to existing ones, and so can be implemented with small incremental changes.
• Features that don't fit into the existing code architecture - so implementing them requires significant refactoring.

I don't know if SSA has anything in the final category - but if it looks like it does then please come talk to us before spending ages refactoring SsaDecoder, as we might have some ideas about the best way to approach the problem.

[cosmo321]

I'm not a developer, so I can't exactly contribute a lot, but proper support of ass/ssa in Exoplayer is something I'd very much would like to have, as Plex for Android currently makes me sad... I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

[icbaker]

I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

Thanks for the pointer to libass, I took a quick look. I think you're possibly suggesting two alternative things:

• ExoPlayer could depend on libass and use it's logic for all the SSA rendering.
• People working on improving ExoPlayer's existing SSA rendering logic could use libass as a reference implementation to see how different pieces of content should be rendered, as an augmentation to the "spec".

In case anyone is considering trying the first of these, I'd like to pre-emptively warn against it. libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java - so it's extremely unlikely we'd accept a PR that introduced a native dependency to achieve this. There's more discussion about (not) using native deps for parsing subtitles in #7746 and #7749.

The second suggestion (use it as a reference implementation) makes perfect sense - and afaict it's used by VLC, so it should be enough to just play any content with VLC and compare to ExoPlayer's behaviour.

[cosmo321]

I leave that up to actual developers to decide. I just thought I'd point out that libass exists and is probably the best reference we have, and that rendering subs differently from libass could potentially make people like me wonder why the subs behave differently in ExoPlayer. The second alternative is probably the better option for ExoPlayer. :)

[szaboa]

hey @icbaker , I'll check what the current SsaDecoder supports and what is missing in regards of styling, based on the specs you linked above, then I could probably start and implement those step by step (with my limited time).

@jakobkukla did you have anything specific in mind? I mean is there a specific styling option which you would like to see first?

[icbaker]

Thanks for taking a look! Breaking down the work into small PRs definitely sounds good.

[jakobkukla]

@szaboa To be honest I'm not really aware what is currently supported and what not. But the ability to display fonts and colors properly would be nice!

[szaboa]

@icbaker I've checked out and it seems that only the v4+ "alignment" is supported. Here is the list of all the style attributes of v4 and v4+ from the specs. Do you see specific styles that would be very cumbersome to implement at first sight? E.g. thinking of "fontname", and "encoding"..

Style name	v4	v4+	Note	Format line support	Override support	PR
Name	✅	✅	-	✅	-	-
Fontname	✅	✅	-	❌	❌	-
Fontsize	✅	✅	-	✅	❌	#8615
PrimaryColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	✅	❌	#8490
SecondaryColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	❌	❌	-
OutlineColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR Called OutlineColour in v4+, and TertiaryColour in v4	❌	❌	-
BackColour	✅	✅	v4 BBGGRR, v4+ AABBGGRR	❌	❌	-
Bold	✅	✅	-	✅	❌	#8654
Italic	✅	✅	-	✅	❌	#8654
Underline	❌	✅	-	✅	❌	#8851
Strikeout	❌	✅	-	✅	❌	#8851
ScaleX	❌	✅	-	❌	❌	-
ScaleY	❌	✅	-	❌	❌	-
Spacing	❌	✅	-	❌	❌	-
Angle	❌	✅	-	❌	❌	-
BorderStyle	✅	✅	-	❌	❌	-
Outline	✅	✅	-	❌	❌	-
Shadow	✅	✅	-	❌	❌	-
BorderStyle	✅	✅	-	❌	❌	-
Alignment	✅	❌	values may be 1=Left, 2=Centered, 3=Right. Add 4 to the value for a "Toptitle". Add 8 to the value for a "Midtitle"	❌	❌	-
Alignment	❌	✅	layout of the numpad (1-3 sub, 4-6 mid, 7-9 top)	✅	✅	-
MarginL, MarginR, MarginV	✅	✅	-	❌	-	#10169
AlphaLevel	❌	✅	-	❌	❌	-
Encoding	✅	✅	-	❌	❌	-

My plan is to start implement some easy ones first :)

[icbaker]

The spec I linked above (https://github.com/google/ExoPlayer/issues/8435#issuecomment-756651245) seems to think that Underline, Strikeout, ScaleX, ScaleY, Spacing and Angle are supported in v4+. In fact are possibly only supported there and not in v4. Maybe those rows in your table are accidentally flipped? Or maybe I'm misreading the spec...

The SsaDecoder currently ignores v4 styling completely. It's not clear if that's sensible - I don't know how prevalent each version is in the wild.

If you're going to add v4 support we probably need to do think about whether we can reuse the SsaStyle class or split it into two. It's up to you, but for now imo it would be easiest to focus on v4+ only.

These all look like they should be relatively easy (I've referenced the relevant Android styling span where it seems obvious):

• Fontname (TypefaceSpan)
• PrimaryColour (ForegroundColorSpan)
• BackColour (BackgroundColorSpan)
• Bold (StyleSpan)
• Italic (StyleSpan)
• Underline (UnderlineSpan)
• Strikeout (StrikethroughSpan)
• AlphaLevel (read together with PrimaryColour, though the spec I'm reading says "SSA does not use this yet" so maybe not worth bothering with)
• MarginL, R and V (in fact there's already a TODO for these :)). These will feed into Cue's position & line fields. This might be a bit messy - suggest you don't start here.

The spec I'm looking at doesn't really define these clearly enough to be sure:

• Fontsize (I suspect you can use RelativeSizeSpan or AbsoluteSizeSpan but I don't know which is relevant)
  • EDIT: Small correction - if we work out how the spec defines this, we can use Cue.textSize and textSizeType for encoding size information from the "Format" section of the file that applies to the whole cue. We only need to use spans for overrides that affect substrings of the cue text.

These look harder. They have basically no support in the existing Cue interchange format:

• SecondaryColour
• OutlineColour
• ScaleX and ScaleY
• Spacing
• Angle
• BorderStyle
• Outline
• Shadow

These are hard for other reasons:

• Encoding - I'm not sure how you're supposed to read this since it's in a text file - so you've already had to guess the encoding (right?). Currently we assume it's always UTF-8.

[szaboa]

Yep, you are right, the Underline, Strikeout, ScaleX, ScaleY, Spacing and Angle are supported in v4+ only, and not in v4. I will correct the table.

Thanks for the tips, will definitely start with the easy ones.

[szaboa]

Initial PR for "Primary Colour" is here #8490.

[moneytoo]

Is there any option/API to disable, customize or detect the styling? If an app sets custom CaptionStyleCompat, it now accepts all styling while the text color is now variable based on line from subtitle file. Also, how about when user uses platform CaptioningManager to customize subtitle look - again, text color is now variable. With static background/border colors and variable text color, the subtitle line may end up low contrast on unreadable (right...?). Does my thinking here make any sense? (I've never used SSA/ASS outside some testing.)

[icbaker]

@moneytoo Assuming you're using the ExoPlayer UI components, you can force the SubtitleView to ignore 'embedded' styles (i.e. those encoded in subtitle files) by calling subtitleView.setApplyEmbeddedStyles(false). You can get the SubtitleView from playerView.getSubtitleView().

[szaboa]

I just want to note that I haven't forgotten about this, just need to find time to continue it :) Updated the table above with the PrimaryColour support.

[szaboa]

Initial PR for "FontSize" is here: #8615

[viggy96]

I don't understand how this is a 'low priority'. Proper SSA/ASS subtitle support is crucial to a video player.

[Weevild]

What's the latest update? And would this progress and (hopefully) final integration of ass work with the Nvidia Shield TV?

[icbaker]

What's the latest update? And would this progress and (hopefully) final integration of ass work with the Nvidia Shield TV?

The table in the comment above has the latest status (it's been updated with PRs merged since then).

I don't see any reason this wouldn't work on Nvidia Shield.

[szaboa]

I plan to continue with the contributions on this, just haven't had too much free time lately. Is there any preferred priority regarding the styling features?

[cosmo321]

No preference on priority, really. Glad to hear it's not forgotten. :)

[Weevild]

I don't see any reason this wouldn't work on Nvidia Shield.

It works but when using home media services such as Plex a transcode is always guaranteed, I'd assume this is because ASS isn't fully supported. Plex could probably do something about this at their end but that doesn't seem like an active inquiry.

[13steinj]

@Weevild ASS isn't fully supported on Plex for android, ironically, because ExoPlayer doesn't support ASS yet, and Plex devs don't want to go through the effort themselves. It's actually quite silly because they could transcode the ASS subtitles to a different format rather than the entire video, but don't. Even when this issue is resolved on ExoPlayer's end it'll take some time for Plex to pull it in.

[Weevild]

Plex devs don't want to go through the effort themselves.

Not a shocker there, but one could only wish. I switched to emby because they support virtually every format that Plex doesn't (subtitle-wise), but to have native support would just be a relief knowing my media is supported regardless.

transcode the ASS subtitles to a different format rather than the entire video

I think emby even does this too, guess they've got their hands full on that free content.

[albino1]

@sixones what's needed from szaboa's list to get this working well enough to merge ASS support into Plex on Android devices? If there's some obvious priority thing I think it would be beneficial for a lot of people if it gets tackled first.

[sixones]

@albino1 Probably not the right place for this, but Plex on Android does support ASS subtitles without the styling already (be sure to set your subtitle burn option to only image formats). We are working on full ASS support with positioning, styling and animations at the moment. If we transcode them into another subtitle format they would end up being the same as the basic level of support which we already offer - without the burn in option changed we burn them into the video to keep the full ASS styling with animations, styles and positions above the video.

[albino1]

Thanks for the reply, makes perfect sense. Appreciate everything you do!

[an-ghinid]

@sixones is there an issue open on your github to track that? Very keen on avoiding ASS transcoding as it's the only thing keeping me from running PMS entirely on my NAS rather than on my Shield TV (and would keep me paying for Plex Pass for the hardware acceleration support that allows this)