Multiple simultaneous PGS subtitles

[linkandzelda91]

Version

Media3 main branch

More version details

I checked in the app i was having this issue with and found this reference androidx-media3 = "1.6.1"

Devices that reproduce the issue

OnePlus 12 + that one $20 walmart brand androidtv box that everyone buys https://www.walmart.com/ip/onn-Google-TV-4K-Streaming-Box-New-2023-4K-UHD-Resolution/2835618394?wl13=2257&selectedSellerId=0&wmlspartner=wlpa&gStoreCode=2257&gQT=1

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

YES

Reproduction steps

sort of a repost of an older issue from the now deprecated exoplayer repo https://github.com/google/ExoPlayer/issues/7458 so I think it's the latest. the issue is still present. More or less makes Anime with PGSSUB unwatchable as there are frequently signs that have translations pasted over them with PGSSUB and also audio subtitles going at the same time.

Expected result

PGSSUB displays eg a sign translation and also an audio track translation at the same time

Actual result

one of the two will display for a quarter of a second then disappear.

Media

http://ex.mayogirl.com/exoplayer2.mkv well, I would like to share the original file but copyright, so hopefully this is good enough I destroyed the video quality so it's unwatchable, but the audio and subtitles are still there. 14:00 and start, 14:04 a subtitle appears, and at 14:05 a character begins to speak, which has a corresponding subtitle, it should look like this (reference VLC):

instead the first subtitle disappears prematurely, and the subtitle associated with the audio track never appears to begin with. it stays like this until there is only 1 subtitle attempting to be presented, at which point subs resume normally.

if this isn't good enough I would appreciate a little help figuring out how to create a test video, extracting the PGSSUBs from this one and slapping it on an unrelated video would be fine but I dont even know how to do that.

Bug Report

• [x] You will email the zip file produced by adb bugreport to [email protected] after filing this issue.

[linkandzelda91]

15:00 also a good example

[icbaker]

ExoPlayer now has much better general support for overlapping-in-time subtitles, since 1.4.0. Unfortunately it's still a bit tricky to make this work with PGS-in-MKV, because we don't have a good way to convey the the 'sample duration' from MatroskaExtractor into PgsParser via the TrackOutput interface and SubtitleTranscodingTrackOutput implementation. Without the duration, we can't switch from REPLACE to MERGE here:

https://github.com/androidx/media/blob/108236f00ca96a52134027e81f852ca01666edf7/libraries/extractor/src/main/java/androidx/media3/extractor/text/pgs/PgsParser.java#L44-L45

We have some awkward hacks for conveying the matroska 'sample duration' for SSA, SubRip & WebVTT where we recreate the cue timestamp string based on the matroska duration, which is then parsed again by SsaParser, SubripParser and WebvttParser:

https://github.com/androidx/media/blob/108236f00ca96a52134027e81f852ca01666edf7/libraries/extractor/src/main/java/androidx/media3/extractor/mkv/MatroskaExtractor.java#L1798-L1807

There's no PGS string format to use that PgsParser already understands.

If we instead had a different way to communicate sample duration from an Extractor implementation to a SubtitleParser implementation, we could use this for PGS and likely remove the string-formatting code for the other formats too. Without this plumbing, I think this support for concurrent subtitles is hard to implement.

[linkandzelda91]

ExoPlayer now has much better general support for overlapping-in-time subtitles, since 1.4.0. Unfortunately it's still a bit tricky to make this work with PGS-in-MKV, because we don't have a good way to convey the the 'sample duration' from MatroskaExtractor into PgsParser via the TrackOutput interface and SubtitleTranscodingTrackOutput implementation. Without the duration, we can't switch from REPLACE to MERGE here:

media/libraries/extractor/src/main/java/androidx/media3/extractor/text/pgs/PgsParser.java

Lines 44 to 45 in 108236f

public static final @CueReplacementBehavior int CUE_REPLACEMENT_BEHAVIOR = Format.CUE_REPLACEMENT_BEHAVIOR_REPLACE; We have some awkward hacks for conveying the matroska 'sample duration' for SSA, SubRip & WebVTT where we recreate the cue timestamp string based on the matroska duration, which is then parsed again by SsaParser, SubripParser and WebvttParser:

media/libraries/extractor/src/main/java/androidx/media3/extractor/mkv/MatroskaExtractor.java

Lines 1798 to 1807 in 108236f

private static void setSubtitleEndTime(String codecId, long durationUs, byte[] subtitleData) { byte[] endTimecode; int endTimecodeOffset; switch (codecId) { case CODEC_ID_SUBRIP: endTimecode = formatSubtitleTimecode( durationUs, SUBRIP_TIMECODE_FORMAT, SUBRIP_TIMECODE_LAST_VALUE_SCALING_FACTOR); endTimecodeOffset = SUBRIP_PREFIX_END_TIMECODE_OFFSET; break; There's no PGS string format to use that PgsParser already understands.

If we instead had a different way to communicate sample duration from an Extractor implementation to a SubtitleParser implementation, we could use this for PGS and likely remove the string-formatting code for the other formats too. Without this plumbing, I think this support for concurrent subtitles is hard to implement.

very over my head, but does this mean that if I remuxed to a different container format it'd start working?

[icbaker]

I believe that ExoPlayer only supports PGS in Matroska containers (based on usages of our MimeTypes.APPLICATION_PGS constant).

[linkandzelda91]

I believe that ExoPlayer only supports PGS in Matroska containers (based on usages of our MimeTypes.APPLICATION_PGS constant).

ah okay. I misunderstood thanks for explaining further.