Streamed subtitles

[ramidzkh]

Allows for subtitles loaded as a URL to be loaded slowly, instead of waiting on the server/network for the entire file first.

I haven't tested Font API and stress test. I have tested the rest (including the other Local Fonts API

[ramidzkh]

I'm not sure on how to do the font API test, so telling me how would be nice. Also, should I push the build artifacts, since I updated the WASM part? I also left in some code to simulate throttling so you can test it yourself, and I'll remove it once you tell me to.

[ThaUnknown]

question:

why?

if subs are saya 80MB, then you won't be able to render them anyways because of lag

[ramidzkh]

question:

why?

if subs are saya 80MB, then you won't be able to render them anyways because of lag

This allows the server to stream subtitle data, e.g. if it's live muxing, and let the player see data immediately instead of waiting for it to finish. Like ffmpeg, it doesn't need to read the whole video to use the subtitles filter

[ThaUnknown]

that's what the createEvent API is for: from any EBML parser output:

const stylesMap = [{ style: 'yourstyle' }]
function constructSub (subtitle, subtitleIndex, trackNumber) {
    return {
      Start: subtitle.time,
      Duration: subtitle.duration,
      Style: stylesMap[trackNumber][subtitle.style || 'Default'] || 0,
      Name: subtitle.name || '',
      MarginL: Number(subtitle.marginL) || 0,
      MarginR: Number(subtitle.marginR) || 0,
      MarginV: Number(subtitle.marginV) || 0,
      Effect: subtitle.effect || '',
      Text: subtitle.text || '',
      ReadOrder: 1,
      Layer: Number(subtitle.layer) || 0,
      _index: subtitleIndex
    }
}

const renderer = new JASSUB(...)

const set = new Set()
set.add(EBMLOutputObject)
renderer.createEvent(constructSub(EBMLOutputObject), set.size - 1, 1)

this is pseudocode, but it should give you a rough idea

[ramidzkh]

It's much more convenient for the library to handle parsing the event format and whatnot for you; lets say the server's just dumping ffmpeg -i {} -copyts -map 0:{} -an -vn -f ass - to Videos/{...}/Subtitles/{...}/Stream.ass (like Jellyfin), this would be a seamless change

[ThaUnknown]

imho jellyfin should parse the subtitles client-side not server-side, what you're suggesting is a lazy fix, rather than a solution and it doesn't sit well with me

it's not that I don't want to merge this, it's just it doesn't actually provide anything that useful imho

[ramidzkh]

imho jellyfin should parse the subtitles client-side not server-side

How would the subtitles be parsed client side? It's not sent along with the video/audio stream. Jellyfin just streams the file once the client requests it (client indicates which language stream or whatever) and receives an ass file.

Besides, once the server locates the subtitles, either external to the video or inside, it's still going to be IO-bound which I'd rather not the client see as a 5 second gap when switching subtitle languages.

what you're suggesting is a lazy fix, rather than a solution and it doesn't sit well with me

It's a seamless optimization for slower networks and hardware, not a lazy fix

[ThaUnknown]

It's a seamless optimization for slower networks and hardware, not a lazy fix

what I mean by "lazy fix" is that you really should append new subtitles, rather than overwrite all existing subtitles with new ones

[ramidzkh]

It doesn't overwrite anything, ass_process_chunk or ass_process_data is used to update the existing track which appends the new events/sections. Some of the contributors on the IRC recommended it:

15:04 <ramidzkh> is there a nice way to stream ass file content to libass, instead of passing the entire file content at once?
16:01 <rcombs> yup
16:02 <rcombs> ass_process_chunk, or ass_process_data
17:27 <ramidzkh> ass_process_chunk talks about the MKV format so I didn't read much into it
17:27 <ramidzkh> wonder why I didn't see ass_process_data before, let me try that
17:29 <JEEB> :)
17:31 <ramidzkh> parse_memory has a few more steps after reading the lines:
17:31 <ramidzkh>     // external SSA/ASS subs does not have ReadOrder field
17:31 <ramidzkh>     for (i = 0; i < track->n_events; ++i)
17:31 <ramidzkh>         track->events[i].ReadOrder = i;
17:31 <ramidzkh>     if (track->track_type == TRACK_TYPE_UNKNOWN) {
17:31 <ramidzkh>         ass_free_track(track);
17:31 <ramidzkh>         return 0;
17:31 <ramidzkh>     }
17:31 <ramidzkh>     ass_process_force_style(track);
17:31 <ramidzkh> do I need to worry about any of these?
22:33 <Oneric> ramidzkh: as long as the first packet contains a full header ([Script Info], a matching style section; like MKVs codec private) you do not need to worry about it.
22:33 <Oneric> the data you stream should be line buffered though; process_data cannot deal with incomplete lines

[ThaUnknown]

It deletes the existing track and overwrites it with new content, instead of appending new content, which I feel defeats the purpose of this as if the file is large you'll end up waiting for the newest lines just as much if not even longer, just the older subtitles will show up earlier, which is why I said this is lazy, as I think you should use the createEvent API to append newly streamed content rather than rebuilding the subtitle track each time

I'll properly review this in a few days once I have time since it feels like a lot of code for what it does, and them merge

[ThaUnknown]

How would the subtitles be parsed client side?

you can honestly use any proxy for the file being loaded, eg: This is the code I run every time a stream to an mkv/webm file is created: https://github.com/ThaUnknown/miru/blob/master/src/background/background.js#L64-L75

tho I have my own custom EBML implementation, I assume JF uses 3rd party tools for this