Preload API

[joeyparrish]

We should have an API to allow an application to pre-load a manifest and some of the media segments without involving MediaSource or a <video> element. An application could speculatively pre-load the beginning of several pieces of content while the user is making a decision. That content could then begin playing almost instantly.

This requires some redesign. Currently, new Player() requires an HTMLMediaElement, and the load() pipeline connects from manifest parsing all the way to streaming and MediaSource. These things would need to be decoupled, and a new preload() method would have to be introduced.

Note, though, that if we had it today, this new preload() method would not be usable from a service worker just yet, because our DASH manifest parser depends on DOMParser for XML parsing, and DOMParser is not available in a service worker.

Until manifest parsing can be done from a service worker context, operations involving a manifest must be done from the page context.

[joeyparrish]

API ideas, copied from discussion in https://github.com/google/shaka-player/issues/961#issuecomment-326415628

We will have to change the API so that a video element is not required in the Player constructor. MediaSource would be set up later. Examples:

// current flow
player = new shaka.Player(video);  // attach right away, set up MediaSource
player.load('foo.mpd');

// preload flow
player = new shaka.Player();
player.preload('foo.mpd');  // start loading content
player.attach(video);  // now set up MediaSource and complete the pipeline

// wait to set up MediaSource, without preloading
player = new shaka.Player();
// time passes...
player.attach(video);  // MediaSource setup happens now, when you want it to

The HTMLMediaElement argument to the constructor becomes optional, which gives us API backward compatibility. If a video element is supplied, we will automatically attach in the constructor.

[chrisfillmore]

This is a feature we anticipate wanting to provide to our clients. And more than that, I expect we'd want to preload more than just one piece of content. This is basically to support rapid channel switching behaviour, and the like. There's another wrinkle: I also anticipate we'd want to preload some piece of content and then provide it to a specific Player instance. The reason for this is we may have multiple Players going at once.

If I'm told we need this feature, I think I could work on a PR. But given the above I'd like your input.

[chrisfillmore]

Idea, perhaps total brain fart:

/**
 * Create a Preload, which can be passed to any Player so that it
 * instantly has preloaded content.
 * @static
 * @param {object} params Anything it would need to preload content
 * @returns {shaka.Player.Preload}
 */
Player.createPreload (params) {
  // Do preloading work
}

[bennypowers]

I'd really like to be able to pre load segments or ranges from a single manifest

player = new shaka.Player(video);
// preload with optional ranges param 
player.load('foo.mpd', [
  [0, 1234], // time range in ms
  [2345, 3456],
  [4567, 5678],
  6789, // time index of single segment to pre cache
]);
// party 🎉
player.attach(video)

[chrisfillmore]

Another thought about this: it would be nice if Player#load did not require a video element to be set. It would also be nice if we could configure the player to fetch segments or not. A player with no video element attached would then just fetch manifest updates, and (optionally) build a video buffer.

[joeyparrish]

Hi @chrisfillmore, @bennypowers,

Since #1087, a video element is no longer required in the constructor, but it must be attached by the time load() is called. I do not expect to change this, so that the meaning of load() will continue to be "load all into MediaSource".

For preload, I am thinking along the lines of what Chris suggested in https://github.com/google/shaka-player/issues/880#issuecomment-360896727, where we return a token that can be used to complete the load process. The player would be capable of starting multiple preloads in parallel, to support speculation about what the end-user will click on. When the user finally clicks on a thing, you can then choose which preload to continue with. The others would then be invalidated.

Maybe something like this:

let player = new shaka.Player();
let token1 = player.preload('foo1.mpd');
let token2 = player.preload('foo2.mpd');
let token3 = player.preload('foo3.mpd');
// time passes...
player.attach(video);
await token3.load();  // like player.load('foo3.mpd'), but we already fetched segments
// tokens 1, 2, and 3 are all now invalidated.
// buffered data in tokens 1 and 2 has been dropped.
// buffered data in token3 has been transferred to Player and MediaSource.

As for Benny's suggestion in https://github.com/google/shaka-player/issues/880#issuecomment-365308775, Shaka Player does not buffer multiple independent ranges in general, so I don't plan to incorporate this into the preload design.

Thanks!

[chrisfillmore]

@joeyparrish that looks pretty good, the only suggestion I have is that it would be preferable if the client could explicitly call something like token.unload(), instead of having Player do the invalidation implicitly. This would enable e.g. the user switching back and forth between two or more channels.

[joeyparrish]

I can see how that might be useful for a TV-like scenario: you could keep the next and previous channel preloaded so the user could "channel surf" with low latency.

But, I wasn't thinking of how this would interact with live streams. I was imagining that we could buffer up to rebufferingGoal and then stop. For live streams, we would need to keep the manifest up-to-date and potentially continue buffering as the live edge moves.

It's not undoable, but I had envisioned something much smaller and simpler. We'll need to give this some careful thought.

What would your expectation be for having several tokens representing live streams? What should they be doing while they wait for a call to load()?

[chrisfillmore]

I would expect that a token for a live stream would, at minimum, continually fetch and parse manifests. At any time, the client could tell it to start downloading segments as well, via an explicit call like token.startBuffering().

This way the client could define its own user behaviours that signal an upcoming channel change (browsing a guide, rapid channel up-down behaviour, "previous channel" button on the remote, etc). Furthermore, the end-user could even tell the application how many channels to preload, potentially useful if they have a very high quality connection.

[joeyparrish]

Okay. I was thinking of keeping the API as simple as possible. Calling preload() would start buffering, and token.load() would fully connect the pipeline to MediaSource.

It seems like you're suggesting something more like this: preload() starts fetching and updating the manifest, token.startBuffering() would start buffering, and token.load() would then fully connect the pipeline.

Is that right?

In this three-stage process, when would each stage be triggered by the app? (Manifest, buffering, playback.) I guess I'm having trouble understanding why buffering would need to be a separate stage.

[chrisfillmore]

Speculatively buffering multiple streams at the same time might be undesirable, particularly on slower connections. We may also already have up to 4 streams already playing on-screen, so we don't necessarily want to use bandwidth doing more buffering.

I have seen a proof-of-concept on Android of a custom player doing rapid channel switching without buffering the background streams, and it looks good. I was actually impressed, it looked like real TV channel switching. So I think this is worth investigating.

We want to be able prefetch, where possible:

1. Manifests
2. Initialization segment
3. License

I think doing so should substantially reduce stream startup time. Regarding (3), perhaps, when preloading, the player could fetch a single segment from the stream in order to generate the challenge and fetch the license.

Thoughts on this?

[bennypowers]

If that rapid channel switching PoC is public I'd love to see a link, since my use case is similar :D

[chrisfillmore]

@bennypowers Sorry unfortunately it's not, it was work done by someone else on my team.

[chrisfillmore]

Random thought: perhaps Shaka could define an interface for e.g PreloadManager, and have a SimplePreloadManager implementation. The client app could override this implementation to customize functionality.

[chrisfillmore]

While writing my response to #1500, I realized a problem with preloading may be, where does NetworkingEngine come from? Should a "preload instance" (whatever it looks like) have its own NetworkingEngine?

[joeyparrish]

In my plan, Player would have a preload() method alongside load(). Since there's no separate PreloadManager, there's no separate NetworkingEngine. The preload() method would return an object that represents that particular preload operation, and allows you to upgrade any preload to a full-blown load(). If Player were destroyed, any associated preload object would be invalidated.

[chrisfillmore]

Will licenses be preloaded? If so, how will the client handle custom authorization? We would need to be able to handle formatting license requests for individual streams (our license proxy resides at a static URL, and identifying information is provided in the request body).

[joeyparrish]

Quick update: based on our timeline for v2.6 in Q2 2019, we expect to be working on preload in Q3. Refactors planned for v2.6 stand in the way of preload.

[OrenMe]

Hi @joeyparrish I saw this has been moved to backlog? is there any due date estimation for this feature?

[joeyparrish]

There is no due date yet. We haven't begun work yet, but it's on the top of our list, along with several other features like HLS discontinuities and ad SDK integration. Though preload was originally planned for v2.6, refactoring took much longer than anticipated, so we cut the v2.6 milestone down.

[OrenMe]

Thanks @joeyparrish, if I must use one video element I assume the current "workaround" is to: when it is time to preload content do

1. create new Instance
2. attach dummy video element
3. load requested content to be preloaded

when it is time to promote preloaded content do

1. call detach
2. call attach with the main video element
3. call load again with same video content - it will load content form cache

Is there another way to accomplish this?

[osmestad]

How we do it currently is to always have two video elements (and Shaka instances) and then switch which instance is visible to the the user (and that we listen to events from) at preloaded content switchover time :-)

[OrenMe]

Yes, this will generally work, but the are some limitations like managing autoplay restrictions on video elements, or devices that support only one video element at a time(smart TVs, cast device etc) Having such an api to load the data and possibly storing it in an array of array buffers and feeding it to a source buffer that will be attached to only one vide element will enable support across platforms with different limitations.

[joeyparrish]

Thanks @joeyparrish, if I must use one video element I assume the current "workaround" is to: when it is time to preload content do

1. create new Instance
2. attach dummy video element
3. load requested content to be preloaded

when it is time to promote preloaded content do

1. call detach
2. call attach with the main video element
3. call load again with same video content - it will load content form cache

Is there another way to accomplish this?

I don't think this will work at all. When we're attached to a media element, the buffered content will be in MSE SourceBuffers attached to that video element. When you detach from that element, you will lose everything you've buffered.

This is why preload is more than just a missing API. We have to rearchitect some of the components to separate StreamingEngine (which fetches content) from MediaSourceEngine (which feeds it to the video element). To preload content, we need to be able to store fetched content in memory until we're ready to send it through MediaSource to a committed video element.

[diogoazevedos]

Is there a way to achieve the preloading by caching the manifest and segments somewhere else, like in the fetch layer?

[enjikaka]

@diogoazevedos Like @osmestad mentioned earlier, having two shaka instances and switching between them is the best you can do for now. By doing that it will "prefetch" the upcoming chunks to be able to start pretty quickly.

If you instead of that method cache the manifest in for example the service worker you are still going to have boot time (shaka fetching chunks) when switching songs, so that's not going to be faster than two instances.

For gapless, proper integration with MediaSource is needed like so -> https://developers.google.com/web/fundamentals/media/mse/seamless-playback

[NoChance777]

@enjikaka The approach with two instances sometimes doesn't work, because a browser's policy requires a user activations/interactions for producing actions such as autoplay. See details here https://developers.google.com/web/updates/2017/09/autoplay-policy-changes

[enjikaka]

@NoChance777 You won't ever get around the autoplay block. But you only need to activate the media element played first, the second one in shaka instance two will be allowed to play after.

[NoChance777]

@enjikaka @OrenMe gave the more extended answer https://github.com/google/shaka-player/issues/880#issuecomment-590261093 why two instances won't work and from my experience, I agree with him

[enjikaka]

@joeyparrish Would this planned Preload API also have support for gapless playback or should that be a separate issue? In our case the dash chunks of the next song needs to be appended to the SourceBuffer of the MediaSource for gapless to happen, ref this link from before.