Support I-frame only tracks for DASH trick play

[KarlGallagher]

Similar to https://github.com/google/ExoPlayer/issues/474 But for DASH presentations.

This would be great for AndroidTV in which typical user interaction is via remote control.

[stevemayhew]

Thanks @ojw28 for your commit fea4376 this does match the design we agreed on for #474. I'm working with some origin vendors on specs for how to properly add live (specifically the trickmode property must match an adaptation set id).

We are adapting based on frame rate, I would appreciate the chance to collaborate on a specific setting that the AdaptiveTrackSelection could reference to indicate playback is in trick-mode.

I think there are two use cases:

1. seek with first frame display (set playwhenready false and disable audio)
2. high speed playback with a target frame rate

We have implemented this (by uncommenting the exclusion of the trick-mode tracks) and overriding the AdpativeTrackSelection with a method to set the trick-mode desired.

This is enough an an API change I don't want to simply make a pull request without discussing it a bit.

[tonihei]

(Note: ojw28 is no longer working on the project)

Can you describe in a bit more detail how you plan to integrate this into AdaptiveTrackSelection and the overall app workflow? Once we get a better understanding of the full picture, we can look into how to turn this design into PRs.

[stevemayhew]

Sure @tonihei Thanks, yes the real key take away from my discussions with Oliver in #474 was to include the ROLE_FLAG_TRICK_PLAY tracks with the rest of the related video tracks in the same adaptation set (TrackGroup). His change in the commit mentioned carries this forward for DASH trickmode AdaptationSet as well.

The reason this is a good choice is it avoids a full track selection (that potentially flushes the renderers and empties the SampleStream which really slows down the entry and exit from trick-mode

What we do currently is:

1. Reverse the one line in this commit bdc0db3 in DefaultTrackSelector so ROLE_FLAG_TRICK_PLAY tracks are included in the adaptative set of Formats
2. Subclass AdaptiveTrackSelection and override AdaptiveTrackSelection.canSelectFormat()

Basically we implement what the DASH-IF suggests in Trick Mode Section, blocking the use of trickmode tracks during regular playback.

So, if this was a Player API (e.g. Player.setTrickMode(boolean enabled, float targetFrameRate) it could cover the use cases.

For example, Player.setTrickMode(true, 0.0f) would enable scrub mode, specifically:

1. Disable audio render (track selection)
2. Set play when ready to false
3. Instruct AdativeTrackSelection to select only the ROLE_TRICK_MODE tracks

We have already coded this outside of ExoPlayer with overrides and custom logic (along with uncommenting the code in the commit above). My changes to enable seek to nearest keyframe for HLS have this working very well for HLS (DASH already worked well). One issue still opened (can't remember specific, @icbaker and I were discussing solutions) is that broadcom decoders are reluctant to decode a single IDR passed to MediaCodec. This is probably why Apple's mediafilesegmenter includes a couple of sacrificial P/B frames after the iFrame in their iFrame only segments, a bug #7512 we fixed quite a while ago.

So, sorry for the long history.. But in summary, to start we need an API that is plumbed all the way to the TrackSelector, similar to ExoTrackSelection.onPlaybackSpeed() that informs the AdaptiveTrackSelector that:

1. It should enter or exit trick-mode
2. A suggested target frame rate (if applicable)

Obviously this is a major enough change we would want to do it in androidx.media, we would be more than happy to back port it to our version of ExoPlayer and fully test it.

We have a huge amount of experience with trick-mode playback and a working implementation we are shipping now. I am able to sharing the code needed to enable fast scrub (which many in ExoPlayer community are asking for) with this API change. Sure would be nice to get rid of commented out code ;-)

[tonihei]

Sorry for the delay in getting back to this! And thanks for the detailed explanations.

I'm not sure if you've seen it, but there is long list of missing features and building blocks that make trick play work nicely listed in https://github.com/google/ExoPlayer/issues/3752#issuecomment-1147346684. I think you are currently talking about item 4 in this list (track selector changes). If we have a concrete proposal to move forward with changes in this area, it's probably worth filing a new issue specifically for this part. (This particular issue here is actually solved, right? The i-frame only tracks in DASH are detected and put into the TrackGroups as such)

Subclass AdaptiveTrackSelection and override AdaptiveTrackSelection.canSelectFormat()

This is interesting. I think there are two different scenarios for this: (1) paused preview (e.g. during scrubbing) and (2) high-speed playback

For (1) paused preview, it makes more sense to me to actually change the track selection (via player.setTrackSelectionParameters) because we want to change to the new track as quickly as possible and do not want to keep already buffered data in the queue or wait until the next natural track selection point. So while this could use a nice user-facing API and a solution to the issues outlined in point 4 of https://github.com/google/ExoPlayer/issues/3752#issuecomment-1147346684, it doesn't really affect AdaptiveTrackSelection.

For (2) high-speed playback, the decision to use an i-frame-only track sounds like it should be solely based on playback speed and could be enabled regardless of any other settings. The logic could roughly be: allow these tracks if the effective frame rate becomes acceptable. If it's just based on playback speed, then there is no need to plumb any new API to AdaptiveTrackSelection because onPlaybackSpeed already exists and we "only" need to change the selection logic.

[AlmantK]

This functionality is of top importance. We are telco migrating hundreds of thousands of Linux STB users to Android TV and this is No1 complaint from users - "Fast Forward does not work!". On 20 years old VCR boxes users just press >> remote button and FF starts full screen, no any menus needed, just 6-8 i-frames per second optimal, and users press the Play button when the wanted position is reached, for example end of unwanted ad. Same for rewind << key. On Apple TV STB and on iPhones/iPads i-frame tracks work out of box. On Android TV DASH must be used by telcos due to graphical subtitle formats, DRM and other reasons. Small thumbnails was a complex workaround due to the lack of CPU/video performance in the past but not with modern popular chipsets like AmLogic S905X4, where i-frame tracks can play full screen. DASH trick-play is fully implemented on Linux STBs from Arris, it plays 6-8 fps (configurable) of i-frames from dedicated i-frame track that is already provided by all major origin/packaging/CDN platforms - at maximum resolution that fits into available bandwidth with at least 4 fps (minimum FF speed is x8 with every i-frame shown if i-frames are every 2 sec). User can raise the FF speed from x8 to x32, x64, x128 speeds. Thank you for raising the attention to this... if I can help by providing URLs with i-frame tracks for testing - please ask.