Added setApplyConstraintsFrameRate() track selection parameter

[zacklukem]

Adds the ability to set a applyConstraintsFrameRate track selection parameter which allows for selective application of maxVideoWidth, maxVideoHeight, maxVideoFrameRate, maxVideoBitrate, minVideoWidth, minVideoHeight, minVideoFrameRate, and minVideoBitrate constraints only when the format being evaluated is known to have a frame rate greater than applyConstraintsFrameRate.

applyConstraintsFrameRate would not effect setViewportSize so content above the viewport limits will be filtered out regardless of applyConstraintsFrameRate

This feature allows limiting high frame rate content while still allowing high resolution low framerate content and low resolution high framerate content

This is currently in use with overheating Android TV boxes as a temporary alternative to the Android Q Thermal API (#6284)

[stevemayhew]

This is the API we are using, combined with detecting the thermal throttle condition based on frame drops and frame rate, based on @ojw28 's comment here https://github.com/google/ExoPlayer/issues/6284#issuecomment-539695298, we do not call this API until their is both excessive frame drops and thermal throttle reported.

What we have found with the non-tunneled decoders (AmLogic based in particular) is that higher frame rate 4k content (60FPS) is more CPU intensive, it is the frame rate that really drives the CPU so reducing to a lower resolution / frame rate for a period of time until the thermal condition recovers solves the issue.

People are much more sensitive to frame drops then resolution changes.

Happy for any suggestions, the core is to only constrain over the frame rate threshold tracks.

[tonihei]

I'm not sure this setting makes sense for the generic track selector because it creates quite a complicated logic with conditions being conditional on something else.

Potential alternatives:

• Can you just specify a minVideoBitrate or maxVideoFrameRate to achieve the same goal? If you want to set this parameter during playback, you can easily look at the available tracks and choose a matching parameter depending on the available alternatives.
• You could override DefaultTrackSelector.selectVideoTrack, call the super method and then filter out the unwanted tracks from the returned ExoTrackSelection.Definition.
• For more dynamic track selection filtering, you can also override AdaptiveTrackSelection.canSelectFormat to filter unwanted formats without interrupting playback.

[zacklukem]

We are using this constraint to allow either higher frame rate lower resolution or lower frame rate higher resolution content when a device is throttling, the reasoning being that some content may value frame rate over resolution such as in sports channels where you may want smoother video.

Here is a code sample of how we are using it where the stress policy level is determined based on frame drops:

switch (newPlayerStressPolicyLevel) {
case PLAYER_STRESS_NONE:
    trackSelectionParameters = trackSelectionParameters.buildUpon()
            .clearVideoSizeConstraints()
            .setApplyConstraintsFrameRate(Format.NO_VALUE)
            .build();
    break;
case PLAYER_STRESS_MODERATE:
    trackSelectionParameters = trackSelectionParameters.buildUpon()
            .setMaxVideoSize(2560, 1440)
            .setApplyConstraintsFrameRate(30)
            .build();
    break;
//...
}

Assuming that the available tracks are [2K@60, 2K@30, 4K@60, 4K@30], at PLAYER_STRESS_NONE there would be no change to the available tracks, and at PLAYER_STRESS_MODERATE the available tracks would become [2K@60, 2K@30, 4K@30]

---

I agree that setApplyConstraintsFrameRate complicates the track selector logic by making setMaxVideoSize conditional on another constraint, so perhaps a better option would consolidate the desired behavior into one constraint

For example:

// Constrains framerate only for video above the given resolution
setHighResolutionMaxFrameRate(int framerate, int videoWidth, int videoHeight)

(needs a better name than setHighResolutionMaxFrameRate)

which would change the above code sample to something like:

switch (newPlayerStressPolicyLevel) {
case PLAYER_STRESS_NONE:
    trackSelectionParameters = trackSelectionParameters.buildUpon()
            .unsetHighResolutionMaxFrameRate() // or clearHighResolution, or pass NO_VALUE
            .build();
    break;
case PLAYER_STRESS_MODERATE:
    trackSelectionParameters = trackSelectionParameters.buildUpon()
            .setHighResolutionMaxFrameRate(30, 2560, 1440)
            .build();
    break;
//...
}

[tonihei]

It still sounds very specific to the logic you are trying to implement and I don't see how this is a generally useful parameter. The reason I'm so cautious about adding a new parameter like this is that the track selection can never be fully correct and parameterizable into all possible details. There will always be cases where one app likes the selection logic to work one way and the next app likes the same piece of logic to work in the exact opposite way. While we could add additional parameters to cover both cases, it soon becomes very unwieldy and hard to follow. So we try to find a middle ground where we provide as many simple configuration parameters as possible (e.g. setting min/max for single values like bitrate), but ask apps to customize the relevant components themselves if the logic becomes more complicated than that.

In your case, there seem to be multiple ways to achieve the goal you like with relatively simple custom code:

• Since you already know all tracks at the point where you make this decision, you could use TrackSelectionOverrides to select any subset of tracks you prefer.
• You may be able to use maxVideoBitrate if 4K@60 has the highest bitrate in your example above.
• You could filter tracks in selectVideoTrack as described in my previous post.

Would one of these alternatives work for you as well? In the long run, we should implement a frame-drop based track blacklisting (as suggested in https://github.com/google/ExoPlayer/issues/6284#issuecomment-539695298), which would solve the problem without additional configuration.

[stevemayhew]

• Since you already know all tracks at the point where you make this decision, you could use TrackSelectionOverrides to select any subset of tracks you prefer.

Very possible, we wouldn't want this to conflict with other uses of overrides.

• You may be able to use maxVideoBitrate if 4K@60 has the highest bitrate in your example above. Thanks @tonihei, these are great ideas.

Yes, a combination of bitrate, frame rate and resolution is used to determine the tracks that should be pruned to reduce load.

• You could filter tracks in selectVideoTrack as described in my previous post.

This seems like the best alternative. The subclass can invalidate the track selection if conditions change.

Would one of these alternatives work for you as well? In the long run, we should implement a frame-drop based track blacklisting (as suggested in #6284 (comment)), which would solve the problem without additional configuration.

Yes, but it is better to perform a new trackselection and thus flush the buffered content (to more quickly mitigate the situation and bring the temperature under control). I would encourage you to consider this with any solution you implement for this. FWIW, we will have several 100,000 AMLogic boxes this will be deployed to, with analytics, so we will have good feedback on how well it works.

[icbaker]

Closing all PRs on this deprecated project. We are now unable to merge PRs from here. If you would like us to consider this change again please file a new PR on the media3 project: https://github.com/androidx/media/pulls