Decoder freezes on quality switch when playing HLS LL stream

[strangesource]

Version

Media3 main branch

More version details

main, Tested with different versions, seems to be unrelated

Devices that reproduce the issue

Seems to be device unrelated. Can be reproduced on Android 14 emulator, Pixel 7a with Android 15 and other devices. Emulators seem to reproduce the issue faster than physical devices though.

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Yes

Reproduction steps

1. modify AdaptiveTrackSelection to switch qualities often, e.g. by doing

Index: libraries/exoplayer/src/main/java/androidx/media3/exoplayer/trackselection/AdaptiveTrackSelection.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/trackselection/AdaptiveTrackSelection.java b/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/trackselection/AdaptiveTrackSelection.java
--- a/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/trackselection/AdaptiveTrackSelection.java	(revision 10fd1ee2704e2e8b669ae0c3120cf37b62291759)
+++ b/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/trackselection/AdaptiveTrackSelection.java	(date 1743068827615)
@@ -482,7 +482,15 @@
     reason =
         newSelectedIndex == previousSelectedIndex ? previousReason : C.SELECTION_REASON_ADAPTIVE;
     selectedIndex = newSelectedIndex;
+
+    if(lastCalledTimeStamp + 5_000 < System.currentTimeMillis()){
+      selectedIndex = 0;
+      lastCalledTimeStamp = System.currentTimeMillis();
+    }else {
+      selectedIndex = 1;
+    }
   }
+  private long lastCalledTimeStamp = System.currentTimeMillis();
   @Override
   public int getSelectedIndex() {

2. play media shared via e-mail e.g. on android emulator Pixel 8 Pro Android 14
3. observe freeze, usually within ~1 minute

Expected result

No freeze on quality switch OR freeze is reported as player error to enable recovery.

Actual result

After a quality switch the video frame freezes while audio and the playhead continue. This happens regardless of the quality being switched from/to.

• after the freeze MediaCodedAdapter.dequeueInputBufferIndex always returns -1 indicating that the decoder is stuck
• overwriting the decoder reuse evaluation on quality switch to REUSE_RESULT_NO fixes the problem but drops frames on every quality change reducing the overall quality of experience

Media

Will be send via e-mail

Bug Report

• [x] You will email the zip file produced by adb bugreport to [email protected] after filing this issue.

[tianyif]

Thanks for the sample media and perfetto traces, and I can easily reproduce this issue on emulator with more aggressive track updating frequency (changing 5_000 in your code snippet to 1_000).

The DecoderCounter.queuedInputBufferCount and DecoderCounter.renderedOutputBufferCount didn't increase from the freezing point (0:30):

With adding some logs when debugging, it looks like the MediaCodec.Callback.onInputBufferAvailable / onOutputBufferAvailable were no longer triggered for video once the playback freezes https://github.com/androidx/media/blob/51efcad6720d3016d4b17fe8a1e9a11ca89e8b67/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/mediacodec/AsynchronousMediaCodecCallback.java#L214-L236

I assume this may be related with the frequent reconfiguration of the codec. Assigning to my colleague for more insights.

[strangesource]

Thanks @tianyif, that aligns with our observations.
It does seem to happen also if the codec reconfiguration happens less often but it takes longer - not sure if it's the same "chance" to freeze on every quality switch or if the likelihood is influenced by frequent reconfigurations.

[luckygoyal-bitmovin]

Hi @tonihei , is there any update on this ticket please? Since this behaviour impacts real devices and not just emulator, will really appreciate your thoughts on probable root cause and next steps.

[tonihei]

Thanks for brining it back to our attention :)

I found a minute to take a look and it seems that

• It's a duplicate of https://github.com/androidx/media/issues/1285 - I will keep this one as it has more debugging info.
• It has nothing to do with the codec. There is always an input buffer available and we get stuck because we don't fill in new buffers anymore.
• There is something weird going on filling the buffers in from the source though. I can see that it repeatedly jumps back in time, violating some basic assumptions about how our sample queue works. This is, I think, what causes it to get stuck because we can't read the samples anymore even though they are in the queue. Need to investigate further.

[luckygoyal-bitmovin]

Hi @tonihei , thanks for looking into it. Will eagerly await ant further updates/insights. Please let us know if we can provide any further data or any other assistance to root cause the issue.

[tonihei]

There are two main issues here:

• The video can get stuck entirely if a track selection switches to a new format while the last loaded chunk is one of the preload hints that hasn't been resolved to a fully published part in a playlist yet. As we switch media playlists, this gets never resolved and playback eventually hangs when trying to decode data from this preload hint (which isn't allowed to leave the option to discard the data again in case the part if removed). The fix for this is to discard the data from the preload hint if we are switching between formats.
• We evaluate whether to load a new format after every single load and then do our best to splice in the new data as needed. When loading independent segments from the end of the previous item, this works well because there isn't any overlap. However, if we attempt to load a new format after loading some non-independent parts we need to go back to the beginning of the segment and splice the new data in from there. If we are playing very close to the live edge, this data has already been read and the splice operation only succeeds at the beginning of the next chunk, leaving a gap of no data, which is visible as a "freeze". The solution is not allow track updates that would result in such gaps.

Fixing both issues seems to resolve the problems reported for the streams here and in #1285. When playing them very close to the live edge, the only remaining hiccup is an occasional buffering at the track switch points if the pipeline runs out of data (which is more working as intended and goes through the normal state transitions, unlike the issue reported here).

[tonihei]

Both changes are now submitted, will close the issue. Please feel free to reopen or file a new one if anything doesn't look fixed.

[strangesource]

Thanks for the fast investigation & fix, much appreciated.

[strangesource]

He @tonihei, took me a while to validate the fix but unfortunately I still see freezes happen. I'll be sending a new e-mail with a new test stream as the old one is expired, could you please have another look?

I tested on the latest main branch, at the time of testing this was 6d49937e9e20f216fcac3dcc1989497e290b4e02 Reproduced on Pixel 8 Pro emulator API level Level 34

[tonihei]

Thanks - I can see the remaining problem as well. We stopped switching tracks if we already know the splice won't succeed. However, it might be that we are reading samples from the data we intend to discard, and then are prevented from discarding it later.

[tonihei]

Submitted an additional fix that makes the constantly switching track selection work well with the stream above.

[strangesource]

Thanks @tonihei I'll give it another shot. 🙇