ExoPlayer.setVideoEffects causes freezes etc

[steelbytes]

media3 1.2.1 Pixel7Pro/Android14

ExoPlayer.setVideoEffects causes freezes in DefaultVideoFrameProcessor.flush() or MediaCodec$CodecException("client does not own the buffer") when a video effect is a bit slow in TextureOverlay.getTextureId (depends how slow and if the user clicks skip back/forwards or waits for it to die).

A video effect can legitimately be a bit slow if the effect changes every frame.

ExoPlayer should instead drop/skip frames. Freezing or dying is not a result I can give to my users.

If my effect is fast then it doesn't have problems. Also using Transformer it is completely fine..

Simple repro: take the demo app and add a TextOverlay video effect to the ExoPlayer that I've made slow with some pointless loops to simulate some processing every frame (in my real code I render to a Bitmap and transfer that into a GL texture). Testing with the 'One hour frame counter (mp4)' selection that is built into the demo app.

add this to PlayerActivity.initializePlayer

        TextOverlay overlay = new TextOverlay() {
        private final SpannableString text = new SpannableString("test ");// + id);
        private final byte[] x = new byte[1000];
        @Override public SpannableString getText(long presentationTimeUs) { return text; }
        @Override public int getTextureId(long presentationTimeUs) throws VideoFrameProcessingException {
          for (int k = 0; k < 1000; ++k)
            for (int j = 0; j < 1000; ++j)
              for (int i = 0; i < x.length; ++i)
                x[i] = (byte)(x[i] + 1);
          return super.getTextureId(presentationTimeUs);
        }
      };
      List<Effect> effects = new ArrayList<>();
      effects.add(new OverlayEffect(new ImmutableList.Builder<TextureOverlay>().add(overlay).build()));
      player.setVideoEffects(effects);
      };
      List<Effect> effects = new ArrayList<>();
      effects.add(new OverlayEffect(new ImmutableList.Builder<TextureOverlay>().add(overlay).build()));
      player.setVideoEffects(effects);

[claincly]

ExoPlayer does have frame skipping logic around late frames. As a rule of thumb, if a frame is dropped if it's late by a threshold, in both places after decoding (prior to processing) and after processing. That said, if a video effect is super slow, real-time playback will be hard.

Can you elaborate on your real use-case? I'm afraid spinning the CPU is not a good way to force slow effects, because they should run on the GPU.

[steelbytes]

ExoPlayer does have frame skipping logic around late frames

it does drop some frames and then it just freezes or throws an exception. my assertion is that with video effects the dropping is not ereliable.

I'm afraid spinning the CPU is not a good way to force slow effects, because they should run on the GPU.

you appear to misunderstand my example. My real code does not try and force slow effects. My real code does some thinking that isn't quite quick enough to keep exoplayer from dieing.

[steelbytes]

Found the problem:

when there are dropped frames then ExternalTextureManager.maybeExecuteAfterFlushTask() doesn't call the onFlushComplete callback so DefaultVideoFrameProcessor.flush() hangs.

some pseudo / simplified code

DefaultVideoFrameProcessor.flush() 
{
  // register onFlushComplete callback
  inputSwitcher.activeTextureManager().setOnFlushCompleteListener(latch::countDown);

  // start the flush
  videoFrameProcessingTaskExecutor.submit(finalShaderProgramWrapper::flush);

  // stuck here because ExternalTextureManager.maybeExecuteAfterFlushTask sometimes skips the callback
  latch.await();
}

FinalShaderProgramWrapper.flush()
{
  calls ExternalTextureManager.maybeExecuteAfterFlushTask via a long path
}

ExternalTextureManager.maybeExecuteAfterFlushTask() 
{
  // numberOfFramesToDropOnBecomingAvailable is sometimes > 0
  if (... numberOfFramesToDropOnBecomingAvailable > 0)
  {
    // *** this happens when freezing but not other times ***
    return; 
  }

  videoFrameProcessingTaskExecutor.submitWithHighPriority(onFlushCompleteTask);
}

[droid-girl]

Thank you for sharing the details on the problem. @claincly could you investigate this?

[steelbytes]

I think I deserve a sticker or badge for this :-)

[steelbytes]

@claincly could you investigate this?

Any news?

[peterstickermaker]

I got the same issue. It often happen when you play a local video file and use more than 1 effects After some seeking to a random position, the video got stuck, no error reported I tried both on Emulator (Android 14) and Samsung S20 FE( Android 12) with surface_type="texture_view" Step to reproduce: Open a video from gallery , then tap to seek button aggressively , the player then freezes

[droid-girl]

This bug is prioritized and @claincly is looking how to fix it now

[how8570]

Is there any news here? Another month has passed. (´;ω;`)

[claincly]

Sorry for not updating this issue, please see if https://github.com/androidx/media/commit/bef3d518d2dd11155a759d4e62bd649e32752a2d fixes the issue, thanks!

[how8570]

Hi, I tried the main branch (tested on b047e81e025c3e4dbb013bad48066524935e4713, which comes after bef3d518d2dd11155a759d4e62bd649e32752a2d) on the demo app and my production. The seeking issue is basically fixed. I can get the correct video orientation, and the seeking function works well.

However, I found that in extreme cases, seeking forward/backward rapidly and repeatedly can still cause it to get stuck in buffering. (It's hard for me to reproduce, but I did reproduce it a few times. Maybe there's some timing issue?)

Overall, the current version is acceptable for me. Thanks for your help!

[claincly]

We'll take some further look into it, but glad that it improved the seeking and you can use it now

[claincly]

I'll close this issue for now, please open another if you see similar issue happening. Kudos to @steelbytes for digging into it!