androidx
OpenGL Processing with multiple renderers
I have an exoplayer implementation with 4 renderers. I want 3 of these to output to a regular surface view, while the main ( first renderer) outputs to a GLSurface view ( I am currently doing a basic filer shader). I was able to do the same with a single renderer (So I assume there is no error in my GLVideoRenderer). But incase of multiple renderers I am facing this issue. Kindly help!
Below is my customRenderersFactory
private final int totalVideoRenderers;
/**
* @param context A {@link Context}.
*/
public CustomRenderersFactory(Context context, int totalVideoRenderers) {
super(context);
this.totalVideoRenderers = totalVideoRenderers;
}
@Override
protected void buildVideoRenderers(
@NonNull Context context,
@ExtensionRendererMode int extensionRendererMode,
@NonNull MediaCodecSelector mediaCodecSelector,
boolean enableDecoderFallback,
@NonNull Handler eventHandler,
@NonNull VideoRendererEventListener eventListener,
long allowedVideoJoiningTimeMs,
@NonNull ArrayList<Renderer> out
) {
for (int i = 0; i < totalVideoRenderers; i++) {
super.buildVideoRenderers(
context,
extensionRendererMode,
mediaCodecSelector,
enableDecoderFallback,
eventHandler,
eventListener,
allowedVideoJoiningTimeMs,
out
);
}
}
This is my player manager where I initialise the player
public BasePlayerManager(Context context, SurfaceViewManager surfaceViewManager) {
this.context = context.getApplicationContext();
this.surfaceViewManager = surfaceViewManager;
}
@OptIn(markerClass = UnstableApi.class)
public ExoPlayer initializePlayer(String Url, int numOfViews) {
DashMediaSource dashMediaSource = new DashMediaSource.Factory(new DefaultDataSource.Factory(context))
.createMediaSource(MediaItem.fromUri(Uri.parse(Url)));
DefaultBandwidthMeter.Builder bandwidthMeterBuilder = new DefaultBandwidthMeter.Builder(context);
BandwidthMeter bandwidthMeter;
if (FeatureFlagManager.getInstance().isEnabled(FeatureFlag.CUSTOM_ABR)) {
bandwidthMeter = bandwidthMeterBuilder.build();
} else {
bandwidthMeter = bandwidthMeterBuilder.setInitialBitrateEstimate(35_700_000L).build();
}
player = new ExoPlayer.Builder(context)
.setRenderersFactory(new CustomRenderersFactory(context, numOfViews+1))
.setTrackSelector(new CustomTrackSelector(context))
.setBandwidthMeter(bandwidthMeter)
.build();
player.setMediaSource(dashMediaSource);
player.prepare();
player.setRepeatMode(Player.REPEAT_MODE_ALL);
player.setPlayWhenReady(true);
return player;
}
@OptIn(markerClass = UnstableApi.class)
public ArrayList<Renderer> setupRenderers() {
for (int i = 0; i < player.getRendererCount(); i++) {
if (player.getRendererType(i) == C.TRACK_TYPE_VIDEO) {
videoRenderers.add(player.getRenderer(i));
Log.d("HELLO", "123");
}
}
surfaceViewManager.attachRenderersToSurfaces(player, videoRenderers);
return videoRenderers;
}
This is my Surface view manager class
private final GLSurfaceView mainSurfaceView;
private final List<SurfaceView> surfaceViewList;
private final List<TextView> surfaceTextList;
public SurfaceViewManager(
GLSurfaceView mainSurfaceView,
List<SurfaceView> surfaceViewList,
List<TextView> surfaceTextList
) {
this.mainSurfaceView = mainSurfaceView;
this.surfaceViewList = surfaceViewList;
this.surfaceTextList = surfaceTextList;
}
private int prevSurfaceIndex = -1;
public List<SurfaceView> getSurfaceViewList() {
return surfaceViewList;
}
@OptIn(markerClass = UnstableApi.class)
public void attachRenderersToSurfaces(ExoPlayer player, ArrayList<Renderer> videoRenderers) {
glRenderer.setOnSurfaceTextureReadyListener(surfaceTexture -> {
Surface glSurface = new Surface(surfaceTexture);
new Handler(Looper.getMainLooper()).post(() -> {
player.setVideoSurface(glSurface);
});
});
for (int i = 1; i < surfaceViewList.size(); i++) {
SurfaceView surfaceView = surfaceViewList.get(i);
Renderer renderer = videoRenderers.get(i);
surfaceView.getHolder().addCallback(new CustomSurfaceHolderCallback(player, renderer));
}
}
and this is my activity onCreate
protected void onCreate(@Nullable Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_live_view); configureWindowSettings(); getEventDataFromIntent();
mainView = findViewById(R.id.main_gl_view);
mainView.setEGLContextClientVersion(2);
glVideoRenderer = new GLVideoRenderer();
glVideoRenderer.setGLSurfaceView(mainView);
mainView.setRenderer(glVideoRenderer);
mainView.setRenderMode(GLSurfaceView.RENDERMODE_WHEN_DIRTY);
surfaceViewManager = new SurfaceViewManager(mainView, surfaceViewList, surfaceTextList);
playerManager = BasePlayerManager.getInstance(this, surfaceViewManager);
player = playerManager.initialize(surfaceViewManager.getSurfaceViewList().size(), liveUrl);
videoRenderers = playerManager.setupRenderers();
}
can someone point out what am I doing wrong. I just get a black screen with the following error
androidx.media3.exoplayer.mediacodec.MediaCodecRenderer$DecoderInitializationException: Decoder init failed: c2.qti.avc.decoder, Format(0, Player Cam 1, null, video/avc, avc1.640028, 5137749, null, [1920, 1080, 25.0, ColorInfo(BT709, Limited range, SDR SMPTE 170M, false, 8bit Luma, 8bit Chroma)], [-1, -1])
@ssent-dlb
Thank you for your patience.
I'm not sure if you are missing in your code snippets the section where you set the other surfaces to the additional video renderers? Your call to player.setVideoSurface(glSurface) would set that glSurface to all the video renderers set onto the player. So all the video renderers are trying to set up different decoders to the same surface instance.
You'll probably need to extend your customization of your video renderers to intercept the calls in handleMessage with MSG_SET_VIDEO_OUTPUT to provide the individualized surface per renderer.
Note: I would also check the logic in the player for ExoPlayer's setVideoSurface/setVideoOutputInternal code to make sure that in intercepting the surface assignment downstream that all the logic from higher up in the pipeline still works.
Hope that helps!
@microkatz , Thanks for the reply.
The CustomSurfaceViewHolderCallback in the SurfaceViewManager class is responsible for setting up the other renderers-surface pairs. Below is the code for the CustomSurfaceHolderCallback
public class CustomSurfaceHolderCallback implements SurfaceHolder.Callback {
private final ExoPlayer player;
private final Renderer renderer;
public CustomSurfaceHolderCallback(ExoPlayer player, Renderer renderer) {
this.player = player;
this.renderer = renderer;
}
@Override
public void surfaceCreated(@NonNull SurfaceHolder holder) {
player.createMessage(renderer)
.setType(MSG_SET_VIDEO_OUTPUT)
.setPayload(holder.getSurface())
.send();
}
@Override
public void surfaceChanged(@NonNull SurfaceHolder holder, int format, int width, int height) {
player.createMessage(renderer)
.setType(MSG_SET_VIDEO_OUTPUT)
.setPayload(holder.getSurface())
.send();
}
@Override
public void surfaceDestroyed(@NonNull SurfaceHolder holder) {
holder.getSurface().release();
}
}
This code is called in the attachRenderersToSurfaces function after setting up the glSurfaceView using player.setVideoSurface(glSurface).
I had also tried setting up the glSurfaceView previously by replacing the above code with the below but it gave the same error.
glRenderer.setOnSurfaceTextureReadyListener(surfaceTexture -> {
Surface glSurface = new Surface(surfaceTexture);
new Handler(Looper.getMainLooper()).post(() -> {
player.createMessage(videoRenderers.get(0))
.setType(MSG_SET_VIDEO_OUTPUT)
.setPayload(glSurface)
.send();
});
});
Any further help would be greatly appreciated. TIA !
@ssent-dlb
Thank you for the additional info! I see that you get the same error after no longer using player.setVideoSurface.
Firstly, would you be able to provide a bug report? If you cannot provide one publicly then please send to [email protected] with the subject Issue #2455. If you can please make sure to add event logging to your project, https://developer.android.com/media/media3/exoplayer/debug-logging then that would be great. Please also update this issue to indicate you've done this.
It may be that the device you are testing on cannot handle creating so many decoders, hence the initialization failure? What happens if you use just two renderers with the surfaces(No glSurface)? Or one renderer with gLsurface view and one regular renderer? Do you get the error if you have all these renderers but don't attach surfaces to any or some of them?
Hey @ssent-dlb. We need more information to resolve this issue but there hasn't been an update in 14 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.
If you have more information that will help us get to the bottom of this, just add a comment!
Since there haven't been any recent updates here, I am going to close this issue.
@ssent-dlb if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.