Rendering to SurfaceView sometimes does not work in Android

[mihai1voicescu]

My initial scenario was:

1. initializeWebRtc in Application (not in Activity, so it survives a tilt)
2. make a call
3. hang up
4. call again
5. now video is black
6. tilt the screen in any direction, activity gets rebuilt and everything works OK

According to the logs, webGL is rendering frames but it's all black until the first tilt.

Unfortunately I could not get a minimal replication scenario for this. Should be something similar to this branch https://github.com/mihai1voicescu/WebRTCCompose/tree/with-mainApp

I however managed to replicate the bug (or at least it looks like it) in a different scenario using Activity as the context and rebuilding the webrtc connection each tilt. You can find the minimal replication scenario here https://github.com/mihai1voicescu/WebRTCCompose/tree/master

To replicate, as stated in the readme:

1. Run App
2. Connection is established
3. Tilt screen
4. Screen is now black but it is rendering frames

I assume it's something regarding some leftover metadata in the webGL context but I am kinda stuck 😅 .

Working render logs:

 01-27 17:40:34.224  I  [10501/15589] org.webrtc.Logging CameraStatistics: Camera fps: 25.
 01-27 17:40:34.307  I  [10501/15678] org.webrtc.Logging EglRenderer: Duration: 4002 ms. Frames received: 100. Dropped: 0. Rendered: 100. Render fps: 25.0. Average render time: 7989 us. Average swapBuffer time: 2912 us.
 01-27 17:40:36.225  I  [10501/15589] org.webrtc.Logging CameraStatistics: Camera fps: 25.
 01-27 17:40:38.226  I  [10501/15589] org.webrtc.Logging CameraStatistics: Camera fps: 25.
 01-27 17:40:38.310  I  [10501/15678] org.webrtc.Logging EglRenderer: Duration: 4003 ms. Frames received: 44. Dropped: 0. Rendered: 44. Render fps: 11.0. Average render time: 7018 us. Average swapBuffer time: 3635 us.
 01-27 17:40:40.227  I  [10501/15589] org.webrtc.Logging CameraStatistics: Camera fps: 25.
 01-27 17:40:42.228  I  [10501/15589] org.webrtc.Logging CameraStatistics: Camera fps: 25.
 01-27 17:40:42.313  I  [10501/15678] org.webrtc.Logging EglRenderer: Duration: 4002 ms. Frames received: 72. Dropped: 0. Rendered: 72. Render fps: 18.0. Average render time: 7423 us. Average swapBuffer time: 3183 us.

Black render logs:

 01-27 17:42:13.401  I  [10501/18707] org.webrtc.Logging EglRenderer: Duration: 4004 ms. Frames received: 120. Dropped: 0. Rendered: 120. Render fps: 30.0. Average render time: 6803 us. Average swapBuffer time: 2567 us.
 01-27 17:42:13.691  I  [10501/18694] org.webrtc.Logging CameraStatistics: Camera fps: 30.
 01-27 17:42:15.691  I  [10501/18694] org.webrtc.Logging CameraStatistics: Camera fps: 30.
 01-27 17:42:17.405  I  [10501/18707] org.webrtc.Logging EglRenderer: Duration: 4002 ms. Frames received: 120. Dropped: 0. Rendered: 120. Render fps: 30.0. Average render time: 4767 us. Average swapBuffer time: 1798 us.
 01-27 17:42:17.692  I  [10501/18694] org.webrtc.Logging CameraStatistics: Camera fps: 30.
 01-27 17:42:18.159  I  [10501/15594] org.webrtc.Logging NetworkMonitorAutoDetect: capabilities changed: [ Transports: WIFI Capabilities: NOT_METERED&INTERNET&NOT_RESTRICTED&TRUSTED&NOT_VPN&VALIDATED&NOT_ROAMING&FOREGROUND&NOT_CONGESTED&NOT_SUSPENDED LinkUpBandwidth>=430185Kbps LinkDnBandwidth>=677812Kbps SignalStrength: -64 AdministratorUids: [] RequestorUid: -1 RequestorPackageName: null]
 01-27 17:42:19.693  I  [10501/18694] org.webrtc.Logging CameraStatistics: Camera fps: 30.
 01-27 17:42:21.410  I  [10501/18707] org.webrtc.Logging EglRenderer: Duration: 4004 ms. Frames received: 120. Dropped: 0. Rendered: 120. Render fps: 30.0. Average render time: 4716 us. Average swapBuffer time: 1845 us.
 01-27 17:42:21.694  I  [10501/18694] org.webrtc.Logging CameraStatistics: Camera fps: 30.
 01-27 17:42:23.694  I  [10501/18694] org.webrtc.Logging CameraStatistics: Camera fps: 30.
 01-27 17:42:25.420  I  [10501/18707] org.webrtc.Logging EglRenderer: Duration: 4006 ms. Frames received: 120. Dropped: 0. Rendered: 120. Render fps: 30.0. Average render time: 5039 us. Average swapBuffer time: 1979 us.
 01-27 17:42:25.695  I  [10501/18694] org.webrtc.Logging CameraStatistics: Camera fps: 30.
 01-27 17:42:27.698  I  [10501/18694] org.webrtc.Logging CameraStatistics: Camera fps: 29.

Any ideas or pointers?

Note that I am using Jetpack Compose which kills the Surfaces after they exit the tree.

[shepeliev]

I've just checked out your sample. Unfortunately, I couldn't reproduce the issue with my environment. However, I can see the potential problem: the local MediaStream is not released after the call ends in your code. It's required to call MediaStream.release() to stop video capturing and release the camera. I guess it might be the cause of the issue.

[mihai1voicescu]

The bug seems to not replicate on Android 9. It replicates on Android 11.

For the moment I can't afford to look into it anymore, just made a workaround, but as soon as time allows it I will look more into it.