opentok
Lost reference to publisher camera
After a time using the app in background or sometimes after a reconnect the publisher loss the reference to the camera. The audio still work for the publisher but the video is lost(Black screen). We don't received any errors in console. Any ideas to fix it?
@Carlosp16 This seems like a duplicate of the following: https://github.com/opentok/opentok-react-native/issues/141
Could you please check out the solution mentioned in the issue?
I think this is distinct from #141
We have an issue that is somewhat difficult to reproduce where a user may put the app in the background for some time, then restore the app to foreground, but their video is gone. It doesn't happen every time, sometimes the video comes back as expected. When it does disappear, toggling the video on/off doesn't have any effect - both users just see a black feed.
I should note that it's not only moving the app to the background. This happens occasionally while the users are in a session - but it has seemed to happen more frequently in the cases where one user leaves the app, then returns.
We've focused primarily on iOS - not sure about behavior on Android.
@msach22 Ticket #141 is not related to this issue, because I have "Background Mode" capability activated on Xcode. While you are on a call and App goes into background mode the session is not being destroyed, the real problem is when the App comes into foreground again, for some reason the publisher video feed is gone (black screen), I tried forcing the publisher component to render again based on the AppState(Active, Inactive, Background) but no luck, once the app goes to background you lose the publisher video feed.
@msach22 Any pointers on where we could look? Still haven't been able to figure this one out...
@evillemez @cmedinadeveloper Apologies for the delay on this, will check this out and report my findings in the next few days!
@msach22 Cool man - thanks for taking a look. If there's any more info we can provide, let us know and we'll do what we can.
@msach22 Hey, just checking in. Did you have any luck uncovering anything related to this? If there's any information we can provide, or something you would like us to try - please let us know and we'll do what we can.
@evillemez Sorry for the delay, here's what I did and I'm still unable to see this issue:
- Created a new RN project
- Added this library
- Added the code from https://github.com/opentok/opentok-react-native-samples/tree/master/BasicVideoChat
- Ran it on my iPhone 7 device
- Went into background and foreground many times and the SDK was able to reclaim the camera each time.
Is there any other information that might help dig deeper?
- iOS version
- iPhone version
Edit: When this does happen, how long has the app been in background mode? Are you still transmitting audio in the session or are you completely disconnected?
@evillemez Could you please email me at [email protected] with a recent sessionId where this happens? This will allow me to see if you are still sending video packets or not. I'm trying to see if the camera gets reclaimed but somehow it doesn't bubble up to the RN view.
You can also use Inspector to check out the video packets information
@msach22 Yeah - we can create a session to reproduce and reach out to you over email. Thanks so much!
Any luck with this issue? Im facing the same one in SWIFT SDK for opentok. The publisher feed goes black sometimes when the app comes back to foreground.
@devsujith No luck, this is still happening to us, we are not able to find a workaround
Are you facing this issue each time the app is backgrounded and comes back to foreground? Or is it the scenario when the session window is reconnected once and then followed by background and resume?
Just a note that, for us, this seems to be an iOS only issue. On Android the video continues if the app is in the background. So switching back and forth doesn't seem to cause a problem.
@msach22 Based on recent conversations, I started thinking about this again. Would force unmounting/remounting the publisher be an approach we should try if we can detect that video has been lost?
@evillemez Yes, if you unmount and remount a publisher component, you will essentially remove the old publisher and create a new publisher.
@evillemez Is there a way to detect if video is lost. On android if camera is taken away by any other app like whatsapp video call, video stream freezes for me too
Eh, that's tricky... my first guess would be to hook into Subscriber.videoNetworkStats and Subscriber.audioNetworkStats - you can see if the a/v bytes received stop changing, or go to zero. But, you also have to check if that was intentional or not, if your app allows users to disable their video.
@evillemez Have you tried using AppState and unmounting and remounting the OTPublisher component?
If you can see that the publisher changed from background mode to foreground and they are not sending any videoNetworkStats, you can unmount and remount to create a new publisher instance
One more question, is this happening on all iOS devices or just specific ones?
@msach22 @cmedinadeveloper
I'm not sure if we've tried remounting the Publisher in the case of an app state change. I think we did, but it's been a while, my memory is fuzzy on this now... @cmedinadeveloper do you remember?
As far as I'm aware it's all iOS devices. We had shift our focus on to something else for a bit, but are coming back to this soon.
@msach22 Hi everyone! Also face with this issue, when sometimes subscriber looks on black screen, but audio is still available. In my case this issue is reproducing even the publisher just publish the stream and rotate the device.
[In publish mode]: I was reproduce black screen when start stream with device portrait orientation, then put the device on leftLanscape mode, then put the device on the table to change device orientation to faceUp.
The record orientation is portrait.
switch recordOrientation { case .portrait: return .left case .portraitUpsideDown: return .right case .landscapeLeft, .faceUp, .faceDown: return .up case .landscapeRight: return .down case .unknown: return .up }
Also i have enabled publisher.audioFallbackEnabled = true
And the screen of the network monitor when the stream freeze.
PS. I use image buffer delegate and consume frames. Each frame from publisher is correct, not black.
I think there is a problem with network or breaking of orientation?
<img width="1160" alt="screenshot 2019-02-01 at 13 19 22" src="https://user-images.githubusercontent.com/8685210/52117531-00c3fb00-2625-11e9-8983-ed4785ce1c65.png"
The last session where issue was reproduced "sessionId": "1_MX40NjE3NjQ5Mn5-MTU0OTAxODU1Nzc2N344RTF4SUpmd09QYjhHNWhPanBVTUtZWlB-fg" "archiveId": "a2d49121-988e-40b5-bc4a-b336013a5c89", "streamId": "C04E2079-F807-48BA-B1CC-D0F0B7BA950C", iOS 11.01(15A402) iPhone 6+
For what it's worth, I'm still having this issue on iOS in the latest version, but I also still haven't been able to figure out a consistent way to reproduce it.
What I do know is that when this does happen the remote side still thinks the received stream has video.
@evillemez I resolved my issue with black frames. The last steps i've made is add didDrop sampleBuffer: and reconfigure session with another preset.
https://paste.ofcode.org/VhwY9YEmnxqEqFbkepSb2N CameraController - the object that configure and manages AVCaptureSession
@Rjayone - Link is broken. Can you share it again.
@msach22 - This issue is still happening. Checking on iPhone and iPad - iOS version- 12.1.4 - XCode 10.2. Any updates for the resolution?
@evillemez - Have you figured out any solution on this?
@nagenderpratapqss We still don't have a solution, but we also still can't reliably reproduce the problem. It's odd, when it does happen, it happens consistently, but I still don't know what the initial trigger is.
@evillemez Have you check if this issue happens also in a native iOS app using the latest Opentok SDK? Just wondering if it's related to opentok-react-native or it's a bug on the iOS SDK.
@ggoldens I haven't tried this in a native iOS app, so I'm not sure.
We have seen (very rare) cases of this happening on Android also... but it was a while ago, with older versions of the lib, on old Android devices with other issues, so I'm honestly not sure it's relevant.
I wish I could reproduce this issue consistently, but still no luck.
Here is a code i used. Hope its would be helpful :)