OpenTok / TokBox support

[noodlemichaelmartin]

After compiling and running it on my iPhone, I attempted to load a page that I've been working on that uses the OpenTok/TokBox libraries (WebRTC-based video chat platform).

The page loads with no errors, but the pre-detection of whether or not WebRTC is present fails. The variables/objects that it's looking for are below:

if (!navigator.mozGetUserMedia && !navigator.webkitGetUserMedia) {
...
return false;
}

If I remove the pre-detection, I get a JS error that prevents the page from fully loading.

I'm assuming that perhaps the functions are implemented differently in Bowser, so I have the following questions about the WebRTC implementation in Bowser:

1. Were navigator.mozGetUserMedia or nagivator.webkitGetUserMedia implemented under different object/method names?

2. Is there any documentation regarding the potential differences in building web pages for Bowser?

3. Where in the source would the definition of the methods attached to the nagivator object be found?

It's possible that TokBox/OpenTok would not be compatible with your WebRTC implementation or that perhaps only some minor shims would be required to use the correct object/method names - I'm just trying to determine in which direction I should go.

[superdump]

1. navigator.getUserMedia should be present and it seems we also overwrite navigator.webkitGetUserMedia: https://github.com/EricssonResearch/openwebrtc/blob/master/bridge/client/webrtc.js#L1224 (note there and just below, also note the FIXME)

2. Not yet but that is a good idea. I think we may be able to put that together and it would help in general.

3. See above. bridge/client/webrtc.js it seems.

Thanks for the feedback, we're definitely interested in either fixing bugs to allow interoperability with higher-level WebRTC application frameworks or to investigate them to get them to fix mistakes in WebRTC API usage.

[stefanalund]

If you are comfortable, and can, share a URL for your app we can also take it for a spin.

[noodlemichaelmartin]

Unfortunately, I cannot - it's not public.

I can say that we're using the standard implementation of TokBox, which is basically a paid version of the "OpenTok" standards (they host the TURN/STUN services and all the streams are routed through their Media server to enable things like recording and changes in frame rates or stream resolution).

If you want to test the libraries themselves, they do let you setup a free 30-day trial developer account prior to setting up billing - https://tokbox.com/

The only thing that I do in my implementation that's not present in the out of the box TokBox implementation, is the above JS detection to put a custom message into the DOM for the user. When I remove the JS detection code, it causes a JS error that prevents the rest of the JS on the page from executing. I can't seem to see what the error is, though. It should be noted that while the navigator object is defined, neither the navigator.mozGetUserMedia nor the navigator.webkitGetUserMedia are not defined, when tested via JavaScript in Bowser.

Is it possible that this is just a compilation issue of some kind or is it just that it's throwing exceptions / errors that I just can't see within Bowser?

[stefanalund]

You can use the Developer Tools in Safari to debug errors in Bowser (UIWebViews):

https://github.com/EricssonResearch/bowser#debugging-webrtc-scripts

[noodlemichaelmartin]

I'm developing this on an iPhone, which does have Inspection enabled, but I don't see any additional options or tools that seem to affect Bowser. Also, Bowser does not run in the simulator, so I can't use any debugging tools there either..

When running the JS in any platform that currently does not support WebRTC (IE, Safari, etc), it does not produce any actual JavaScript errors - it's accurately detected either by my JS (the IF statement above) or, even if I remove that IF statement, it's still cleanly detected by the TokBox JS libraries and informs the user properly.

In Bowser, when I omit my JS detection (the IF statement above) code, it instead produces a JS error or exception that I cannot see, which breaks the page. This implies that the library is likely incompatible with Bowser, but more importantly both navigator.mozGetUserMedia and navigator.webkitGetUserMedia are still "undefined" when I check for them in my detection code. So, if those two methods are "undefined", then it stands to reason that it's still not going to work anyways.

I compiled Bowser from the Git repository and did not modify it, with the exception of setting the developer account in the settings. Is there maybe something missing?

[noodlemichaelmartin]

Actually, additional information - that error was due to something unrelated. Now that it's suppressed, the only thing it's telling me is that "This browser is not compatible with WebRTC" from the TokBox platform.

[superdump]

So the getUserMedia API is found now? But TokBox says that the browser is not compatible with WebRTC? Do you know if they do any user agent sniffing?

[noodlemichaelmartin]

No, that is not correct - GetUserMedia is not found (it's undefined) - TokBox is displaying that message because it cannot find GetUserMedia.

The error that I mentioned above ("In Bowser, when I omit my JS detection (the IF statement above) code, it instead produces a JS error or exception that I cannot see, which breaks the page.") was unrelated TokBox, it was a bug in my code that I had never seen, because my JS detection code was always in the way of it. When I fixed the error in my JS, the TokBox platform was able to load normally and it agreed with my JS detection code and said that Bowser was not WebRTC compatible (because GetUserMedia was not found).

[superdump]

Does this getUserMedia test app or our point-to-point audio/video test app work?

[stefanalund]

Looks like the Google getUserMedia test app has moved to http://googlechrome.github.io/webrtc/samples/web/content/getusermedia/gum/

[noodlemichaelmartin]

Tested with that sample GUM page above and it does work, however, when I load my page, none of that seems to be defined.

In fact, I added this to the top of all of my other JS includes in

to see if maybe one of them was overriding the function directly, but instead, each of the following alerts "undefine" for each of the alert lines below):

<!doctype html>
<html>
    <head>
    <script type="text/javascript">
        alert('before ANY other scripts IN theme.tpl...');
        alert(navigator.getUserMedia);
        alert(navigator.mozGetUserMedia);
        alert(navigator.webkitGetUserMedia);
        alert(window.webkitRTCPeerConnection);
        alert(window.mozRTCPeerConnection);
    </script>
   ...

Is there something I'm missing here or does it just look like none of this is defined?

[noodlemichaelmartin]

Ok, so now that I've ripped apart a non-minified version of opentok.js from TokBox, I can see where it's truly failing:

1. The first failure is on Browser version. It's detecting webkit version 27, but a minimum of webkit 34 is required.

2. It fails when checking the capabilities of webkitRTCPeerConnection, as both of these conditions of this if statement return false:

if (typeof(window.webkitRTCPeerConnection) === 'function' && !!window.webkitRTCPeerConnection.prototype.addStream)

If I had to guess, I'd say that the reason is entirely due to the webkit version.

An chance of upgrading to webkit 34 or higher?

[stefanalund]

That's easy actually: The UserAgent is set in BowserAppDelegate. It's actually set to 'not Chrome' but I guess that is ignored :-)

[noodlemichaelmartin]

Yes, I see that #define KUserAgent @"xxxx" line, but that's not the only issue.

In opentok.js, I actually removed the "return" part of the if statement referenced in point # 1 above so that the function just ignores the version requirement (temporary hack for debugging). However, even ignoring the version deficiency, as it continues with the requirement checks, it then fails on the very next requirement (listed as point # 2 above).

I was only speculating that if the webkit version is actually, physically 27, then perhaps the functionality it's trying to use is not present?

[Rugvip]

if (typeof(window.webkitRTCPeerConnection) === 'function' && !!window.webkitRTCPeerConnection.prototype.addStream)

This check fails because addStream is not a property of the prototype in our implementation, but instead assigned directly to the peer connection object.

Regarding the webkit version it will be whatever the iOS WebView is using. We are building a part of WebKit as a dependency (JavaScriptCore), but that's only being built for Linux and Android to be used as JavaScript engine.

[noodlemichaelmartin]

So then the nature of the check is technically invalid? As in they're assuming it's defined in the prototype, but you're just attaching it to the object directly? If that's the case, I can hack that...

[noodlemichaelmartin]

I can see where it's referencing the .addStream() method, but I can't seem to locate where you've attached that method. I'll just manually assign window.webkitRTCPeerConnection.prototype.addStream = [the function], if I can locate said function. How can I reference it?

[Rugvip]

window.webkitRTCPeerConnection.prototype.addStream = function() {} should be enough, it'll use the correct function anyway

[noodlemichaelmartin]

Adding window.webkitRTCPeerConnection.prototype.addStream = function() {} to the JS before it checks the requirements, basically only allows you to get past the requirement checks. When it tries to load the video chat box, it never actually connects or does anything. I get the normal loader box that it displays before you grant permissions to use the camera/microphone, but it never asks for permissions and it never loads the video stream. Attached a screen shot of what I see in Bowser and also what I see in FireFox or Chrome at the same URL.

[noodlemichaelmartin]

I should point out that it IS detecting the number of streams being published and will show multiple video boxes for each browser that's got the page loaded into it, but it's never able to render the video streams inside the boxes.

[Rugvip]

are you rendering a self-view or only remote streams?

[noodlemichaelmartin]

Both - It dynamically detects if new streams are published. In fact, right now, I was testing - I closed all browsers that had it open, opened the test page in Bowser and it had only 1 box (the larger, main box) and then a waited a few seconds. Then, I opened the page on my laptop and started streaming. On my phone, it rendered another (smaller) video box below the main one, as expected. Problem is that neither of them are physically rendering the stream - locally or remotely. It appears that the normal data exchanges and normal logging events are being fired back and forth, as designed, but the rendering of the video streams is just not happening. I'm guessing it's still related to the addStream issue that we've been discussing, but i could be wrong - it could be related to something else that I'm not seeing.

[stefanalund]

Are you using the latest OpenWebRTC? A couple of important interoperability bugs were fixed about 2 days ago: https://github.com/EricssonResearch/openwebrtc/wiki/Building-OpenWebRTC#update-openwebrtc

[noodlemichaelmartin]

Attempting a re-build now, will continue to test/debug. After trapping some errors with try/catch, I can get it to publish a video stream locally, but none of the peer-to-peer or routed connections are working, nor remote streams rendering locally. I'll test after the rebuild and continue to debug. Thanks for the update.

[noodlemichaelmartin]

Going back to our previous point of discussion, it appears that none of these have a function called AddStream, either natively or in the prototype, so where is your addStream function?

These are all undefined:

webkitRTCPeerConnection.prototype.addStream RTCPeerConnection.prototype.addStream webkitRTCPeerConnection.addStream RTCPeerConnection.addStream

[Rugvip]

It's assigned once the object is created, try new webkitRTCPeerConnection({iceServers:[]}).addStream

[noodlemichaelmartin]

This is difficult to put in a "small" post, but I'll try.

First, @Rugvip, I hacked the RTCPeerConnection objects to have the addStream function you referenced above:

            if(window.webkitRTCPeerConnection) {
                var tempHack = new webkitRTCPeerConnection({iceServers:[]});
                window.webkitRTCPeerConnection.prototype.addStream = tempHack.addStream;
                window.RTCPeerConnection.prototype.addStream = tempHack.addStream;
            }

I'm not 100% sure if it worked yet, because I'm iterating through the TokBox/OpenTok code and finding other issues, so this might have solved a problem, but I can't be sure yet.

Prior to debugging further issues, the video and audio stream was rendering locally in Bowser on my iPhone, but it still wasn't publishing to the stream or TokBox. It was notifying the other participants that I connected, but no stream ever came through. So, I started troubleshooting / stepping through some JS errors I was catching via try/catch. I was able to work my through or hack my way through those, but I've come to a point where one particular issue has me stuck.

I started troubleshooting a MediaStreamTrack object issue that was throwing JS errors. MediaStreamTrack objects have an "enabled" property (boolean) that controls whether or not the individual track is enabled. TokBox had the process of updating the enabled property wrapped in a function that they attached to the prototype:

if (window.MediaStreamTrack && typeof(window.MediaStreamTrack) !== 'undefined') {
        if (!window.MediaStreamTrack.prototype.setEnabled) {
          window.MediaStreamTrack.prototype.setEnabled = function (enabled) {
            this.enabled = OT.$.castToBoolean(enabled);
          };
        }
      }

In FireFox and Chrome, window.MediaStreamTrack and window.MediaStreamTrack.prototype exist and this code works, attaching the nearly superfluous function to the prototype.

In Bowser, window.MediaStreamTrack and window.MediaStreamTrack.prototype are not defined in this context, so it does not attach setEnabled to the prototype and it was breaking the JS code.

At some point in their code, it separately calls functions to get the streams of both video and audio and it loops through them and tries to toggle the "enabled" property based on whether or not a provided "muted" variable is set to true or false. Here's the pseudo code:

var videoTracks = webRTCStream.getVideoTracks();
...
var audioTracks = webRTCStream.getAudioTracks();

Then it looped like this:

for (var i=0, num=audioTracks.length; i<num; ++i) {
    audioTracks[i].setEnabled(!_muted); }

This thew a JS error when it hit the .setEnabled line, so I wrapped it in a try/catch. This was never setting audioTracks[i].enabled or videoTracks[i].enabled to boolean true, so this left the property as "undefined", but catching the error let the JavaScript was letting me continue past the problem. At that point, it was rendering my audio and video in Bowser, but it was not even attempting to publish to the stream, so other participants could not see or hear me, nor I them. With more troubleshooting, I added some additional code to ensure that it wasn't throwing the exception in the first place:

for (var i=0, num=audioTracks.length; i<num; ++i) {
    try {
        if(audioTracks[i].setEnabled) {
            audioTracks[i].setEnabled(!_muted);
        } else { 
            audioTracks[i].enabled = !_muted;
        }
    } catch(error) {
            alert(error.message);
    }
}

So at this point, it's setting the enabled property to true, which it wasn't before, but now the audio and video streams aren't rendering in Bowser anymore (which really doesn't make any sense, logically), but it IS attempting to publish the feed to the stream and throws the following error:

"Publisher PeerConnection Error: The Media resource indicated by the src attribute was not suitable," which is an error code 4 from the window.MediaError.MEDIA_ERR_SRC_NOT_SUPPORTED list.

With the window.MediaStreamTrack and window.MediaStreamTrack.prototype not being defined in Bowser, I'm not sure if there's a problem with my build of if this was done on purpose. This also raises the concern of "what else is not defined the same way between Bowser and Chrome?" and whether or not this difference has anything to do with the issue that I'm working on.

I've done a bunch of reading on various WebRTC implementations recently and I haven't been able to find an answer to the MediaStreamTrack issue.

[noodlemichaelmartin]

Also, I've been given permission to throw a copy of our testing/debugging page and scripts on my bluehost instance, so if anyone has a build of Bowser and want's to take a stab at it, maybe that will help. I can upload that tomorrow morning my time (eastern).

[adam-be]

As @Rugvip said earlier, we don't add anything to the constructor function prototypes in the JS API, all functions and attributes are added to the object when created.

webkitRTCPeerConnection.prototype.addStream is undefined, but new webkitRTCPeerConnection(config).addStream is defined.

MediaStreamTrack doesn't have public constructor (in the standard) so I'm not sure why Chrome and Firefox expose one. You can't use it since it throws "Illegal constructor", but I guess it can be used for monkey patching (like setEnabled() above).

Anyhow, there's nothing wrong with your build.

[noodlemichaelmartin]

Alright, good, so the troubleshooting continues and thanks for the verification. With MediaStreamTrack.enabled = true, it's attempting to publish but failing, which is likely related to something in RTCPeerConnection. It could be a similar situation where they're monkey patching and it's failing because of it, but they don't need to or some other type Chrome specific expectation that's in Chrome, but not exactly the same in Bowser.. I'll keep plugging through it.