unable to get remote stream in case when a “remote stream” is attached by calling “peer.addStream(remoteStream)

[muaz-khan]

Issue: unable to get remote stream in case when a “remote stream” is attached by calling “peer.addStream ( remoteStream )”

Is it beneficial to attach remote stream?

Yeah, it allows us overcome burden from a single peer. Burden will be shared.

Also, we can support a wide range of peer’s connectivity.

How to attach remote stream?

var MediaStream = window.webkitMediaStream || window.MediaStream;

firstPeer.onaddstream = function(remoteSteam) {
      remoteStream = new MediaStream(remoteSteam.audioTracks, remoteSteam.videoTracks);
      otherPeer.addStream(remoteStream);  /* attaching remote stream */
};

What I want to do?

1. First peer will handle first three peers
2. 2nd peer will handle next three peers (5, 6, 7)
3. 5th peer will handle next three peers (8,9,10)
4. 8th peer will handle next three peers (11, 12, 13)
5. And so on.

We will get following benefits (in case of success):

1. Video will never freeze for 11th and upper peers
2. 1st peer don’t need to handle all participants
3. You don’t need to install/buy a middle server (like Asterisk) for small projects (to support a few hundred peers’ connectivity)

A known bug in chromium: peer.onicecandidate not fires for 11th peer. You don’t need to worry about that bug!!

Pitfall: If peer number 1, 2, 5, 8 … tries to leave the room without informing other peers to play a host role.

If peer number 2 leaves the room, without informing master peer or peer number 5 to handling upcoming peers.

A demo experiment to test it:

https://googledrive.com/host/0B6GWd_dUUTT8V1Fodm9WQldkb28/

Open 4 tabs....1st table should create room……..join room from other tabs.....see the behavior of the 4th tab.

[muaz-khan]

unable to generate ice candidates (on the answerer side) when attaching remote stream..

a=candidate:2437072876 1 udp 2113937151 192.168.1.2 54082 typ host generation 0 a=candidate:2437072876 2 udp 2113937151 192.168.1.2 54082 typ host generation 0 a=candidate:2437072876 1 udp 2113937151 192.168.1.2 54082 typ host generation 0 a=candidate:2437072876 2 udp 2113937151 192.168.1.2 54082 typ host generation 0 a=candidate:941443129 1 udp 1845501695 39.47.69.177 10136 typ srflx raddr 192.168.1.2 rport 54082 generation 0 a=candidate:941443129 2 udp 1845501695 39.47.69.177 10136 typ srflx raddr 192.168.1.2 rport 54082 generation 0 a=candidate:941443129 1 udp 1845501695 39.47.69.177 10136 typ srflx raddr 192.168.1.2 rport 54082 generation 0 a=candidate:941443129 2 udp 1845501695 39.47.69.177 10136 typ srflx raddr 192.168.1.2 rport 54082 generation 0

--------offer sdp provided by offerer

v=0 o=- 2216700829 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS m=audio 1 RTP/SAVPF 103 104 111 0 8 107 106 105 13 126 c=IN IP4 0.0.0.0 a=rtcp:1 IN IP4 0.0.0.0 a=ice-ufrag:Yy0jsw+XcU7Ovmw4 a=ice-pwd:9kjBUxNEoOOO0yViUNEH9Yhe a=ice-options:google-ice a=fingerprint:sha-256 ED:B9:CF:19:5F:A3:F4:D3:31:FC:F5:18:3C:AF:60:9E:B1:52:A5:32:69:8A:22:63:05:AD:98:93:ED:33:2A:7E a=recvonly a=mid:audio a=rtcp-mux a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:CFkD+WaozWTFg41l4mi5rdoAscTYMJa8CB1HgVpW a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:111 opus/48000/2 a=fmtp:111 minptime=10 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:107 CN/48000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:126 telephone-event/8000 a=maxptime:60 m=video 1 RTP/SAVPF 100 116 117 c=IN IP4 0.0.0.0 a=rtcp:1 IN IP4 0.0.0.0 a=ice-ufrag:Yy0jsw+XcU7Ovmw4 a=ice-pwd:9kjBUxNEoOOO0yViUNEH9Yhe a=ice-options:google-ice a=fingerprint:sha-256 ED:B9:CF:19:5F:A3:F4:D3:31:FC:F5:18:3C:AF:60:9E:B1:52:A5:32:69:8A:22:63:05:AD:98:93:ED:33:2A:7E a=recvonly a=mid:video a=rtcp-mux a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:CFkD+WaozWTFg41l4mi5rdoAscTYMJa8CB1HgVpW a=rtpmap:100 VP8/90000 a=rtpmap:116 red/90000 a=rtpmap:117 ulpfec/90000 RTCPeerConnection-v1.4.js:191

--------answer

sdp: v=0 o=- 315182876 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS m=audio 1 RTP/SAVPF 103 104 111 0 8 107 106 105 13 126 a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:Do0n1o1bjqhwgfo7jsfqBi2pgBx9i9egkjngrjkf c=IN IP4 0.0.0.0 a=rtcp:1 IN IP4 0.0.0.0 a=ice-ufrag:ZQglAeIL9/0MtR+R a=ice-pwd:gbAu8knZaPhZrBQc9mDjK2AT a=fingerprint:sha-256 50:DD:32:5B:31:92:C0:CF:77:2A:DC:66:45:29:6C:58:73:A8:77:AF:10:38:D0:FA:53:D9:55:26:0A:DC:B2:6A a=recvonly a=mid:audio a=rtcp-mux a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:111 opus/48000/2 a=fmtp:111 minptime=10 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:107 CN/48000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:126 telephone-event/8000 a=maxptime:60 m=video 1 RTP/SAVPF 100 116 117 a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:Do0n1o1bjqhwgfo7jsfqBi2pgBx9i9egkjngrjkf c=IN IP4 0.0.0.0 a=rtcp:1 IN IP4 0.0.0.0 a=ice-ufrag:ZQglAeIL9/0MtR+R a=ice-pwd:gbAu8knZaPhZrBQc9mDjK2AT a=fingerprint:sha-256 50:DD:32:5B:31:92:C0:CF:77:2A:DC:66:45:29:6C:58:73:A8:77:AF:10:38:D0:FA:53:D9:55:26:0A:DC:B2:6A a=recvonly a=mid:video a=rtcp-mux a=rtpmap:100 VP8/90000 a=rtpmap:116 red/90000 a=rtpmap:117 ulpfec/90000

Maybe RTP ports already opened....We may need to change them dynamically.

[muaz-khan]

Latest one-page demo on attaching remote media streams. Still fails.

[peili]

Any process on this issue so far?

[muaz-khan]

This issue is not fixed yet. Attachment of remote media streams still failing on both chrome and firefox. I tried audio-only remote streams too but no success.

[peili]

Do you know when Google/Mozilla are going to support this feature? I couldn't find clear informations on the roadmaps..

[muaz-khan]

I'm not sure, however it seems that audio-only remote streams' attachment works fine.

[muaz-khan]

A general comment is that we only support one audio source today in Chrome given that we only have one ADM. To me it sounds like we need a new type of source here where the audio is driven by the remote peer. Perhaps it would be less complicated to build something initially if we only worked in relay mode (one source).

Reference: https://code.google.com/p/webrtc/issues/detail?id=383

.....take a remote stream from one peer connection and adding it to another peer connection and having it Just Work™ for that second peer.

Implementation awaited!

[RonMen]

Hi @muaz-khan,

I think this is what I thought about on my other issue questioning. The use case is:

1.) Peer1 connects with audio only 2.) Peer2 connects with audio only ... later on one/both switch 3.) Peer1/Peer2 switches over to remove audio only stream and instead add a video+audio stream

In this case I get a black video on the peer this stream is delivered to sending a "new" offer and there are removeStream + addStream events fired up but also the data of the newly added stream isn't send.

I tried multiple settings and also tried to find an answer but it doesn't seem to work. Also the sample implementation presented in the webrtc issue tracker isn't working like expected: https://code.google.com/p/webrtc/issues/detail?id=1086

Do you have any idea how to bring this up and running or is there something missing in browsers webrtc implementation for now?

[muaz-khan]

Try this demo: Switching streams from screen-sharing to audio+video at runtime. (Renegotiation)

1. Both peers are sharing screen
2. You can switch to audio+video stream; old stream (i.e. screen-sharing) will be removed.

Don't need to add ice candidates when renegotiating.

[RonMen]

Dear @muaz-khan, thx for your sample. I also found something else since setting the localDescription failed while switching like explained here: https://groups.google.com/forum/#!msg/discuss-webrtc/ets-eex6H-0/bkN2UwZI-gEJ

I changed the {optional: [{DtlsSrtpKeyAgreement: true}]} of the peerconnection and I tried without it and the streams are switched well now.

[skanagavelu]

We need this idea. This communication is low cost for end user. Mauz; How this is supported for screen share?

[skanagavelu]

Hi Muaz,

In your example below

firstPeer.onaddstream = function(remoteSteam) { remoteStream = new MediaStream(remoteSteam.audioTracks, remoteSteam.videoTracks); otherPeer.addStream(remoteStream); /* attaching remote stream */ };

Why we are recreating remoteStream again using new MediaStream ?

And pls tell me what we have to do for screen share while creating new instance of MediaStream...?

[muaz-khan]

There are some bugs preventing video tracks to be forwarded. In screen sharing apps; we need to retransmit video RTP packets; which seems failing on both chrome and firefox.

In the example; new media stream is created; because in old days, chrome were throwing not-supported exceptions for MediaStream because only LocalMediaStream objects were permitted to be attached using peer.addStream method.

Nowadays, you can attach remote stream directly, on chrome; without initiating new object; because both local and remote streams have identical MediaStream type.

I think identical ports (RTP or UDP) are causing failures to attach remote media streams. Though, I don't know the exact issue. Maybe ICE Agent is unable to locate right ports to retransmit remote media stream.

Maximum peer connections limit is 256; so it is preferred to use application specific bandwidth parameters to stream low-quality packets; also specify width/height constraints to handle freezing issues.

[skanagavelu]

FYI : https://groups.google.com/forum/#!topic/discuss-webrtc/CnEHFxjxE_8

[peili]

Any updates on "attaching remote audio/video stream"? Audio seems to work. Video not?

[skanagavelu]

https://code.google.com/p/webrtc/issues/detail?can=2&start=0&num=100&q=&colspec=ID%20Pri%20Mstone%20ReleaseBlock%20Area%20Status%20Owner%20Summary&groupby=&sort=&id=383

[skanagavelu]

https://code.google.com/p/webrtc/issues/detail?id=2192

[abhishanksahu]

Hi Muaz, Is there any progress in this issue? I am able to attach remote streaming on the same page. But when I try to do the same over two different pages it is failing.

[muaz-khan]

It works. Try following demo on chrome stable (desktop):

<script src="//www.webrtc-experiment.com/RTCMultiConnection-v1.6.js"> </script>

<button id="open-main-session">open main session</button>
<button id="forward-main-session">forward main session</button><br />

<button id="join-main-session">join main session</button>
<button id="join-forwarded-session">join forwarded session</button>
<script>
var h1 = document.createElement('h1');
h1.innerHTML = 'user-id: ' + prompt('What is Your Name');
document.body.insertBefore(h1, document.body.firstChild);

// http://www.rtcmulticonnection.org/docs/constructor/
var mainConnection = new RTCMultiConnection();

// http://www.rtcmulticonnection.org/docs/session/
mainConnection.session = {
    video: true,
    oneway: true
};

// http://www.rtcmulticonnection.org/docs/constructor/
var dummyConnection = new RTCMultiConnection('dummy-connection');

// http://www.rtcmulticonnection.org/docs/session/
dummyConnection.session = {
    video: true,
    oneway: true
};

// http://www.rtcmulticonnection.org/docs/dontAttachStream/
dummyConnection.dontAttachStream = true;

// http://www.rtcmulticonnection.org/docs/onstream/
mainConnection.onstream = function(e) {
    // see this line--------------------------
    dummyConnection.attachStreams.push(e.stream);
    document.body.appendChild(e.mediaElement);
};

// http://www.rtcmulticonnection.org/docs/onstream/
dummyConnection.onstream = function(e) {
    alert('Wow, got forwarded stream!');
    document.body.appendChild(e.mediaElement);
};

document.querySelector('#join-main-session').onclick = function() {
    this.disabled = true;

    // http://www.rtcmulticonnection.org/docs/connect/
    mainConnection.connect();
};

document.querySelector('#join-forwarded-session').onclick = function() {
    this.disabled = true;

    // http://www.rtcmulticonnection.org/docs/connect/
    dummyConnection.connect();
};

document.querySelector('#open-main-session').onclick = function() {
    this.disabled = true;

    // http://www.rtcmulticonnection.org/docs/open/
    mainConnection.open();
};

document.querySelector('#forward-main-session').onclick = function() {
    this.disabled = true;

    // http://www.rtcmulticonnection.org/docs/open/
    dummyConnection.open();
};
</script>

dummyConnection is used to forward media stream. Main forwarding stuff happened here:

mainConnection.onstream = function(e) {
    dummyConnection.attachStreams.push(e.stream);
};

[abhishanksahu]

Hi Muaz,

Thanks for your help, I tried this demo and its work fine for video, but when I enable audio also, on the forwarded stream audio is missing, there is no sound from the third tab. Can you tell me why?

Thank You

[muaz-khan]

Did you appended audio:true in the session object?

mainConnection.session = {
    audio: true,
    video: true,
    oneway: true
};

[abhishanksahu]

Yeah I did that, on the second tab I am able to listen audio, but when on third tab I join the forwarded session, I can only see the video and there is no sound coming from it. With latest version of chrome u can see which tab is playing audio right? There was no such sign over this tab.

On 5 February 2014 22:52, Muaz Khan [email protected] wrote:

Did you appended audio:true in the sessionhttp://www.rtcmulticonnection.org/docs/session/object?

mainConnection.session = { audio: true, video: true, oneway: true};

Reply to this email directly or view it on GitHubhttps://github.com/muaz-khan/WebRTC-Experiment/issues/2#issuecomment-34213645 .

[harshg0910]

@muaz-khan @abhishanksahu I did some experiments with audio only : Computer 1 :

1. Opened the main session in one tab
2. Joined the main session and forwarded it in the other tab of same computer. Computer 2
3. Joined the forwarded session but the audio player was there with no sound.
4. Joined the main session in other tab and I was able to hear the sound.
5. Now the tab containing the forwarded stream was also playing the sound ( I checked it by pausing the sound of the main session tab).
6. I close the tab with main session, the sound from the forwarded session also dies.

Seems interesting as I am unable to find the reason for failure of forwarded stream at the same time its relation with main stream.

[pandalaw]

Hi Muaz,

I tried to forward stream of screen, but it failed. Can you tell me why? Computer 2 cannot get the screen of computer 1 as well.

Thank You

[muaz-khan]

I tried this demo; changed this line:

mainConnection.session = {
    screen: true,
    oneway: true
};

It worked when I shared screen from canary. In the below gif; you can see that I tested between chrome canary, chrome stable and opera next 19.0.

[abhishanksahu]

Hi Muaz,

Did you find any remedy for the problem I described you (no sound from the forwarded stream).

Thanks.

On 5 February 2014 22:56, Abhishank Sahu [email protected] wrote:

Yeah I did that, on the second tab I am able to listen audio, but when on third tab I join the forwarded session, I can only see the video and there is no sound coming from it. With latest version of chrome u can see which tab is playing audio right? There was no such sign over this tab.

On 5 February 2014 22:52, Muaz Khan [email protected] wrote:

Did you appended audio:true in the sessionhttp://www.rtcmulticonnection.org/docs/session/object?

mainConnection.session = { audio: true, video: true, oneway: true};

Reply to this email directly or view it on GitHubhttps://github.com/muaz-khan/WebRTC-Experiment/issues/2#issuecomment-34213645 .

[muaz-khan]

See this comment:

Due to some architecture problems in the chromium remote audio track implementation, today we are always mixing the data in WebRtc, and passing the mixed data to Chrome. This ways prevents us getting the correct stream for the specific audio track, and it also blocks us feeding the correct remote audio track data to the peer connection.

After talking to a couple of colleagues about this feature, we decided to postpone this feature until we fix the architecture problems.

That's why remote audio recording doesn't work.

[pandalaw]

Hi Muaz,

I tried this demo. and changed line 135

mainConnection.session = {
    video: true,
    audio: true,
    screen: true,
    oneway: true
};

When I joined forwarded session, I can get screen stream only.

Please help me to find out the problem :)

Thank you.

[muaz-khan]

After looking into logs; it is appeared that only single "remote" media stream is attached to peer connection object:

In theory, it MUST attach both audio+video and screen streams. You can see that we are trying to attach both:

After going a little bit deeper; it is appeared that both remote streams has identical ids and labels. It can be RTCMultiConnection implementation issue.

[shtefcs]

Hi Muaz,

Is this problem fixed by chrome guys ?