elixir-webrtc
Negotation issues between browsers and elixir webrtc
I have a scenario where I'm trying to connect a client (browser) to a server using Elixir WebRTC. The PeerConnection on the server side does not accept any video codecs; the server is audio only. I noticed problems with negotiation in a specific scenario where we first add only video and negotiate for it, which rejects the video track (audio-only server), and then we add audio (to the same PeerConnection) and start negotiation again, which fails (the browser does not accept the SDP answer, or an ICE candidate).
In the case of the Chrome browser, the reason seems to be sending a value of 0 in the m-line port in response to the first negotiation. As a result, the second SDP offer does not contain the video track that was sent in the first offer. For comparison, Chrome in the SDP answer sets the m-line port value to 9, thanks to which the second offer contains both tracks.
In the case of the Firefox browser, after applying the second SDP answer, the next ICE candidate from the server causes an error. Firefox sends the correct SDP offer (unlike Chrome) hence accepts the SDPAnswer. It seems to me that the problem with the error during the candidate application is that in the first SDP answer the m-line port was set to 0 by the server and it should not then send ICE candidates for that m-line.
Interestingly, Firefox (unlike Chrome) returns in the SDP answer an m-line port equal to 0 for video (just like Elixir WebRTC).
Code used to check how browsers behave in a similar situation:
pc1 = new RTCPeerConnection();
pc2 = new RTCPeerConnection();
pc1_tr1 = pc1.addTransceiver("video", {direction: "sendonly"});
offer = await pc1.createOffer();
await pc1.setLocalDescription(offer);
await pc2.setRemoteDescription(offer);
pc2.getTransceivers()[0].stop()
answer = await pc2.createAnswer();
await pc2.setLocalDescription(answer);
await pc1.setRemoteDescription(answer);
pc1_tr2 = pc1.addTransceiver("audio", {direction: "sendonly"});
offer2 = await pc1.createOffer();
await pc1.setLocalDescription(offer2);
await pc2.setRemoteDescription(offer2);
answer2 = await pc2.createAnswer();
await pc2.setLocalDescription(answer2);
await pc1.setRemoteDescription(answer2);
Looking at Firefox, it also includes port 9 and a=inactive. There is no port 0.
Sections that may have something to do with this behavior:
- https://www.rfc-editor.org/rfc/rfc8843#section-a.2
- https://www.w3.org/TR/webrtc/#dom-rtcrtptransceiver-stop
In particular, in the W3C there is:
A stopping transceiver will cause future calls to createOffer to generate a zero port in the media description for the corresponding transceiver, as defined in [RFC9429] (section 4.2.1.) (The user agent MUST treat a stopping transceiver as stopped for the purposes of RFC9429 only in this case). However, to avoid problems with [RFC8843], a transceiver that is stopping, but not stopped, will not affect createAnswer.
Here is also an interesting example:
pc1 = new RTCPeerConnection();
pc2 = new RTCPeerConnection();
pc1_tr1 = pc1.addTransceiver("audio", {direction: "sendonly"});
offer = await pc1.createOffer();
await pc1.setLocalDescription(offer);
await pc2.setRemoteDescription(offer);
pc2.getTransceivers()[0].stop();
answer = await pc2.createAnswer();
await pc2.setLocalDescription(answer);
console.log(pc2.getTransceivers());
await pc1.setRemoteDescription(answer);
console.log(pc1.getTransceivers());
stream = await navigator.mediaDevices.getUserMedia({audio: "true", video: "true"})
pc1.addTrack(stream.getAudioTracks()[0], stream);
offer = await pc1.createOffer();
console.log(offer.sdp);
await pc1.setLocalDescription(offer);
console.log(pc1.getTransceivers());
await pc2.setRemoteDescription(offer);
answer = await pc2.createAnswer();
await pc2.setLocalDescription(answer);
console.log(pc2.getTransceivers());
console.log(answer.sdp);
The first script in ex_webrtc
alias ExWebRTC.PeerConnection
{:ok, pc1} = PeerConnection.start_link()
{:ok, pc2} = PeerConnection.start_link()
{:ok, _transceiver_1_1} = PeerConnection.add_transceiver(pc1, :video, direction: :sendonly)
{:ok, offer} = PeerConnection.create_offer(pc1)
:ok = PeerConnection.set_local_description(pc1, offer)
:ok = PeerConnection.set_remote_description(pc2, offer)
[transceiver_2_1] = PeerConnection.get_transceivers(pc2)
:ok = PeerConnection.stop_transceiver(pc2, transceiver_2_1.id)
{:ok, answer} = PeerConnection.create_answer(pc2)
:ok = PeerConnection.set_local_description(pc2, answer)
:ok = PeerConnection.set_remote_description(pc1, answer)
{:ok, _transceiver_2} = PeerConnection.add_transceiver(pc1, :audio, direction: :sendonly)
{:ok, offer2} = PeerConnection.create_offer(pc1)
:ok = PeerConnection.set_local_description(pc1, offer2)
:ok = PeerConnection.set_remote_description(pc2, offer2)
{:ok, answer} = PeerConnection.create_answer(pc2)
:ok = PeerConnection.set_local_description(pc2, answer)
:ok = PeerConnection.set_remote_description(pc1, answer)
PeerConnection.get_local_description(pc2)
With the following changes the echo example from ex_webrtc works on chrome:
diff --git a/examples/echo/lib/echo/peer_handler.ex b/examples/echo/lib/echo/peer_handler.ex
index 550c215..f3a2e33 100644
--- a/examples/echo/lib/echo/peer_handler.ex
+++ b/examples/echo/lib/echo/peer_handler.ex
@@ -16,11 +16,6 @@ defmodule Echo.PeerHandler do
]
@video_codecs [
- %RTPCodecParameters{
- payload_type: 96,
- mime_type: "video/VP8",
- clock_rate: 90_000
- }
]
@audio_codecs [
@@ -42,15 +37,12 @@ defmodule Echo.PeerHandler do
)
stream_id = MediaStreamTrack.generate_stream_id()
- video_track = MediaStreamTrack.new(:video, [stream_id])
audio_track = MediaStreamTrack.new(:audio, [stream_id])
- {:ok, _sender} = PeerConnection.add_track(pc, video_track)
{:ok, _sender} = PeerConnection.add_track(pc, audio_track)
state = %{
peer_connection: pc,
- out_video_track_id: video_track.id,
out_audio_track_id: audio_track.id,
in_video_track_id: nil,
in_audio_track_id: nil
@@ -168,13 +160,6 @@ defmodule Echo.PeerHandler do
end
defp handle_webrtc_msg({:rtp, id, rid, packet}, %{in_video_track_id: id} = state) do
- # rid is the id of the simulcast layer (set in `priv/static/script.js`)
- # change it to "m" or "l" to change the layer
- # when simulcast is disabled, `rid == nil`
- if rid == "h" do
- PeerConnection.send_rtp(state.peer_connection, state.out_video_track_id, packet)
- end
-
{:ok, state}
end
diff --git a/examples/echo/priv/static/script.js b/examples/echo/priv/static/script.js
index 23f5b50..98f8fd7 100644
--- a/examples/echo/priv/static/script.js
+++ b/examples/echo/priv/static/script.js
@@ -9,6 +9,7 @@ ws.onopen = _ => start_connection(ws);
ws.onclose = event => console.log("WebSocket connection was terminated:", event);
const start_connection = async (ws) => {
+ var audioAdded = false;
const pc = new RTCPeerConnection(pcConfig);
// expose pc for easier debugging and experiments
window.pc = pc;
@@ -32,7 +33,15 @@ const start_connection = async (ws) => {
});
// replace the call above with this to disable simulcast
// pc.addTrack(localStream.getVideoTracks()[0]);
- pc.addTrack(localStream.getAudioTracks()[0]);
+
+ const addAudioTrack = async (pc) => {
+ pc.addTrack(localStream.getAudioTracks()[0]);
+ const offer = await pc.createOffer();
+ await pc.setLocalDescription(offer);
+ console.log("Sent SDP offer:", offer)
+ ws.send(JSON.stringify({ type: "offer", data: offer }));
+ audioAdded = true;
+ }
ws.onmessage = async event => {
const {type, data} = JSON.parse(event.data);
@@ -41,6 +50,9 @@ const start_connection = async (ws) => {
case "answer":
console.log("Received SDP answer:", data);
await pc.setRemoteDescription(data)
+ if (!audioAdded) {
+ addAudioTrack(pc);
+ }
break;
case "ice":
console.log("Received ICE candidate:", data);
I didn't take a look at these changes but just saw this issue and it might be related: https://github.com/w3c/webrtc-pc/issues/2927
We should not set the port to zero when creating answer if the transceiver is stopping.
In the W3C specification, that is described in the stop method documentation:
https://www.w3.org/TR/webrtc/#methods-8
When m-line is rejected it's port has to be set to 0, according to RFC9429 section 5.3.1-15. This happens e.g. when no compatible media codecs could be found.