RTMP推流RTC拉流时，黑屏只有声音，视频流全丢包了

[zhOrange]

1. Description(描述)

在使用原始的rtmp2rtc.conf配置文件启动srs服务时，出现以下现象：

1. 使用obs进行rtmp时，rtmp拉流、rtc拉流均可以正常播放；
2. 第一次使用RTC推流时，rtmp拉流、rtc拉流也可以正常播放；
3. 但是，一旦使用了RTC推流后，关闭RTC推流网页的话，再次使用RTMP推流，RTC拉流就不能正常播放了，只有声音可以正常播放，视频只是黑屏。通过chrome://webrtc-internals查看视频流数据时，发现视频流数据还是可以正常接收，但全都Drop掉了，解码帧数始终为0。
4. 上述场景下，在RTC拉流时出现上述异常，但同时用RTMP拉流的话，可以正常播放；另外，如果重新换回RTC推流的话，RTMP拉流、RTC拉流也都可以正常播放。

采用默认的srt2rtc.conf配置文件时，也会出现上述问题。

2. SRS Version(版本): 4.0release 5.0.28

3. SRS Log(日志):

[2022-05-14 20:59:15.164][Trace][221869][q8567159] TCP: clear zombies=1 resources, conns=3, removing=0, unsubs=0
[2022-05-14 20:59:15.164][Trace][221869][7173158y] TCP: disposing #0 resource(HttpConn)(0x557db80fab20), conns=3, disposing=1, zombies=0
[2022-05-14 20:59:15.420][Trace][221869][7173158y] RTC: session address init RTC拉流端公网IP地址:49305
[2022-05-14 20:59:15.420][Trace][221869][7173158y] RTC: session STUN done, waiting DTLS handshake.
[2022-05-14 20:59:15.421][Trace][221869][42u1778l] <- RTC RECV #12, udp 937, pps 8/14, schedule 937
[2022-05-14 20:59:15.434][Trace][221869][7173158y] DTLS: State Passive RECV, done=0, arq=0/0, r0=1, r1=0, len=157, cnt=22, size=144, hs=1
[2022-05-14 20:59:15.435][Trace][221869][7173158y] DTLS: State Passive SEND, done=0, arq=0/0, r0=-1, r1=2, len=681, cnt=22, size=82, hs=2
[2022-05-14 20:59:15.450][Trace][221869][7173158y] DTLS: State Passive RECV, done=0, arq=0/0, r0=1, r1=0, len=580, cnt=22, size=301, hs=11
[2022-05-14 20:59:15.451][Trace][221869][7173158y] DTLS: State Passive SEND, done=1, arq=0/0, r0=1, r1=0, len=554, cnt=22, size=466, hs=4
[2022-05-14 20:59:15.451][Trace][221869][7173158y] RTC: DTLS handshake done.
[2022-05-14 20:59:15.451][Trace][221869][7173158y] RTC: session pub=0, sub=1, to=30000ms connection established
[2022-05-14 20:59:15.451][Trace][221869][7173158y] RTC: Subscriber url=/live/rtcbugstream established
[2022-05-14 20:59:15.451][Trace][221869][7173158y] create consumer, no gop cache
[2022-05-14 20:59:15.451][Trace][221869][7173158y] RTC: start play url=/live/rtcbugstream, source_id=0q79mo9j/b6gv2366, realtime=1, mw_msgs=0
[2022-05-14 20:59:15.518][Trace][221869][7173158y] RTC: NACK ARQ seq=3924, ssrc=2241098775, ts=1469880, count=1/1, 1214 bytes
[2022-05-14 20:59:17.207][Trace][221869][k557z4lp] Hybrid cpu=16.98%,28MB, cid=1,0, timer=61,0,0, clock=0,37,9,1,1,0,0,0,0, objs=(pkt:494,raw:130,fua:363,msg:1031,oth:1,buf:100)
[2022-05-14 20:59:17.207][Trace][221869][k557z4lp] RTC: Server conns=1, rpkts=(1,rtp:0,stun:1,rtcp:1), spkts=(23,rtp:22,stun:1,rtcp:2), rtcp=(pli:1,twcc:2,rr:1), snk=(24,a:12,v:12,h:0), rnk=(1,1,h:9,m:0), fid=(id:1,fid:1,ffid:0,addr:1,faddr:1)
[2022-05-14 20:59:17.709][Trace][221869][6939h41e] <- CPB time=105013635, okbps=0,0,0, ikbps=2706,2703,0, mr=0/350, p1stpt=20000, pnt=5000
[2022-05-14 20:59:19.225][Trace][221869][6939h41e] -> HLS time=111532644ms, sno=7, ts=livestream-6.ts, dur=12266ms, dva=0p
[2022-05-14 20:59:22.208][Trace][221869][k557z4lp] Hybrid cpu=13.99%,28MB, cid=1,0, timer=61,0,0, clock=0,37,9,1,1,0,0,0,0, objs=(pkt:494,raw:130,fua:363,msg:1031,oth:1,buf:100)
[2022-05-14 20:59:22.208][Trace][221869][k557z4lp] RTC: Server conns=1, rpkts=(1,rtp:0,stun:1,rtcp:1), spkts=(23,rtp:22,stun:1,rtcp:2), rtcp=(pli:1,twcc:2,rr:1), snk=(24,a:12,v:12,h:0), rnk=(1,1,h:9,m:0), fid=(id:1,fid:1,ffid:0,addr:1,faddr:1)
[2022-05-14 20:59:22.709][Trace][221869][6939h41e] <- CPB time=110019424, okbps=0,0,0, ikbps=2707,2703,0, mr=0/350, p1stpt=20000, pnt=5000
[2022-05-14 20:59:25.439][Trace][221869][42u1778l] <- RTC RECV #12, udp 351, pps 10/35, schedule 351
[2022-05-14 20:59:25.591][Trace][221869][7173158y] RTC: NACK ARQ seq=6379, ssrc=2241098775, ts=2375910, count=1947/1947, 1214 bytes
[2022-05-14 20:59:27.208][Trace][221869][k557z4lp] Hybrid cpu=14.99%,28MB, cid=3,28, timer=61,0,38, clock=0,39,7,1,1,0,0,0,0, free=1, objs=(pkt:711,raw:192,fua:518,msg:1248,oth:1,buf:99)
[2022-05-14 20:59:27.208][Trace][221869][k557z4lp] RTC: Server conns=1, rpkts=(1,rtp:0,stun:1,rtcp:1), spkts=(23,rtp:22,stun:1,rtcp:2), rtcp=(pli:1,twcc:2,rr:1), snk=(24,a:12,v:12,h:0), rnk=(1,1,h:9,m:0), fid=(id:1,fid:1,ffid:0,addr:1,faddr:1
[2022-05-14 20:59:29.250][Trace][221869][6939h41e] -> HLS time=121556624ms, sno=8, ts=livestream-7.ts, dur=5600ms, dva=0p
[2022-05-14 20:59:32.209][Trace][221869][k557z4lp] Hybrid cpu=15.98%,28MB, cid=3,28, timer=61,0,38, clock=0,39,7,1,1,0,0,0,0, free=1, objs=(pkt:711,raw:192,fua:518,msg:1248,oth:1,buf:99)
[2022-05-14 20:59:32.209][Trace][221869][k557z4lp] RTC: Server conns=1, rpkts=(34,rtp:0,stun:1,rtcp:34), spkts=(489,rtp:489,stun:1,rtcp:0), rtcp=(pli:21,twcc:0,rr:0), rnk=(8,8,h:197,m:0), fid=(id:0,fid:34,ffid:0,addr:1,faddr:34)
[2022-05-14 20:59:32.710][Trace][221869][6939h41e] <- CPB time=120016150, okbps=0,0,0, ikbps=2706,2692,0, mr=0/350, p1stpt=20000, pnt=5000
[2022-05-14 20:59:35.439][Trace][221869][42u1778l] <- RTC RECV #12, udp 346, pps 12/34, schedule 346
[2022-05-14 20:59:35.702][Trace][221869][7173158y] RTC: NACK ARQ seq=8885, ssrc=2241098775, ts=3245940, count=4118/4118, 1214 bytes
[2022-05-14 20:59:37.211][Trace][221869][k557z4lp] Hybrid cpu=13.99%,28MB, cid=1,34, timer=61,0,48, clock=0,39,8,1,0,0,0,0,0, objs=(pkt:813,raw:206,fua:606,msg:1344,oth:1,buf:98)
[2022-05-14 20:59:37.211][Trace][221869][k557z4lp] RTC: Server conns=1, rpkts=(34,rtp:0,stun:1,rtcp:34), spkts=(489,rtp:489,stun:1,rtcp:0), rtcp=(pli:21,twcc:0,rr:0), rnk=(8,8,h:197,m:0), fid=(id:0,fid:34,ffid:0,addr:1,faddr:34)
[2022-05-14 20:59:39.273][Trace][221869][6939h41e] -> HLS time=131573119ms, sno=8, ts=livestream-7.ts, dur=15634ms, dva=0p
[2022-05-14 20:59:42.211][Trace][221869][k557z4lp] Hybrid cpu=16.00%,28MB, cid=1,34, timer=61,0,48, clock=0,39,8,1,0,0,0,0,0, objs=(pkt:813,raw:206,fua:606,msg:1344,oth:1,buf:98)
[2022-05-14 20:59:42.212][Trace][221869][k557z4lp] RTC: Server conns=1, rpkts=(34,rtp:0,stun:1,rtcp:33), spkts=(515,rtp:515,stun:1,rtcp:0), rtcp=(pli:20,twcc:0,rr:0), rnk=(8,8,h:211,m:0), fid=(id:0,fid:34,ffid:0,addr:1,faddr:34)
[2022-05-14 20:59:43.081][Trace][221869][0q79mo9j] <- CPB time=39986538, okbps=0,1,0, ikbps=2566,2520,0, mr=0/350, p1stpt=20000, pnt=5000
[2022-05-14 20:59:45.440][Trace][221869][42u1778l] <- RTC RECV #12, udp 343, pps 14/34, schedule 343
[2022-05-14 20:59:45.736][Trace][221869][7173158y] RTC: NACK ARQ seq=11446, ssrc=2241098775, ts=4133880, count=6221/6221, 1214 bytes

4. SRS Config(配置):

listen              1935;
max_connections     1000;
daemon              off;
srs_log_tank        console;

http_server {
    enabled         on;
    listen          8080;
    dir             ./objs/nginx/html;
}

http_api {
    enabled         on;
    listen          1985;
}
stats {
    network         0;
}
rtc_server {
    enabled on;
    listen 8000; # UDP port
    # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#config-candidate
    candidate $CANDIDATE;
}

vhost __defaultVhost__ {
    rtc {
        enabled     on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtmp-to-rtc
        rtmp_to_rtc on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtc-to-rtmp
        rtc_to_rtmp on;
    }
    http_remux {
        enabled     on;
        mount       [vhost]/[app]/[stream].flv;
    }
}

5. Replay(重现)

1. 使用rtmp2rtc.conf 启动srs服务;
2. 使用RTC推流，并使用RTC拉流、RTMP拉流验证可以正常播放；
3. 关闭RTC推流网页；
4. 使用OBS或FFMPEG，采用RTMP协议推流；
5. 再次使用RTC拉流播放，发现只有声音，视频黑屏；
6. 可以再次使用RTMP拉流播放，验证此时可以正常播放。

6. Expect(期望行为)

RTC推流与RTMP推流可以正常根据需要切换。

[winlinvip]

Confirm if using FFmpeg to push doc/source.flv can reproduce the issue. If it cannot be reproduced, this issue will be closed directly.

TRANS_BY_GPT3

[zhOrange]

Currently, two sets of SRS services have been deployed locally (Ubuntu) and on Alibaba Cloud respectively. On the local computer, ffmpeg is used to push the stream (not using the source.flv in the source code, but using other flv files). Today, following the above reproduction steps, the aforementioned issue can be reproduced both in the local environment and in the Alibaba Cloud environment. The simplified steps are as follows:

1. Use RTC to push the stream.
2. Use RTC to pull the stream, and it can be played normally.
3. Close the RTC push stream webpage.
4. Use ffmpeg to push the stream.
5. Use RTC to pull the stream. At this time, there is only audio and no video (during today's testing, it was found that no video frame data was received in the webrtc-internals page. Previously, sometimes it was found that although video frame data was received, the decoded frames were 0 and all were dropped).

TRANS_BY_GPT3

[ellenWei11]

Has this problem been resolved? Is the streamName for rtc pushing the same as rtmp pushing? The phenomenon on my side seems very similar to yours. Did you select VP9 for the video in the webrtc-internals page?

TRANS_BY_GPT3

[zhOrange]

Has this issue been resolved? Is the streamName for rtc pushing the same as rtmp pushing? The situation on my end seems similar to yours. Did you select VP9 for the video in the webrtc-internals page?

The problem still exists. I tried to find the abnormal reason from the SRS logs, but without success. This phenomenon occurs only when the streamName for RTC and RTMP pushing is the same. When RTC and RTMP push different streamNames, there are no abnormalities, and both pushing and pulling streams with different streamNames work normally. In webrtc-internals, during normal RTC pushing and pulling, the negotiated protocol is always H264. In the abnormal case, RTC pushes with H264, but the pulling stream does not parse the protocol, as if it was not negotiated properly. However, the answer in the pulling stream shows that the protocol is H264.

TRANS_BY_GPT3

[ellenWei11]

srs_error_t SrsRtmpConn::acquire_publish(SrsLiveSource* source)
{
    srs_error_t err = srs_success;
    
    SrsRequest* req = info->req;

    // Check whether RTMP stream is busy.
    if (!source->can_publish(info->edge)) {
        return srs_error_new(ERROR_SYSTEM_STREAM_BUSY, "rtmp: stream %s is busy", req->get_stream_url().c_str());
    }
    
    // Check whether RTC stream is busy.
#ifdef SRS_RTC
    SrsRtcSource *rtc = NULL;
    bool rtc_server_enabled = _srs_config->get_rtc_server_enabled();
    bool rtc_enabled = _srs_config->get_rtc_enabled(req->vhost);
    if (rtc_server_enabled && rtc_enabled && !info->edge) {
        if ((err = _srs_rtc_sources->fetch_or_create(req, &rtc)) != srs_success) {
            return srs_error_wrap(err, "create source");
        }

        if (!rtc->can_publish()) {
            return srs_error_new(ERROR_SYSTEM_STREAM_BUSY, "rtc stream %s busy", req->get_stream_url().c_str());
        }
    }
#endif

    // Bridge to RTC streaming.
#if defined(SRS_RTC) && defined(SRS_FFMPEG_FIT)
    if (rtc) {

	**rtc->update_stream_desc();**
        SrsRtcFromRtmpBridger *bridger = new SrsRtcFromRtmpBridger(rtc);
        if ((err = bridger->initialize(req)) != srs_success) {
            srs_freep(bridger);
            return srs_error_wrap(err, "bridger init");
        }

        source->set_bridger(bridger);
    }
#endif

    // Start publisher now.
    if (info->edge) {
        return source->on_edge_start_publish();
    } else {
        return source->on_publish();
    }
}

Add rtc->update_stream_desc(), inside this function, release the stream_desc_ that was requested when rtc was pushing the stream, and assign a default stream_desc_. There will be no problem with RTC playback when pushing the stream with RTMP.

void SrsRtcSource::update_stream_desc()
{
    if (stream_desc_) 
    {
        srs_freep(stream_desc_);
    }
	   
    SrsRtcSourceDescription* stream_desc = new SrsRtcSourceDescription();
    SrsAutoFree(SrsRtcSourceDescription, stream_desc);

    // audio track description
    if (true) {
        SrsRtcTrackDescription* audio_track_desc = new SrsRtcTrackDescription();
        stream_desc->audio_track_desc_ = audio_track_desc;

        audio_track_desc->type_ = "audio";
        audio_track_desc->id_ = "audio-" + srs_random_str(8);

        uint32_t audio_ssrc = SrsRtcSSRCGenerator::instance()->generate_ssrc();
        audio_track_desc->ssrc_ = audio_ssrc;
        audio_track_desc->direction_ = "recvonly";

        audio_track_desc->media_ = new SrsAudioPayload(kAudioPayloadType, "opus", kAudioSamplerate, kAudioChannel);
    }

    // video track description
    if (true) {
        SrsRtcTrackDescription* video_track_desc = new SrsRtcTrackDescription();
        stream_desc->video_track_descs_.push_back(video_track_desc);

        video_track_desc->type_ = "video";
        video_track_desc->id_ = "video-" + srs_random_str(8);

        uint32_t video_ssrc = SrsRtcSSRCGenerator::instance()->generate_ssrc();
        video_track_desc->ssrc_ = video_ssrc;
        video_track_desc->direction_ = "recvonly";

        SrsVideoPayload* video_payload = new SrsVideoPayload(kVideoPayloadType, "H264", kVideoSamplerate);
        video_track_desc->media_ = video_payload;

        video_payload->set_h264_param_desc("level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f");
    }

    set_stream_desc(stream_desc);
}

TRANS_BY_GPT3

[johzzy]

See #3079

After the rtc push stream, the video payload type corresponding to the rtc source will change due to the sdp negotiation result.

Afterwards, if rtmp to rtc or srt to rtc occurs, the bridge constructs rtp packets with a default video payload type of 102. This mismatch with the previous situation causes the webrtc receiving end to not correctly obtain the video track.

Solution: Set the value of the video payload type for constructing rtp packets in the bridge to match the video payload type of the rtc source.

TRANS_BY_GPT3

[lihongyang166]

Using Docker for testing, server version SRS/4.0.253, there is an issue with RTMP Over WebRTC where there is only audio but no video.

TRANS_BY_GPT3

[zhOrange]

See #3079

After the rtc push stream, the video payload type corresponding to the rtc source will change due to the sdp negotiation result.

If rtmp to rtc or srt to rtc occurs afterwards, the bridge constructs rtp packets with the default video payload type of 102. This mismatch with the previous situation leads to the webrtc receiver not being able to correctly obtain the video track.

Solution: Set the value of the video payload type for constructing rtp packets in the bridge to match the video payload type of the rtc source.

Thank you all for your help! After multiple environment verifications, the problem is mostly resolved!!! However, there is still a small flaw. When repeating the above steps and using RTC to pull and play RTMP streams, there is still a black screen for 1-2 seconds at the beginning, with only audio, before the video starts playing.

TRANS_BY_GPT3

[xiangshenb]

Indeed, it seems that we cannot use WebRTC for streaming. After streaming, if we use RTMP to stream to the same "/app/stream", then the WebRTC playback will only have audio.

TRANS_BY_GPT3

[johzzy]

Indeed, it does exist. It seems that we cannot use WebRTC for streaming. After streaming with WebRTC, if we try to stream again using RTMP to the same "/app/stream", then the WebRTC playback will only have audio.

Fix details can be found at https://github.com/ossrs/srs/issues/3041#issuecomment-1164636850

TRANS_BY_GPT3

[winlinvip]

Reproduce the path: ' Make sure to maintain the markdown structure.

1. Use Firefox to push WebRTC stream.
2. Use Firefox to play WebRTC stream.
3. Stop pushing stream with Firefox.
4. Use FFmpeg to push RTMP stream.
5. Firefox has no sound and no video.

This seems to be a specific difference with Firefox, Chrome has no issues.

TRANS_BY_GPT3