WebRTC：720p高码率时，RTC推流转RTMP卡顿严重

[superyhee]

注意：不提供以下信息的Issue会被直接删除(Please follow issue template, or we will delete it)

注意：咨询和讨论请提交到SRS星球(Please ask question at) http://bbs.ossrs.net

描述(Description) 用浏览器通过rtc sdk推流，视频分辨率是1280720，rtmp或flv播放卡顿严重，几乎无法播放，但是用rtc播放却很流畅，如果把视频分辨率降低到640480能够正常播放，但是也会偶尔卡顿，请问是配置有什么问题吗？目前rtc推流的视频分辨率有限制吗？

描述你遇到了什么问题(Please description your issue here) rtmp和flv无法正常播放

1. SRS版本(Version): 4.0177
2. SRS的日志如下(Log):

xxxxxxxxxxxx

1. SRS的配置如下(Config):

# main config for srs.
# @see full.conf for detail config.

listen              1935;
max_connections     1000;
srs_log_tank        console;
daemon              off;

http_api {
    enabled         on;
    listen          1985;
}
http_server {
    enabled         on;
    listen          8080;
    dir             ./objs/nginx/html;
}
# stats {
#     network         0;
#     disk            sda sdb xvda xvdb;
# }
rtc_server {
    enabled         on;
    # Listen at udp://8000
    listen          8000;
    #
    # The $CANDIDATE means fetch from env, if not configed, use * as default.
    #
    # The * means retrieving server IP automatically, from all network interfaces,
    # @see https://github.com/ossrs/srs/issues/307#issuecomment-599028124
    candidate       $CANDIDATE;
}
vhost __defaultVhost__ {
   #最小延迟打开，默认是打开的，该选项打开的时候，mr默认关闭。
    # tcp_nodelay     on;
    # min_latency     on;
    #Merged-Read，针对RTMP协议，为了提高性能，SRS对于上行的read使用merged-read，即SRS在读写时一次读取N毫秒的数据
    # mr {
    #     enabled     on;
    #     #默认350ms，范围[300-2000]
    #     latency     800;
    # }
    #Merged-Write,SRS永远使用Merged-Write，即一次发送N毫秒的包给客户端。这个算法可以将RTMP下行的效率提升5倍左右,范围[350-1800]
    # mw_latency      700;
    # enabled         on;
    #https://github.com/simple-rtmp-server/srs/wiki/v2_CN_LowLatency#gop-cache
    # gop_cache       on;
    #配置直播队列的长度，服务器会将数据放在直播队列中，如果超过这个长度就清空到最后一个I帧
    #https://github.com/simple-rtmp-server/srs/wiki/v2_CN_LowLatency#%E7%B4%AF%E7%A7%AF%E5%BB%B6%E8%BF%9F
    # queue_length    10;
    # http_remux {
    #     enabled     on;
    #     mount       [vhost]/[app]/[stream].flv;
    # }
    rtc {
        enabled     on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtmp-to-rtc
        rtmp_to_rtc on;
        # @see https://github.com/ossrs/srs/wiki/v4_CN_WebRTC#rtc-to-rtmp
        rtc_to_rtmp on;
    }

    http_remux {
        enabled     on;
        mount       [vhost]/[app]/[stream].flv;
    }
    # play {
    #     gop_cache       off;
    #     queue_length    10;
    #     mw_latency      100;
    # }

    http_hooks {
        enabled         on;
        on_connect      http://localhost:3000/api/;
        on_close        http://localhost:3000/api/;
        on_publish      http://localhost:3000/api/;
        on_unpublish    http://localhost:3000/api/;
        on_play         http://localhost:3000/api/;
        on_stop         http://localhost:3000/api/;
    }
    
    # publish {
    #     mr off;
    # }
}

重现(Replay)

重现Bug的步骤(How to replay bug?)

1. 浏览器rtc推流
2. xxxxxx
3. xxxxxx

期望行为(Expect) rtmp能够正常播放

描述你期望发生的事情(Please describe your expectation)

[winlinvip]

This is an adaptation problem during the conversion of RTMP and RTC. High bitrate may cause some unknown issues, and it takes time to coordinate and debug together.

I have already notified @xiaozhihong and need to wait for his availability. You can also analyze it more, as this problem is quite difficult.

TRANS_BY_GPT3

[superyhee]

Thank you, the CPU resource consumption is not significant when converting RTC to RTMP with a 720p high bitrate, but there is basically no output on RTMP.

TRANS_BY_GPT3

[xiaozhihong]

Thank you, the CPU resource consumption is not high when RTC is converted to RTMP at a high bitrate of 720p, but the RTMP output is basically not available.

Can you provide me with a video file or let me know the streaming method and the corresponding configuration? For example, the bitrate, fps, and the duration between keyframes.

TRANS_BY_GPT3

[superyhee]

Without using a video file, in Chrome use navigator.mediaDevices.getUserMedia(constraints) to obtain the camera stream, and then pass the stream to rts_sdk for streaming. The information for the constraint is as follows: var constraints = { video: { deviceId: this.props.stateContext.config.cameraId, width: 1280, height: 720, frameRate: 20 }, audio: { deviceId: this.props.stateContext.config.microphoneId, } };

TRANS_BY_GPT3

[xiaozhihong]

Without using a video file, use navigator.mediaDevices.getUserMedia(constraints) in Chrome to obtain camera stream, and then pass the stream to rts_sdk for streaming. The constraint information is as follows: var constraints = { video: { deviceId: this.props.stateContext.config.cameraId, width: 1280, height: 720, frameRate: 20 }, audio: { deviceId: this.props.stateContext.config.microphoneId, } };

When buffering occurs, is it played locally on localhost, or in the local network, or in the public network?

TRANS_BY_GPT3

[superyhee]

Local playback, unrelated to the internet, RTC playback is very smooth, but RTMP streaming is basically unusable.

TRANS_BY_GPT3

[bywwcnll]

Thumbs up, thumbs up, thumbs up

TRANS_BY_GPT3

[superyhee]

I tested it with the latest XCORE-SRS/4.0.195 (Leo) version using the previous method, but it still freezes a lot. Sometimes I can play rtmp, but there is a delay of more than 10 seconds, and the audio and video are not synchronized.

TRANS_BY_GPT3

[xiaozhihong]

I tested it using the latest XCORE-SRS/4.0.195(Leo) version in the same way as before, but it still lags. Sometimes it can play RTMP, but with a delay of over 10 seconds and the audio and video are not synchronized.

Are you in the SRS WeChat group? If you are, please add me so that I can take a look at your setup.

TRANS_BY_GPT3

[superyhee]

Not here, how can I add you on WeChat?

John @.*** wrote on Thursday, November 11, 2021, at 9:55 PM:

I tested it using the latest XCORE-SRS/4.0.195 (Leo) version with the previous method, but it is still lagging. Sometimes it can play RTMP, but there is a delay of more than 10 seconds, and the audio and video are not synchronized.

Are you in the SRS WeChat group? If you are, please add me so that I can see your scenario.

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ossrs/srs/issues/2677#issuecomment-966321180, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCT5ZCCJ2DMRWOL3DUAWYDULPDNPANCNFSM5F6QHLEQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

TRANS_BY_GPT3

[xiaozhihong]

I'm not available. How can I add you on WeChat?

john @.***> wrote on Thursday, November 11, 2021, at 9:55 PM:

[...] I tried testing with the latest XCORE-SRS/4.0.195 (Leo) version using the previous method, but it's still very laggy. Sometimes it can play rtmp, but with a delay of more than 10 seconds, and the audio and video are not synchronized. Are you in the SRS WeChat group? If you are, please add me so I can see your setup.

@superyhee sent you an email.

TRANS_BY_GPT3

[superyhee]

Can't see any information from WeChat groups?

TRANS_BY_GPT3

[KevinLoveGitHub]

SRS 4.0.177 version encountered a similar issue. In a local area network environment, webrtc 480p 15-frame pushing has very good RTC pulling effect. However, RTMP pulling has a delay of about 5-10 seconds and often experiences frequent disconnections.

TRANS_BY_GPT3

[bywwcnll]

Encountered similar issues in SRS 4.0.177 version. In a local area network environment, webrtc 480p, 15 fps streaming has a very good RTC playback effect. However, RTMP playback has a delay of approximately 5-10 seconds and frequent disconnections.

Compiled and tested with the latest code because some issues were resolved recently. Currently, apart from the latency, there are basically no more disconnection problems.

TRANS_BY_GPT3

[KevinLoveGitHub]

Encountered similar issues with SRS 4.0.177 version. In a local area network environment, webrtc 480p 15 fps streaming has excellent playback results. However, there is a delay of around 5-10 seconds and frequent stream interruptions when using RTMP playback.

Compiled and tested with the latest code, as some of the issues have recently been resolved. Currently, apart from the delay, there are basically no stream interruption problems.

Thank you for your reply. I have tested it with version 4.0.205 on Ubuntu 18.04, and the situation is similar to before. There are some issues with the DVR recording feature:

1. There are instances where only audio without video, or only one frame of video, appears for the first 10 seconds or so.
2. In the middle of the recorded file, there may be a period of time where only audio is present, and the video freezes.
3. After the recording finishes, the flv file cannot be accessed via HTTP in the browser.

TRANS_BY_GPT3

[winlinvip]

It seems like the issue still exists, Reopen.

TRANS_BY_GPT3

[BlueAirRL]

The same problem with RTC live streaming, DVR-FLV recording is also very choppy, but RTC playback is smooth. SRS version 4.0.177, resolution: 320*180.

TRANS_BY_GPT3

[winlinvip]

You must attach the FLV file for it to work. This is definitely a problem with the file content. Without the file content, it is impossible to troubleshoot.

TRANS_BY_GPT3

[BlueAirRL]

The previous rtc to rtmp memory overflow should be related. #2680 The package cannot be consumed in time, causing stuttering in rtc to rtmp playback and recording.

TRANS_BY_GPT3

[winlinvip]

Please do not guess what the reason is.

If there is no data, this problem closes.

The problems with audio and video cannot be completely solved, and it's not just about audio and video, there are a bunch of other issues as well.

TRANS_BY_GPT3

[superyhee]

It's actually very easy to reproduce. Use 720p streaming and constantly switch the screen. When pulling the stream with ffmpeg, it will show 'Non-monotonous DTS in output stream 0:0', and then it will start lagging.

TRANS_BY_GPT3

[winlinvip]

How often should we switch screens?

TRANS_BY_GPT3

[superyhee]

No need for frequent occurrence, as long as there is significant change in the screen, Doctor Xiao's preliminary diagnosis is that the sender report crosses in the middle of long I frames, resulting in floating-point calculation error, causing a timestamp synchronization error.

TRANS_BY_GPT3

[winlinvip]

That seems to know where the problem lies, just waiting for the fix patch? Why does it feel like the shortest reproducible path has not been found yet?

TRANS_BY_GPT3

[superyhee]

I really didn't find the shortest necessary path, it has a certain randomness, but it appears easily. This bug indeed affects the use of rtc2rtmp, including subsequent streaming or recording operations with ffmpeg. The exact reason for the specific problem is clearer to xiaozhihong.

TRANS_BY_GPT3

[winlinvip]

We generally don't solve occasional problems, and we don't require anyone to solve necessary ones. The rule of open source is to solve one's own problems.

So it's better to spend more time on how to address the necessary problems.

TRANS_BY_GPT3

[superyhee]

Understood, I will take a look when I have time.

TRANS_BY_GPT3

[funbinary]

I also encountered the same situation. Using SRS version SRS/4.0.240, configuration file rtc2rtmp.conf. Steps to reproduce using SRS built-in signaling:

1. Change the shared camera to shared screen, modify the file srs.sdk.js, change getUserMedia to getDisplayMedia, and set constraints. The video width only needs to be greater than 768. My constraints are as follows:

    self.constraints = {
        audio: true,
        video: {
The translation to English is as follows:

```javascript
//         aspectRatio: 16 / 9, // aspect ratio
//         frameRate: 24, // frame rate
      width: 768, // width
      height: 432 // height

Please note that the commented lines have been omitted in the translation. }

2. Start a call to share a web tab to play videos in 720P or above. I am using Tencent Video to play a 720P video.
3. Then, when using ffplay or potplayer to play the video, it always lags. When using potplayer to play, I noticed that the frame rate drops below 10, as shown in the picture.
![image](https://user-images.githubusercontent.com/44047515/154023212-b3985a65-25a3-488c-8f58-c2571c51c600.png)

`TRANS_BY_GPT3`

[winlinvip]

This pit is not small, hahaha.

TRANS_BY_GPT3

[lipeng19811218]

Can you take a look at the SRS code where the fixed request pli is used? If it is currently every 1 second, can you change it to every 5 seconds and see if it works?

TRANS_BY_GPT3