RTMP to WebRTC fps drop

[danielboros1990]

Describe the bug There are more different ways how I can start a stream. If I am using ffmpeg or DJI mini 3 pro via rtmp everything works on every device fine on the client side using Mediastream and PeerConnnection javascript API. If I am using a DJI Mavic Enterprise (Thermal) then I have huge fps drops on iOS and Mac Safari in all cases I get to stream the same settings like H264, Level 3.1/4 with different resolutions. I would think that SRS applies the same encode decode settings for all incoming RTMP data and push the same through RTC, or maybe there are some differences? Does anybody any idea what could be the issue, or where should I try to start it? I can reproduce the bug using FFmpeg. VP8/VP9/Av1 is not the problem I think, because it is experimental on iOS, but there is always H264 code.

Version Latest of V5.

Expected behavior Smooth WebRTC streaming on all devices.

[zeldan]

I also have the same problem, but I tested with OBS Studio and found the following:

If I stream using the advanced settings and change the Output mode to "Advanced," then:

FFmpeg VAAPI H.264 --> Linux Chrome: OK, Android Chrome: OK NVIDIA NVENC H.264 --> Linux Chrome: FPS drops (max 1-2 frames per second) - very laggy, Android Chrome: OK

I was able to reproduce the issue with a simple ffmpeg command: Dropped frame rate: ffmpeg -re -stream_loop -1 -i clip2.mp4 -vcodec h264_nvenc -c:a copy -f flv rtmp://<host> Correct frame rate: ffmpeg -re -stream_loop -1 -i clip2.mp4 -c copy -f flv rtmp://<host>

It seems that more expensive drones (or controllers) use another algorithm, which is not FFmpeg VAAPI H.264.

Are there any settings I can try to change? Unfortunately, I cannot change anything on the drone's side (no any option in the official DJI app), only on the SRS.

Tested on Ubuntu 22 with the default config, docker

[winlinvip]

Please use h.264 profile baseline if need converting RTMP to WebRTC. Dup to #4052

[dancixx]

@winlinvip Do you mean in ffmpeg.transcode.conf, because it will be the same as HTTP-FLV, there will be the same delay?

[suzp1984]

Maybe it is the web browser don't support h.264 b-frame, because baseline profile don't support b-frame, that's the reason baseline works well.

@zeldan you can verify by disable the b-frame by -bf 0. I don't have a NVidia gpu, so I can't verify, the h264_nvenc, which is NVidia hardware encoder, must be encode the video frame to h.264 main/high profile with b-frames.

ffmpeg -re -stream_loop -1 -i clip2.mp4 -vcodec h264_nvenc -bf 0 -c:a copy -f flv rtmp://<host>

or just use baseline profile.

ffmpeg -re -stream_loop -1 -i clip2.mp4 -vcodec h264_nvenc -profile:v baseline -c:a copy -f flv rtmp://<host> the params, -profile:v, is libx264's params, I don't know whether the h264_nvenc support it or not.

[suzp1984]

Sorry, I reproduced the Stuttering or laggy effect, the b-frame is laggy + part rect screen blur effect. How to calculate or check webrtc fps? did anyone knows?

Here is my way to reproduce the Stuttering effect?

1. ./objs/srs -c conf/rtmp2rtc.conf (don't forget to set CANDIDATE env);
2. publish a video with profile high: ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec h264 -profile:v high -acodec copy -f flv rtmp://localhost/live/livestream
3. use ffmpeg to do transcode, publish the transcoded stream to live/livestream_ff ffmpeg -i rtmp://localhost/live/livestream -vcodec h264 -profile:v baseline -acodec copy -f flv rtmp://localhost/live/livestream_ff (@dancixx don't use ffmpeg.transcode.conf, this cmd will have same effect, the ffmpeg.transcode.conf will result in avalanche effect, the transcoded stream will trigger another transcode process)
4. Now, we have two streams: live/livestream and live/livestream_ff, live/livestream is profile high and have b-frames, while live/livestream_ff is profile baseline without b-frame. You can verify by ffprobe: ffprobe rtmp://localhost/live/livestream ffprobe rtmp://localhost/live/livestream_ff ffprobe -show_frames rtmp://localhost/live/livestream ffprobe -show_frames rtmp://localhost/live/livestream_ff (check the pict_type: I, B, P)
5. ffplay live/livestream and live/livestream_ff, the videos play well.
6. check the webrtc http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream_ff, this stream's profile is baseline, but video seems laggy.

If I do transcode in step 2, don't do next steps. ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec h264 -profile:v baseline -acodec copy -f flv rtmp://localhost/live/livestream then play webrtc http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream, the video seems fine.

[winlinvip]

Dup to https://github.com/ossrs/srs/issues/4052 Fixed by https://github.com/ossrs/srs/commit/1656391c6792baa98e2cd005d1338a0981fc75ed