Restream problem after h264_v4l2m2m encoding

[warez4me]

While configuring Go2rtc for transcode RTSP streams on RPi4 with ffmpeg, I encountered a problem using the h264_v4l2m2m encoder.

If the libx264 encoder is used then everything functions normally and predictably. I can get as many copies of the transcoded video stream as I want adding different audio substreams.

But if to use the h264_v4l2m2m encoder that supports some hardware acceleration, then the features of the output video stream that I don't understand begin to appear, which do not allow parallel use of its copy or even use it with added transcoded audio track(s).

Actually, the RTSP output is produced but its video substream is unsuitable for playback.

For example, when using VLC for playback, the codec info is

• on the left - Ok, starter, link - rtsp://[IP]:8554/stream
• on the middle - bad video, starter, link - rtsp://[IP]:8554/stream?video=all&audio=all
• on the right - bad video, follower, link - rtsp://[IP]:8554/stream

ffprobe for "good" case when consumer is a 1st who opens the stream

$ ffprobe -v error -hide_banner -of default=noprint_wrappers=0 -print_format flat  -select_streams v:0 -show_entries frame=pict_type rtsp://raspberrypi:8554/8m_2
[rtsp @ 0x5555a84fd370] method SETUP failed: 461 Unsupported transport
frames.frame.0.pict_type="I"
frames.frame.1.pict_type="P"
frames.frame.2.pict_type="P"
...
frames.frame.2.pict_type="P"
frames.frame.12.pict_type="I"
frames.frame.13.pict_type="P"
...
^C

for "bad" case when the stream is already opened by another consumer

$ ffprobe -v error -hide_banner -of default=noprint_wrappers=0 -print_format flat  -select_streams v:0 -show_entries frame=pict_type rtsp://raspberrypi:8554/8m_2
[rtsp @ 0x5555a6b1d370] method SETUP failed: 461 Unsupported transport
[h264 @ 0x5555a6b23770] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0x5555a6b23770] decode_slice_header error
[h264 @ 0x5555a6b23770] no frame!
[h264 @ 0x5555a6b23770] non-existing PPS 0 referenced
    Last message repeated 1 times
[h264 @ 0x5555a6b23770] decode_slice_header error
[h264 @ 0x5555a6b23770] no frame!
...
^C

I've tried to compare the libx264 and h264_v4l2m2m generated streams both with capturing live RTSP stream by tcpdump or by redirecting ffmpeg output to file and then using stream analyzer to examine the differences.

Unfortunately my skill isn't good enough to see the source of problem. Moreover, the "m2m" h264 stream looks more "canonical" to me. But the stream from libx264, even without the visible presence of SPS/PPS headers, can be remuxed again, and the stream from h264_v4l2m2m turns out to be unusable with any attempt to use it for remux or copy.

I think this problem is a kinda synthetic and related to both Go2rtc and ffmpeg itself. It's not HW-related because I can use multiple transcoded streams if they were configured the same way, but under different names.

• The Go2rtc is under suspicion because the 1st transcoded instance is good.
• The ffmpeg is under suspicion because the libx264 codec's output stream doesn't have problems

If the Go2rtc is a rock-solid and its way to manipulate streams with ffmpeg is the only possible one - Ok, I'll agree if it can be clearly explained.

I will list the ffmpeg related questions below hoping on the experience and knowledge of the go2rtc community.

• is there a fundamental possibility of obtaining the output h264 video stream after h264_v4l2m2m with parameters similar to those obtained after libx264?
• perhaps the problem is related to the way the memory or CPU is used during the encoding process?
• is there documentation for h264_v4l2m2m encoder with information on how to control it through ffmpeg parameters?

P.S. if required I can share any captured or redirected data.

[AlexxIT]

• I still don't understand what "bad video" means in your description.
• non-existing PPS 0 referenced is OK error for both - direct RTSP restreaming and transcoding. If everything is OK with the stream, the video should appear in a few seconds.
• You will never get the same transcoding parameters with software and hardware methods. This applies to Intel, Nvidia, AMD and other video cards. Hardware algorithms are highly optimized and differ from software ones.

[warez4me]

• related to VLC "bad video" means black screen - video stream is present in properties and can be captured by tcpdump but it's not visible. Also the properties don't have info about resolution, f_rate and so on. BTW the "good" stream props also don't have color info kinda "ITU-R BT.709" or "Planar 4:2:0..." despite the used ffmpeg option "-pix_fmt yuv420p". But such info exists in camera's rtsp.
• if you mean that it's need to wait for I-frame then it's not so. AFAIK this mean that consumer can't get SPS/PPS info from I-frame (or "key-frame" maybe). Just look - 1st who opens the stream - can obtain global header with this info. Others who trying to connect to already opened stream - can't extract it from there. And another effect - it's not possible to get on output the "good" stream with any added or replaced audio track. IMO this tracks are "trying to connect" to once transcoded stream - similar to previous case.
• Under the "same parameters" I mean logical structure - to set all this abbreviations "SPS, PPS, SDP, POC... what else" to the same places. I think it is the source of problem - encoders output the streams with different logical structure. It has no deal with HW algorithms. Again, afaik the "m2m" - it's not so "hard" ware encoder.

sorry for my "bad French" (English also).. Too many manuals and RFC-s for still short time.

[AlexxIT]

Show a stream probe (from go2rtc web) for a good stream. And also a stream probe for a bad stream. And show yaml config for both streams.

[warez4me]

Thanks!

Sorry, but it's not so clear for me how to"Show a stream probe (from go2rtc web)". Do you mean ffprobe? In what file format? What link to use from "streams"? RTSP or HTTP ? What cmdline will be the best? Or You mean tcpdump capturing?

I can propose you try webtorrent links. I've tried external access via mobile GSM 4G - one consumer was the phone, and another - PC. Effects was the same - in "bad" case the "Net" links were the same but stream was visible only for 1st consumer.

What do you prefer?

[AlexxIT]

On the main go2rtc web page you can see "probe" word. I don't need access. A probe and config are enough for now.

[warez4me]

"Semyon Semyonych!"

Here it is.

alexxit.zip

Naming:

• m2m is for 8m_w_264
• libx is for 8m_x_264
• 1st is for the case when "probe" taken without any concurrent consumers
• 2nd is for the case when "probe" taken after the VLC was already successfully opened & shows this stream in separate window
• try# just for sure

BTW I mostly using the Opera which doesn't have the menu for "probe" Json save. Only "pretty print". That's why I never know about such posibilities. These savings are from Firefox (which in turn doesn't have "pretty print").

[warez4me]

What info You exactly need ? Each tab shows in other way... And Json "Save" creates only some short form...

---

Tried to compare savings - didn't see mean diffs...

[warez4me]

Here the raw results from Opera. Firefox doesn't allow to get in such way (non pretty-print). Just for case. Can be easily removed.

alexxit_2.zip

"m2m" doesn't allow "-profile high" prm.

Result will be as follows:

[rtsp] error=streams: exec/rtsp [hevc @ 0x556c6aba60] Could not find ref with POC 21 [hevc @ 0x556c6aba60] frame_post_process: Decode fail [h264_v4l2m2m @ 0x556c6dbbb0] [Eval @ 0x7fdba076a8] Undefined constant or missing '(' in 'high' [h264_v4l2m2m @ 0x556c6dbbb0] Unable to parse option value "high" [h264_v4l2m2m @ 0x556c6dbbb0] Error setting option profile to value high. Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height , exec/rtsp rtsp://127.0.0.1:8554/8m_w_264?audio&source=ffmpeg:8m_w_264%23audio%3Dopus: Invalid data found when processing input stream=8m_w_264

P.S. noticed that VLC is visible in probe as 1st consumer

[warez4me]

And another thing - in all cases probe result appears with some delay. Except for m2m_2nd - this appears almost instantly, without any delay.

[AlexxIT]

I see you are using non-standard ffmpeg and non-standard ffmpeg parameters for transcoding. Repeat your tests on standard ffmpeg and on built-in transcoding parameters for m2m. The situation when VLC shows a black screen.

[warez4me]

Well, here are probes using go2rtc container. I decided not to destroy my environment. Previous camera outputs some non-yuv420 stream which isn't acceptable by h264_v4l2m2m if to follow your demand for go2rtc's "standard" presets. This probes are for another camera, that has another "feature" - it doesn't give a sound track. My config sets "#audio=copy" anyway because of some different, "more tolerate for restreaming" behavior. Don't remember how I got such experience but even this probes show some differences: go2rtc's log for "-c:a copy" at the end "gracefully" informs about disconnecting 2nd concurrent stream (1st logged stream is VLC). And in case when config without "#audio=copy" was used - the log ignores appearance of 2nd concurrent stream. Maybe this fact also mean something. And another thing - the output quality with go2rtc's standard "#hardware=h264_v4l2m2m" preset is very poor. At least it lacks of "-b" option to set adequate bitrate. My custom config isn't unreasonable.

v380.zip

[warez4me]

Sorry. Checked again - info in the log about 2nd consumer has no deal with probe. Seems it's for another VLC - timestamps are later for ~20 min. Well, the both logs ignore probes.

[AlexxIT]

I didn't understand much from your message. You have your own transcoding configuration that is causing a black screen in VLC. How do the built-in ffmpeg and go2rtc's built-in transcoding parameters work in this situation?

[warez4me]

@AlexxIT Seems my English is very bad. Or comments are too long. Your last request was

I see you are using non-standard ffmpeg and non-standard ffmpeg parameters for transcoding. Repeat your tests on standard ffmpeg and on built-in transcoding parameters for m2m. The situation when VLC shows a black screen.

I did exactly as You requested

• stopped go2rtc binary service and started go2rtc container.
• edited go2rtc.yaml to set standard transcoding configuration
• repeated probes 2 times : with and without copying audiostream because of the "silent" camera.
• copied probe's logs as .txt files
• copied go2rtc's logs as html pages
• made VLC screenshots of codec info for both "good" and "bad" case

I don't see any reason to state that this issue is related only to my "own transcoding configuration that is causing a black screen in VLC" Last probes clearly show that this issue doesn't depend of ffmpeg version and cmd options but only depend of used encoder. And the "black screen" it's only the visual tool to see the problem.

v380.zip

[AlexxIT]

It's still unclear from your logs.

-ca_copy\ - ?
-an\ - ?
cfg.yaml - ?
v380_2nd.PNG - Bad? What was the config? What URL opened in VLC? What probe was used for this case?
v380.PNG - Good? What was the config? What URL opened in VLC? What probe was used for this case?

[warez4me]

Sorry, I tried to give explaination in the comment which was didn't understand by you. Also I was sure that all is seen and clear from logs. At least for me it is.

---

Step by step :

• environment is : standard go2rtc container, encoder called using builtin ffmpeg presets for h264 and m2m.
• stream name for probe was v380, for VLC was used URL rtsp://192.168.10.10:8554/v380
• camera is "silent", and I made probe of it's stream as 1) without audio (ffmpeg option "-an") and 2) with audio (ffmpeg option "-c:a copy"). This was done with changing stream description in go2rtc.yaml
• for both probes the 1st consumer of stream was VLC. probe knob was klicked after VLC started to show a picture
• probe's and go2rtc's logs and VLC's screenshots were archived. For now I remade archive's structure as
  • -an folder contains probe log as .txt file and go2rtc log as html page . go2rtc.yaml contains go2rtc config and v380 stream description for produce output stream using the ffmpeg option "-an" for audio track.
  • -ca_copy folder contains probe log as .txt file and go2rtc log as html page . go2rtc.yaml contains go2rtc config and v380 stream description for produce output stream using the ffmpeg option "-c:a copy" for audio track.
  • VLC folder contains 2 screenshots of VLC's codec info. They are always the same for their cases inependently from stream description in go2rtc. Also contain the URL .
    • "good" case : v380.png shows codec info for the case when VLC is a 1st consumer who has opened rtsp://192.168.10.10:8554/v380
    • "bad" case : v380_2nd.png shows codec info for the case when VLC is a 2nd consumer who trying to open rtsp://192.168.10.10:8554/v380.
    • screenshots were taken after releasing probe action, and are given just to illustrate the situation.

v380 (2).zip

[AlexxIT]

I don't understand why you constantly show examples when there is no audio and when there is audio. If I understand correctly, the problem is with the video. There is indeed a problem with the video - there is no codec description in the stream information. This is probably due to the specifics of the m2m codec. I'll check this situation on my raspberry 3. There is a chance that this cannot be fixed at all.

You can also try this urls rtsp://192.168.10.10:8554/v380?pkt_size=1200. But you should use this urls everywhere. For 1rs and for 2nd consumers.

[warez4me]

Thanks for at least you agreed to look closer for this issue. I just trying to give You all possible info and it's Your choice what to use. To be honest I don't thinking that You can do anything except to find what exactly is incorrect with video stream on m2m output. And a little hope is that the way which now is used by go2rtc to make transcoding chain is not single possible and perhaps it can be organized differently. But if it is the m2m codec that is causing problems due to errors or implementation features, then it's probably have to put up with it. Because it seems that there will be no further support for this "acceleration" mechanism. RPi 5 is proof of that. But just for case here's some links that can be useful

• https://github.com/raspberrypi/firmware/issues/1612
• https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-mem2mem.html

I'll try the ?pkt_size=1200 option.

[AlexxIT]

This might be fixed in the distant future. When go2rtc has something like this https://github.com/AlexxIT/go2rtc/pull/1887

[warez4me]

I've tried ?pkt_size=1200 option for several m2m encoded streams using both my own "non-standard" environment and standard go2rtc container, but without any effect. And what it must be? 1st consumer opens Ok but 2nd get black screen as usual. Moreover, I discovered that your container's builtin ffmpeg doesn't support my streams which are configured to use HWA decode with -hwaccel drm option. This option is a whole "legal" for standard RaspiOs ffmpeg. btw The Frigate also using it. afaik To get it working the ffmpeg must be compiled with "--enable-v4l2-request --enable-libdrm --enable-vout-drm" options. Builtin go2rtc container's ffmpeg has only "--enable-libdrm" that isn't enough for Raspberry support. I know that go2rtc's docs declare "only software decoding" for ffmpeg. Well, it is another reason to use "non-standard" binary environment for RPi.

#1887 looks rational. As far as I understand, the current GOP of each original stream will be temporarily "accumulated" and each copy of the stream will use these current cached frames instead of the original ones. In this case, it will be possible to remember all the necessary stream parameters (SPS/PPS) and output them as a header for each new copy of the stream. But now, does the joining consumer just receive a stream consisting of copies of the current frames of the original? And if the original does not contain needed information in keyframes, then such a stream cannot be fully used as in my issue?

[warez4me]

After reading #1887 more carefully, I have a question. If initially the stream does not contain enough information to join it normally, then how does the consumer know that after the first GOP having f_rate 100fps, he needs to change fps and what value it should have? Or is this situation also provided for?