webrtcsink
webrtcsink copied to clipboard
centricular
Running into issues after installing omx
Hello, I was using webrtcsink (hash 55d30db53bb3931f6477b6c1bad4de2a5ec5f7e4) for a while in the Raspbery Pi, with the following pipeline, and it was working fine (and it is still does work on my development machine):
v4l2src device=/dev/video3 \
! video/x-h264,width=320,height=240,framerate=15/1 \
! decodebin3 ! videoconvert ! webrtcsink \
stun-server=stun://0.0.0.0:3478 \
turn-server=turn://user:[email protected]:3478 \
signaller::address=ws://0.0.0.0:6021/ \
video-caps=video/x-h264 \
display-name=\"/dev/video3 - H264 USB Camera: USB Camera\" \
congestion-control=0 \
do-retransmission=false \
do-fec=false \
enable-data_channel_navigation=false
However, if I install the OMX encoders (which works fine for encoding h264), it doesn't work, showing the following error:
root@blueos:~# GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-h264,width=320,height=240,framerate=15/1 ! decodebin3 ! videoconvert ! webrtcsink stun-server=stun://0.0.0.0:3478 turn-server=turn://user:[email protected]:3478 signaller::address=ws://0.0.0.0:6021/ video-caps=video/x-h264 display-name=\"/dev/video3 - H264 USB Camera: USB Camera\" congestion-control=0 do-retransmission=false
do-fec=false enable-data_channel_navigation=false
0:00:00.261615000 4359 0xf44d60 WARN GST_PLUGIN_LOADING gstplugin.c:505:gst_plugin_register_func: plugin "/usr/local/lib/gstreamer-1.0/libwebrtcsink.so" has unknown license "MIT"Setting pipeline to PAUSED ...Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...Setting pipeline to PLAYING ...New clock: GstSystemClock0:00:00.309164271 4359 0xf3b478 FIXME decodebin3 gstdecodebin3.c:1143:update_requested_selection:<decodebin3-0> Implement EXPOSE_ALL_MODE
0:00:00.345123958 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:842:gst_v4l2_buffer_pool_start:<v4l2src0:pool0:src> Uncertain or not enough buffers, enabling copy threshold0:00:01.477050312 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free0:00:01.478125468 4359 0xf3b478 FIXME decodebin3 gstdecodebin3-parse.c:433:unblock_pending_input:<decodebin3-0> Re-use existing input streams if/when possible0:00:01.480217031 4359 0xf3c170 FIXME decodebin3 gstdecodebin3.c:1648:get_output_for_slot:<decodebin3-0> emit autoplug-continue
0:00:01.480427500 4359 0xf3c170 FIXME decodebin3 gstdecodebin3.c:1651:get_output_for_slot:<decodebin3-0> Handle EXPOSE_ALL_MODE0:00:01.538979322 4359 0xf3b478 WARN v4l2src gstv4l2src.c:1123:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.2098567490:00:01.539841979 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free0:00:01.599575156 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 2 is not free
0:00:01.663749218 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 3 is not free
0:00:01.723812343 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:01.783803072 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free
0:00:01.847968333 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 2 is not free
0:00:01.908052395 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 3 is not free
0:00:01.971569947 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:02.031572916 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free
0:00:02.095605520 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 2 is not free
0:00:02.155570989 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 3 is not free
0:00:02.219586926 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:02.279587083 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free
0:00:02.343555208 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 2 is not free0:00:02.403549583 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 3 is not free
0:00:02.467572916 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:02.527541301 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free
Redistribute latency...
0:00:02.540752707 4359 0x75464c90 WARN omxvideo gstomxvideo.c:223:gst_omx_video_find_nearest_frame:<omxh264dec-omxh264dec0> Difference between ts (0:00:01.276523000) and frame 2 (0:00:01.269844666) seems too high (0:00:00.006678334)
0:00:02.591476822 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 2 is not free
0:00:02.593909322 4359 0x75464c90 WARN omxvideo gstomxvideo.c:223:gst_omx_video_find_nearest_frame:<omxh264dec-omxh264dec0> Difference between ts (0:00:01.269844000) and frame 3 (0:00:01.333824760) seems too high (0:00:00.063980760)
0:00:02.640075832 4359 0x70b80b00 FIXME default gstutils.c:4025:gst_pad_create_stream_id_internal:<videotestsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:02.651880624 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 3 is not free
0:00:02.654896978 4359 0x75464c90 WARN omxvideo gstomxvideo.c:223:gst_omx_video_find_nearest_frame:<omxh264dec-omxh264dec0> Difference between ts (0:00:01.333824000) and frame 4 (0:00:01.393853177) seems too high (0:00:00.060029177)
0:00:02.715983905 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:02.718308541 4359 0x75464c90 WARN omxvideo gstomxvideo.c:223:gst_omx_video_find_nearest_frame:<omxh264dec-omxh264dec0> Difference between ts (0:00:01.393853000) and frame 5 (0:00:01.457830916) seems too high (0:00:00.063977916)
0:00:02.769337239 4359 0x6fc43058 WARN GST_CAPS gstpad.c:5757:pre_eventfunc_check:<capsfilter4:sink> caps video/x-h264, stream-format=(string)avc, alignment=(string)au, profile=(string)baseline, level=(string)4, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)15/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, parsed=(boolean)true, codec_data=(buffer)01428028ffe1000d2742802895a0507ec04078913501000528ce025c80 not accepted
0:00:02.769760728 4359 0x6fc43058 WARN GST_CAPS gstpad.c:5757:pre_eventfunc_check:<capsfilter4:sink> caps video/x-h264, stream-format=(string)avc, alignment=(string)au, profile=(string)baseline, level=(string)4, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)15/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, parsed=(boolean)true, codec_data=(buffer)01428028ffe1000d2742802895a0507ec04078913501000528ce025c80 not accepted
0:00:02.769977082 4359 0x6fc43058 WARN GST_CAPS gstpad.c:5757:pre_eventfunc_check:<capsfilter4:sink> caps video/x-h264, stream-format=(string)avc, alignment=(string)au, profile=(string)baseline, level=(string)4, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)15/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, parsed=(boolean)true, codec_data=(buffer)01428028ffe1000d2742802895a0507ec04078913501000528ce025c80 not accepted
0:00:02.770125624 4359 0x6fc43058 WARN GST_CAPS gstpad.c:5757:pre_eventfunc_check:<capsfilter4:sink> caps video/x-h264, stream-format=(string)avc, alignment=(string)au, profile=(string)baseline, level=(string)4, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)15/1, interlace-mode=(string)progressive, colorimetry=(string)2:4:5:1, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, parsed=(boolean)true, codec_data=(buffer)01428028ffe1000d2742802895a0507ec04078913501000528ce025c80 not accepted
0:00:02.770782812 4359 0x6fc43058 WARN omxvideoenc gstomxvideoenc.c:1795:gst_omx_video_enc_loop:<omxh264enc-omxh264enc0> error: Internal data stream error.
0:00:02.770832655 4359 0x6fc43058 WARN omxvideoenc gstomxvideoenc.c:1795:gst_omx_video_enc_loop:<omxh264enc-omxh264enc0> error: stream stopped, reason not-negotiated
0:00:02.775555051 4359 0xf3b478 WARN v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free
0:00:02.776771874 4359 0x6fc205c0 WARN webrtcsink plugins/src/webrtcsink/imp.rs:2107:webrtcsink::webrtcsink::imp:<rswebrtcsink0> Codec discovery pipeline failed: Internal data stream error.
0:00:02.776912603 4359 0x6fc205c0 ERROR webrtcsink plugins/src/webrtcsink/imp.rs:2221:webrtcsink::webrtcsink::imp:<rswebrtcsink0> error: No caps found for stream video_0
ERROR: from element /GstPipeline:pipeline0/RsWebRTCSink:rswebrtcsink0: There is no codec present that can handle the stream's type.
Additional debug info:
plugins/src/webrtcsink/imp.rs(2222): webrtcsink::webrtcsink::imp (): /GstPipeline:pipeline0/RsWebRTCSink:rswebrtcsink0:
Failed to look up output caps: No caps found for stream video_0
Execution ended after 0:00:02.504602811
Setting pipeline to NULL ...
0:00:02.777954062 4359 0x75464c90 WARN omxvideo gstomxvideo.c:223:gst_omx_video_find_nearest_frame:<omxh264dec-omxh264dec0> Difference between ts (0:00:01.457830000) and frame 6 (0:00:01.517852291) seems too high (0:00:00.060022291)
0:00:02.785024062 4359 0xf44d60 WARN omxvideodec gstomxvideodec.c:418:gst_omx_video_dec_shutdown:<omxh264dec-omxh264dec0> Output buffers haven't been freed; still owned downstream?
Freeing pipeline ...
About the OMX:
root@blueos:~# gst-inspect-1.0 omx
Plugin Details:
Name omx
Description GStreamer OpenMAX Plug-ins
Filename /usr/local/lib/gstreamer-1.0/libgstomx.so
Version 1.20.2
License LGPL
Source module gst-omx
Source release date 2022-05-02
Binary package GStreamer OpenMAX Plug-ins
Origin URL Unknown package origin
omxanalogaudiosink: OpenMAX Analog Audio Sink
omxh263dec: OpenMAX H.263 Video Decoder
omxh264dec: OpenMAX H.264 Video Decoder
omxh264enc: OpenMAX H.264 Video Encoder
omxhdmiaudiosink: OpenMAX HDMI Audio Sink
omxmjpegdec: OpenMAX MJPEG Video Decoder
omxmpeg2videodec: OpenMAX MPEG2 Video Decoder
omxmpeg4videodec: OpenMAX MPEG4 Video Decoder
omxtheoradec: OpenMAX Theora Video Decoder
omxvc1dec: OpenMAX WMV Video Decoder
omxvp8dec: OpenMAX VP8 Video Decoder
11 features:
+-- 11 elements
Any thoughts? Thanks!
If omxh264enc is the only h264 encoder available, you probably need to teach webrtcsink how to make use of it.
I'm running in the same issue. I could pinpoint that the logic of selection of the encoders might need some tweaks or there is a bug in GStreamer.
The logic is present here: https://github.com/centricular/webrtcsink/blob/2dc42ba0db1efee376ace7d8d7a7adecede47a0a/plugins/src/webrtcsink/imp.rs#L1097-L1099
I realized that this error ERROR: from element /GstPipeline:pipeline0/RsWebRTCSink:rswebrtcsink0: There is no codec present that can handle the stream's type. comes from the internal discovery pipeline of webrtcsink element. So I dig a bit deeper and could find that the caps that the H264 encoder needs to support is:
video/x-h264,stream-format=avc,profile=constrained-baseline
In my case, the webrtcsink element keeps selecting the first encoder that can encode h264, which is vtenc_h264 element factory. The vtenc_h264 does not support exactly the caps that I mentioned above, looking at the docs it shows:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
interlace-mode: { (string)progressive, (string)interleaved }
stream-format: avc
alignment: au
So the vtenc_h264 does not support profile=constrained-baseline but still gets selected. I do have another encoder available which support all the caps, the x264enc. Here is the src pad caps of the x264enc element:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
framerate: [ 0/1, 2147483647/1 ]
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
stream-format: { (string)avc, (string)byte-stream }
alignment: au
profile: { (string)high-4:4:4, (string)high-4:2:2, (string)high-10, (string)high, (string)main, (string)baseline, (string)constrained-baseline, (strin
g)high-4:4:4-intra, (string)high-4:2:2-intra, (string)high-10-intra }
The call to can_src_any_caps seems to return true for caps that I think shouldn't match. I've tried to set the video-caps property of the webrtcsink element to a more specific required caps that can make the pipeline to work.
let wssink = gst::ElementFactory::make("webrtcsink", Some("ws"))?;
let constrained_caps = gst::Caps::builder("video/x-h264")
.field("stream-format", "avc")
.field("profile", "constrained-baseline")
.build();
wssink.set_property("video-caps", &constrained_caps);
But even like that, the vtenc_h264 element is selected. I think we need to update the way we are selecting the encoder element in the webrtcsink code.
Hi @rafaelcaricio, Nice to know that it isn't affecting only me, and thanks for sharing your investigation.
But even like that, the vtenc_h264 element is selected. I think we need to update the way we are selecting the encoder element in the webrtcsink code.
For now, instead of digging into webrtcsink, I'm simply skipping this unsupported encoder by changing the encoders' rank. Here you can see what I did in my code base in order to achieve that.
In the case of my application, the above mentioned patch was thought as a nice-to-have feature even before this problem arises - to let the user set ranks from the command line arguments, and might serve you as well.
Anyway, I'm still interested in allowing omx encoder to be used by webrtcsink :)
@rafaelcaricio ah, I see, I think I understand the refinement you need, though that still won't cause the omx encoder to work for @joaoantoniocardoso without further intervention / analysis:
At the moment, we only associate one encoder with each Codec, and if the discovery pipeline later fails for it we simply don't use the codec. We should instead change the encoder field of the Codec struct to hold a list of encoders, and try the next encoder in the list if the discovery pipeline does fail. I'll be happy to merge a MR in that spirit, thanks for your analysis :)
@joaoantoniocardoso Thank you for sharing, I haven't considered that option! Nice :)
quick note: my proposed improvement would also make the case where you have a GPU encoder on your machine with a session limit and a CPU encoder with a lower rank work well: webrtcsink would first use up all the available concurrent hw encoding sessions, then start using the software encoder.
Closing in favor of https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/259