firmware
firmware copied to clipboard
raspberrypi
Gstreamer errors on specific h264 bytestream on Bullseye and Buster, works if Buster has firmware downgraded.
Hi there, I work with a project that utilizes a C++ embedded gstreamer pipeline to process and display a h264 stream of video. This pipeline, on a raspberry pi 4, resolves to:
appsrc name=mysrc is-live=true block=false max-latency=100 do-timestamp=true stream-type=stream ! queue ! h264parse ! v4l2h264dec ! videocrop top=0 bottom=0 name=videocropper ! capsfilter caps=video/x-raw name=mycapsfilter ! qtquick2videosink
(a dotfile was captured for this pipeline, here)
We're experiencing an issue where some Pi 4s updated to, or past, commit b4e395b3e87dba4964f314e12871630cabb35f70 error out on this pipeline, throwing
0:00:29.404484942 23266 0xa330e000 WARN videodecoder gstvideodecoder.c:1161:gst_video_decoder_sink_event_default:<v4l2h264dec0> error: No valid frames decoded before end of stream
0:00:29.404541645 23266 0xa330e000 WARN videodecoder gstvideodecoder.c:1161:gst_video_decoder_sink_event_default:<v4l2h264dec0> error: no valid frames found
when these affected pis are downgraded below the stateful decode update commit, the embedded gstreamer pipeline works as expected.
Furthermore, these afflicted Pis can properly execute
gst-launch-1.0 filesrc location=testfile ! h264parse ! v4l2h264dec ! video/x-raw,format=I420 ! autovideosink
where testfile was recorded with
appsrc name=mysrc is-live=true block=false max-latency=100 do-timestamp=true stream-type=stream ! filesink location=testfile
These afflicted Pis can also properly execute an embedded pipeline of
appsrc name=mysrc is-live=true block=false max-latency=100 do-timestamp=true stream-type=stream ! queue ! h264parse ! avdec_h264 ! videocrop top=0 bottom=0 name=videocropper ! capsfilter caps=video/x-raw name=mycapsfilter ! qtquick2videosink
Out of four Pi 4s this has been tested one, two were afflicted, and two worked even post the stateful decode update commit. All tested pis were on bullseye.
Afflicted revision numbers:
c03112
b03114
Unafflicted revision number:
b03111
a03111
I'd appreciate any help in resolving this manner, as I'm fully out of my depth debugging this issue further. Please let me know if you need any additional information.
Found another pi 4 that is unafflicted, revision code a03111
If I had to wager a guess here - if I'm reading how to decode revision codes correctly, the only link I see is that this isn't affecting revision 1 pi 4s
It seems extremely unlikely (not impossible) this is revision based - the differences are on the whole unrelated to the issue being described. I'd suspect a timing problem, but @6by9 is more likely to figure it out than I am.
I agree the board revision is unlikely to be significant. Can you confirm that you are testing the same sdcard on each of the four Pi4's (just in case there is something on some of the sdcards that has an effect)? And can you confirm this is repeatable? (i.e. ten passes on good revision and ten fails on bad revision)?
This has been a different sdcard per Pi (one of them was USB boot, as well), but all set up using the same installation script on top of vanilla raspbian bullseye. I can confirm that it's repeatable - the non working pis consistently don't work past the commit given, consistently do prior - the working Pis continue to work throughout. I don't have an exact number, but they've definitely been tested at least 7-10 times through the course of me trying to figure out what's going on.
Are you able to swap sdcard between a working and non-working board and confirm the problem remains with the board (and not the sdcard)?
A long shot - add over_voltage=2 to config on a bad pi. Any improvement?
Swapping the physical sdcard won't be possible here due to the distances between pis, but what I can do is take an image of the working pi's sdcard, and flash it into the sdcard of a non working pi.
Will report back with results (and over_voltage test as well) when I have them.
Tested b03111, a03111, c03112 on the same image (https://github.com/icecube45/G37_Builder/suites/4934864438/artifacts/142769129) without any other modifications. No change observed (b03111 and a03111 continued to work, whereas c03112 did not). Also tried over_voltage on c03112 and b03114 with no change (did not work)
Managed to source another pi (b03112) locally and tested an sd card swap from a pi that was working (with the same, unmodified image as linked above), and the issue did not appear.
Continuing to gather whatever information I can that may be relevant.
While I can't swap an sd card from a working Pi to one that isn't, here are three sdcard speed tests from the agnostics package, one from a working pi, two from non working pis.
rpdiags_gstreamer_bad_sdcard_pass_b03114.txt
rpdiags_gstreamer_bad_sdcard_pass_c03112.txt
rpdiags_gstreamer_ok_sdcard_fail_b03111.txt
Ok! I did a lot more digging into this and think I found not only the actual core of this issue, but also a proper means of replicating it. I will edit the issue title to properly reflect this.
It's not a matter of revision (as @JamesH65 properly guessed), nor is it an SD card issue (as @popcornmix suggested testing - I was able to swap cards on "bad" and "good" pis finally). Instead, it's a h264 bytestream issue.. kinda?
The attached .h264 file replicates this issue on all pi revisions I've tested on so far, by running
gst-launch-1.0 filesrc location=testfile.h264 ! h264parse ! v4l2h264dec ! video/x-raw,format=I420 ! autovideosink
On bullseye: This command and testfile always fails with "Internal data stream error", regardless of firmware version
On buster: This command fails with "No valid frames decoded before end of stream" when on latest firmware, after a sudo rpi-update 15471b6e99b6db2f3206f0aed67d0aebbfc9f9cf (the commit before the v4l2 stateful update), this command works as expected and the video is shown.
I've done a bit of further debugging here:
It looks like on buster the error is thrown if the color_description matrix_coefficients is set (non zero), and works when set to zero (confirmed by hex editing the above testfile).
And on bullseye the error is thrown if the color_description data is in the header at all (present_flag + the color primaries, transfer characteristics, matrix_coefficients) - confirmed again by hex editing.
At this point I am fully outside of my knowledge, but hoping my novice debugging of h264 can help out in some way.
The colour description fields translates to the V4L2 colourspaces - https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/colorspaces-defs.html The codec will decode these and tell the client if they change, in the same way that it will tell the client if the resolution changes.
Trying on Bullseye, (GStreamer 1.18.4), the kernel logging from V4L2 shows a load of attempts at TRY_FMT on the encoded stream which doesn't inherently have a colourspace), but no format being set or buffers passed in.
[ 1078.558072] videodev: v4l2_open: video10: open (0)
[ 1078.558090] video10: VIDIOC_QUERYCAP: driver=bcm2835-codec, card=bcm2835-codec-decode, bus=platform:bcm2835-codec, version=0x00050f12, capabilities=0x84204000, device_caps=0x04204000
[ 1078.558207] video10: VIDIOC_ENUM_FMT: error -22: index=0, type=vid-out, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1078.558269] video10: VIDIOC_ENUM_FMT: index=0, type=vid-out-mplane, flags=0x1, pixelformat=MPG4 little-endian (0x3447504d), mbus_code=0x0000, description='MPEG-4 Part 2 ES'
[ 1078.558431] video10: VIDIOC_ENUM_FMT: index=1, type=vid-out-mplane, flags=0x1, pixelformat=H264 little-endian (0x34363248), mbus_code=0x0000, description='H.264'
[ 1078.558579] video10: VIDIOC_ENUM_FMT: index=2, type=vid-out-mplane, flags=0x1, pixelformat=MJPG little-endian (0x47504a4d), mbus_code=0x0000, description='Motion-JPEG'
[ 1078.558720] video10: VIDIOC_ENUM_FMT: index=3, type=vid-out-mplane, flags=0x1, pixelformat=VC1G little-endian (0x47314356), mbus_code=0x0000, description='VC-1 (SMPTE 412M Annex G)'
[ 1078.558863] video10: VIDIOC_ENUM_FMT: index=4, type=vid-out-mplane, flags=0x1, pixelformat=H263 little-endian (0x33363248), mbus_code=0x0000, description='H.263'
[ 1078.559001] video10: VIDIOC_ENUM_FMT: index=5, type=vid-out-mplane, flags=0x1, pixelformat=MPG2 little-endian (0x3247504d), mbus_code=0x0000, description='MPEG-2 ES'
[ 1078.559137] video10: VIDIOC_ENUM_FMT: error -22: index=6, type=vid-out-mplane, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1078.559228] video10: VIDIOC_ENUM_FMT: error -22: index=0, type=vid-cap, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1078.559286] video10: VIDIOC_ENUM_FMT: index=0, type=vid-cap-mplane, flags=0x0, pixelformat=YU12 little-endian (0x32315559), mbus_code=0x0000, description='Planar YUV 4:2:0'
[ 1078.559474] video10: VIDIOC_ENUM_FMT: index=1, type=vid-cap-mplane, flags=0x0, pixelformat=YV12 little-endian (0x32315659), mbus_code=0x0000, description='Planar YVU 4:2:0'
[ 1078.559620] video10: VIDIOC_ENUM_FMT: index=2, type=vid-cap-mplane, flags=0x0, pixelformat=NV12 little-endian (0x3231564e), mbus_code=0x0000, description='Y/CbCr 4:2:0'
[ 1078.559762] video10: VIDIOC_ENUM_FMT: index=3, type=vid-cap-mplane, flags=0x0, pixelformat=NV21 little-endian (0x3132564e), mbus_code=0x0000, description='Y/CrCb 4:2:0'
[ 1078.559903] video10: VIDIOC_ENUM_FMT: index=4, type=vid-cap-mplane, flags=0x0, pixelformat=RGBP little-endian (0x50424752), mbus_code=0x0000, description='16-bit RGB 5-6-5'
[ 1078.560046] video10: VIDIOC_ENUM_FMT: error -22: index=5, type=vid-cap-mplane, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1078.563134] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a95, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563170] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a96, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563216] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a0e, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563246] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a0f, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563301] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a67, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563331] video10: VIDIOC_QUERYCTRL: error -22: id=0x990a6b, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1078.563733] videodev: v4l2_release: video10: release
[ 1079.649495] videodev: v4l2_open: video10: open (0)
[ 1079.649567] video10: VIDIOC_QUERYCAP: driver=bcm2835-codec, card=bcm2835-codec-decode, bus=platform:bcm2835-codec, version=0x00050f12, capabilities=0x84204000, device_caps=0x04204000
[ 1079.649891] video10: VIDIOC_ENUMINPUT: error -25: index=0, name=, type=0, audioset=0x0, tuner=0, std=0x00000000, status=0x0, capabilities=0x0
[ 1079.649975] video10: VIDIOC_ENUMSTD: error -25: index=0, id=0x0, name=, fps=0/0, framelines=0
[ 1079.650090] video10: VIDIOC_QUERYCTRL: id=0x980001, type=6, name=User Controls, min/max=0/0, step=0, default=0, flags=0x00000044
[ 1079.650202] video10: VIDIOC_QUERYCTRL: id=0x980927, type=1, name=Min Number of Capture Buffers, min/max=1/1, step=1, default=1, flags=0x00000004
[ 1079.650289] video10: VIDIOC_QUERYCTRL: error -22: id=0x80980927, type=0, name=, min/max=0/0, step=0, default=0, flags=0x00000000
[ 1079.650720] video10: VIDIOC_ENUM_FMT: index=0, type=vid-out-mplane, flags=0x1, pixelformat=MPG4 little-endian (0x3447504d), mbus_code=0x0000, description='MPEG-4 Part 2 ES'
[ 1079.651005] video10: VIDIOC_ENUM_FMT: index=1, type=vid-out-mplane, flags=0x1, pixelformat=H264 little-endian (0x34363248), mbus_code=0x0000, description='H.264'
[ 1079.651287] video10: VIDIOC_ENUM_FMT: index=2, type=vid-out-mplane, flags=0x1, pixelformat=MJPG little-endian (0x47504a4d), mbus_code=0x0000, description='Motion-JPEG'
[ 1079.651508] video10: VIDIOC_ENUM_FMT: index=3, type=vid-out-mplane, flags=0x1, pixelformat=VC1G little-endian (0x47314356), mbus_code=0x0000, description='VC-1 (SMPTE 412M Annex G)'
[ 1079.651656] video10: VIDIOC_ENUM_FMT: index=4, type=vid-out-mplane, flags=0x1, pixelformat=H263 little-endian (0x33363248), mbus_code=0x0000, description='H.263'
[ 1079.651800] video10: VIDIOC_ENUM_FMT: index=5, type=vid-out-mplane, flags=0x1, pixelformat=MPG2 little-endian (0x3247504d), mbus_code=0x0000, description='MPEG-2 ES'
[ 1079.651945] video10: VIDIOC_ENUM_FMT: error -22: index=6, type=vid-out-mplane, flags=0x0, pixelformat=.... little-endian (0x00000000), mbus_code=0x0000, description=''
[ 1079.652221] video10: VIDIOC_ENUM_FRAMESIZES: index=0, pixelformat=H264 little-endian (0x34363248), type=3, min=32x32, max=1920x1920, step=2x2
[ 1079.652437] video10: VIDIOC_ENUM_FRAMEINTERVALS: error -25: index=0, pixelformat=H264 little-endian (0x34363248), wxh=1920x1920, type=0
[ 1079.652513] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=3, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652545] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652561] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=1, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652590] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652603] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=2, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652632] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652643] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=3, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652672] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652684] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=5, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652712] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652727] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=6, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652755] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652769] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=7, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652797] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652810] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=8, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652839] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652852] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=9, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652880] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652893] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=10, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652922] plane 0: bytesperline=0 sizeimage=786432
[ 1079.652936] video10: VIDIOC_TRY_FMT: type=vid-out-mplane, width=1920, height=1920, format=H264 little-endian (0x34363248), field=none, colorspace=11, num_planes=1, flags=0x0, ycbcr_enc=0, quantization=0, xfer_func=0
[ 1079.652964] plane 0: bytesperline=0 sizeimage=786432
[ 1079.673188] videodev: v4l2_release: video10: release
The GStreamer logging shows:
0:00:01.185869542 845 0x5bc450 DEBUG v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps video/x-h264, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], stream-format=(string)byte-stream, alignment=(string)au, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }, parsed=(boolean)true
0:00:01.186374624 845 0x5bc450 DEBUG v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps video/x-h264, width=(int)[ 32, 1920, 2 ], height=(int)[ 32, 1920, 2 ], framerate=(fraction)[ 0/1, 2147483647/1 ], stream-format=(string)byte-stream, alignment=(string)au, colorimetry=(string){ bt709, bt601, smpte240m, 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12:8, bt2020, 2:0:0:0 }, parsed=(boolean)true
0:00:01.186637710 845 0x5bc450 DEBUG v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps EMPTY
0:00:01.186955957 845 0x5bc450 DEBUG v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps EMPTY
0:00:01.187293851 845 0x5bc450 DEBUG v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps EMPTY
0:00:01.187496980 845 0x5bc450 DEBUG v4l2videodec gstv4l2videodec.c:916:gst_v4l2_video_dec_sink_getcaps:<v4l2h264dec0> Returning sink caps EMPTY
ERROR: from element /GstPipeline:pipeline0/GstH264Parse:h264parse0: Internal data stream error.
Additional debug info:
so I suspect it to be a colourspace mapping thing, and h264parse0 ends up saying that it can't produce something that meets the caps advertised by h264v4l2dec
Enabling more logging with GST_DEBUG=caps*:7 before gst-launch-1.0, I get
0:00:00.161445531 949 0x540450 TRACE caps-features gstcapsfeatures.c:191:gst_caps_features_new_empty: created caps features 0xb4916f60
0:00:00.161521955 949 0x540450 DEBUG capsfilter gstcapsfilter.c:298:gst_capsfilter_transform_caps:<capsfilter1> input: video/x-raw, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3
0:00:00.161573102 949 0x540450 DEBUG capsfilter gstcapsfilter.c:299:gst_capsfilter_transform_caps:<capsfilter1> filter: (NULL)
0:00:00.161625545 949 0x540450 DEBUG capsfilter gstcapsfilter.c:300:gst_capsfilter_transform_caps:<capsfilter1> caps filter: video/x-raw, format=(string)I420
0:00:00.161685284 949 0x540450 DEBUG capsfilter gstcapsfilter.c:302:gst_capsfilter_transform_caps:<capsfilter1> intersect: video/x-raw, format=(string)I420, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3
0:00:00.161767819 949 0x540450 DEBUG capsfilter gstcapsfilter.c:298:gst_capsfilter_transform_caps:<capsfilter1> input: video/x-raw, format=(string)I420, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3
0:00:00.161827744 949 0x540450 DEBUG capsfilter gstcapsfilter.c:299:gst_capsfilter_transform_caps:<capsfilter1> filter: video/x-raw, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3
0:00:00.161886650 949 0x540450 DEBUG capsfilter gstcapsfilter.c:300:gst_capsfilter_transform_caps:<capsfilter1> caps filter: video/x-raw, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3, format=(string)I420
0:00:00.161945907 949 0x540450 DEBUG capsfilter gstcapsfilter.c:302:gst_capsfilter_transform_caps:<capsfilter1> intersect: video/x-raw, width=(int)800, height=(int)480, framerate=(fraction)0/1, colorimetry=(string)2:4:16:3, format=(string)I420
so I read that as h264parse viewing the stream as colorimetry=(string)2:4:16:3 (need to find references to decode that).
ffprobe decodes your clip as " bt470bg/bt470bg/smpte170m".
@6by9, thank you for looking into this. From everything you've detailed, I agree that it looks like a colorspace issue. Similarly, I think there might be two different things going on:
- Gstreamer not linking on buster due to the caps advertised by h264v4l2dec
- Potentially something more firmware based (the color space not being supported?)
Bear with me here as I try and explain why I think this is the case - I might be misinterpreting things.
I've repeated your logging on Buster, with fully updated firmware, and downgraded firmware below the commit that this issue first started appearing. I've replaced the gstreamer logging for a dot file export. The dmesg and resultant pipeline can both be found in the files at the end of this issue.
In both cases, it looks as if v4l2 is setting the same format (I see two VIDIOC_S_FMT statements in each log, one for H264, one for YU12). Similarly, the gstreamer pipelines look identical, with no colorspace cap on h264parse, and v4l2h264dec spitting out bt601.
These tests are what made me feel like something more underlying firmware based is occurring.
For the case of bullseye, it does look as if gstreamer is failing to link, for the reasons you mentioned (colorspace caps). That is to say, I can replicate your results. When the video_signal info is removed from the VUI in the testfile, and the same pipeline tested, again we see the same results as buster: no colorimetry from h264parse, bt601 from v4l2h264dec, S_FMT for h264 & YU12.
Similarly, I can get the gstreamer pipeline to link if I lie to it using capssetter:
gst-launch-1.0 filesrc location=testfile.h264 ! h264parse ! capssetter replace=true caps="video/x-h264, width=(int)800, height=(int)480, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)baseline, level=(string)3.1" ! v4l2h264dec ! video/x-raw,format=I420 ! autovideosink
Similarly, I see the two S_FMT lines, though the pipeline still errors with
ERROR: from element /GstPipeline:pipeline0/v4l2h264dec:v4l2h264dec0: No valid frames decoded before end of stream (the error case from an updated buster install)
And because I was curious, I downgraded the firmware and combined it with the capssetter linking - no difference. bullseye_updated_firmware_capssetter_dmesg.txt bullseye_updated_firmware_video_signal_removed_dmesg.txt buster_firmware_up_to_date_dmesg.txt buster_old_firmware_dmsg.txt
Buster, firmware up to date
Buster, old firmware
Bullseye, updated firmware, video signal removed from testfile.
Bullseye, updated firmware, capssetter
So decoding that colorimetry=(string)2:4:16:3
https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/video-color.c#L149
return g_strdup_printf ("%d:%d:%d:%d", cinfo->range, cinfo->matrix,
cinfo->transfer, cinfo->primaries);
range = 1 -> GST_VIDEO_COLOR_RANGE_0_255 matrix = 4 -> GST_VIDEO_COLOR_MATRIX_BT601 transfer = 16 -> GST_VIDEO_TRANSFER_BT601 primaries = 3 -> GST_VIDEO_COLOR_PRIMARIES_BT470BG
So it is correct that that your stream doesn't match any of the "standard" colorimetries .
However there is the question as to whether h264parse should really be caring about the colorimetry of the encoded stream, or v4l2h264dec should care on the sink pad for the encoded data. Unfortunately I don't think v4l2h264dec gets a choice about it, as it just advertises the list of things it'll support in the caps, and the src pad then matches it.
Otherwise the comment at https://github.com/GStreamer/gst-plugins-good/blob/master/sys/v4l2/gstv4l2object.c#L2394
/* step 2: probe all colorspace other than default
* We don't probe all colorspace, range, matrix and transfer combination to
* avoid ioctl flooding which could greatly increase initialization time
* with low-speed devices (UVC...) */
is an invalid choice in v4l2h264dec.
Having enabled the V4L2 and videobuf2 logging, it looks like GStreamer doesn't implement the dynamic resolution change handling required by the V4L2 Stateful Decoder Interface.
After parsing the H264 headers the decoder has noted a change in the colourspace, and so triggered a resolution/format change event.
- After encountering a resolution change in the stream, the decoder sends a V4L2_EVENT_SOURCE_CHANGE event with changes set to V4L2_EVENT_SRC_CH_RESOLUTION.
GStreamer hasn't subscribed to these events (VIDIOC_SUBSCRIBE_EVENT(V4L2_EVENT_SOURCE_CHANGE)), so it doesn't see it.
The decoder will then process and decode all remaining buffers from before the resolution change point.
The last buffer from before the change must be marked with the V4L2_BUF_FLAG_LAST flag, similarly to the Drain sequence above.
Warning: The last buffer may be empty (with v4l2_buffer bytesused = 0) and in that case it must be ignored by the client, as it does not contain a decoded frame.
Note: Any attempt to dequeue more CAPTURE buffers beyond the buffer marked with V4L2_BUF_FLAG_LAST will result in a -EPIPE error from VIDIOC_DQBUF().
On the next DQBUF the decoder does generate a -EPIPE
[ 635.646035] videodev: v4l2_poll: video10: poll: 00000041
[ 635.646429] videobuf2_common: [cap-71bd83c8] __vb2_wait_for_done_vb: last buffer dequeued already, will not wait for buffers
[ 635.646449] video10: VIDIOC_DQBUF: error -32: 00:00:00.000000000 index=0, type=vid-cap-mplane, request_fd=0, flags=0x00000000, field=any, sequence=0, memory=mmap
The client must continue the sequence as described below to continue the decoding process.
The client doesn't (it misinterprets the -EPIPE as only applying to EOS documented under DRAIN), and therefore decode stops.
gstv4l2src calls gst_v4l2_buffer_pool_enable_resolution_change to subscribe to the event, and gst_v4l2_buffer_pool_poll will then respond with GST_V4L2_FLOW_RESOLUTION_CHANGE, but gstvideodec doesn't.
Ah https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/merge_requests/240 Except I'm confused as https://bugzilla.gnome.org/show_bug.cgi?id=752962 is resolved, and I could have sworn that resolution change had worked. Perhaps it was just detecting the formats outside of V4L2, and setting the port formats correctly.
range = 1 -> GST_VIDEO_COLOR_RANGE_0_255 matrix = 4 -> GST_VIDEO_COLOR_MATRIX_BT601 transfer = 16 -> GST_VIDEO_TRANSFER_BT601 primaries = 3 -> GST_VIDEO_COLOR_PRIMARIES_BT470BG
Double checking this (though I don't think it matters in the wider picture of what's going on) isn't the colorspace being reported range =2 -> GST_VIDEO_COLOR_RANGE_16_235 ?
Sounds like an update to gstreamer (or well, the plugins) is needed to correct this, then?
Double checking this (though I don't think it matters in the wider picture of what's going on) isn't the colorspace being reported range =2 -> GST_VIDEO_COLOR_RANGE_16_235 ?
Yes, typo by me. It's 16_235.
Sounds like an update to gstreamer (or well, the plugins) is needed to correct this, then?
AFAICT this is GStreamer not following the spec, so it would be an update required to the plugin.
I went ahead and recompiled gst-plugins-good with the PR you linked above - the capssetter pipeline works in that case, so seems as though you're correct in the issue at play here. Hopefully that gets merged soon 🤞. Any chance of a kernel flag to work around this until it does? 😅
Seeing as that MR was started 2 years ago, the v4l2src rework was 11 months ago, I suspect it's stalled at present. I'm amazed it'll still apply and compile after all that time.
You may find that changing https://github.com/raspberrypi/linux/blob/rpi-5.15.y/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c#L2533 so MMAL_PARAMETER_VIDEO_STOP_ON_PAR_COLOUR_CHANGE is set to 0 instead of 1 will workaround the issue - that could be done as a module parameter. We can't just ignore the parameter at the kernel level as the firmware will already have stopped when it saw the change.
😅 For what it's worth I actually tried that exact thing a few days back. Tested it again just now - sad to say it doesn't workaround the issue.
You also need to stop it setting the last_buffer_dequeued flag when it receives a MMAL_EVENT_FORMAT_CHANGED event.
index 472d97d1d228..cc6a096d8647 100644
--- a/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c
+++ b/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c
@@ -1077,9 +1077,9 @@ static void handle_fmt_changed(struct bcm2835_codec_ctx *ctx,
q_data->field = V4L2_FIELD_NONE;
}
- vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
- if (vq->streaming)
- vq->last_buffer_dequeued = true;
+ //vq = v4l2_m2m_get_vq(ctx->fh.m2m_ctx, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE);
+ //if (vq->streaming)
+ // vq->last_buffer_dequeued = true;
queue_res_chg_event(ctx);
}
@@ -2530,7 +2530,7 @@ static int bcm2835_codec_create_component(struct bcm2835_codec_ctx *ctx)
* Enable firmware option to stop on colourspace and pixel
* aspect ratio changed
*/
- enable = 1;
+ enable = 0;
vchiq_mmal_port_parameter_set(dev->instance,
&ctx->component->control,
MMAL_PARAMETER_VIDEO_STOP_ON_PAR_COLOUR_CHANGE,
All of these command lines have then worked
pi@raspberrypi:~/tmp $ GST_DEBUG=caps*:7,video*:7,v4l2*:7 gst-launch-1.0 -vvv filesrc location=testfile.h264 ! h264parse ! capssetter caps="video/x-h264, colorimetry=(string)bt601" ! v4l2h264dec ! "video/x-raw,format=I420,colorimetry=(string)bt601" ! filesink location=testfile.i420
pi@raspberrypi:~/tmp $ GST_DEBUG=caps*:7,video*:7,v4l2*:7 gst-launch-1.0 -vvv filesrc location=testfile.h264 ! h264parse ! capssetter caps="video/x-h264, colorimetry=(string)bt601" ! v4l2h264dec ! "video/x-raw,format=NV12,colorimetry=(string)bt601" ! filesink location=testfile.nv12
pi@raspberrypi:~/tmp $ GST_DEBUG=caps*:7,video*:7,v4l2*:7 gst-launch-1.0 -vvv filesrc location=testfile.h264 ! h264parse ! capssetter caps="video/x-h264, colorimetry=(string)bt601" ! v4l2h264dec ! "video/x-raw,format=NV12,colorimetry=(string)bt601" ! kmssink
Thank you - confirming I can replicate your results by applying the same patch
Does NXP MR for resolution changes works with your driver? I've been waiting all this time for folks to actually test this, but I see that workaround in drivers / firmware is more tempting. Note that the submitter has be very slow fixing review comments, but I can take over if some tested-by comment showed up.
For an update, I realized you are looking at an old MR, the current implementation is here: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381
@ndufresne once I get some free time (rare for these next 3 weeks), I'll give a go at compiling the MR linked, and testing the above pipeline with it
I'm building GStreamer master now to have a quick look at how https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 interacts.
@ndufresne https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/1381 looks to solve the issue of v4l2h264dec aborting due to a source changed event.
For @icecube45 's testfile.zip in https://github.com/raspberrypi/firmware/issues/1673#issuecomment-1023866682 I do still need to use capssetter to override the caps from h264parse.
h264parse has determined the colorimetry as colorimetry=(string)2:4:16:3, and v4l2h264dec doesn't include that in the caps it advertises, therefore negotiation fails. Seeing as I don't believe V4L2 can express all the colorimetry values that GStreamer lists, and it doesn't matter what the colorimetry value is within the compressed bitstream, is there a way to avoid it validating that?
Playing back Big Buck Bunny 1080p I do get a couple of v4l2h264dec0: Too old frames, bug in decoder -- please file a bug errors logged. Any guidance on the conditions that this gets reported? Is it just that the decoder took too long on a frame, or is it timestamp based?
I have been trying to get UxPlay to work on Raspberry Pi 4B 8GB (Raspberry Pi OS 64bit, Bullseye) with v4l2h264dec for GPU h264 decoding. Yesterday It finally worked great, but later was no longer working again (on a different source). Gets a problem with caps ... not accepted, then "Internal data stream error". I think it's the same problem being discussed here, because the testfile.h264 given here (earlier) by @icecube45 produced the similar failure he described.
gst-launch-1.0 filesrc location=testfile.h264 ! h264parse ! v4l2h264dec ! video/x-raw,format=I420 ! autovideosink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
ERROR: from element /GstPipeline:pipeline0/GstH264Parse:h264parse0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbaseparse.c(3676): gst_base_parse_loop (): /GstPipeline:pipeline0/GstH264Parse:h264parse0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...
My video pipeline is a h264 stream (Apple iOS Airplay)
appsrc name=video_source stream-type=0 format=GST_FORMAT_TIME is-live=true ! queue ! h264parse ! v4l2h264dec ! v4l2convert ! glimagesink name=video_sink sync=false
works (with poor latency) if v4l2h264dec ! v4l2convert is replaced by avdec_h264 ! videoconvert.
Since I have seen this v4l2 pipeline working beautifully on the RPi4 with a source I cannot find again, I am hoping for some workaround in the pipeline, I am not a gstreamer expert (far from it).
My own error, with GST_DEBUG=2 is
(Begin streaming to GStreamer video pipeline)
0:01:23.953059015 3340 0x559ae8acc0 WARN GST_CAPS gstpad.c:5701:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
0:01:23.953450030 3340 0x559ae8acc0 WARN GST_CAPS gstpad.c:5701:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
0:01:23.953715312 3340 0x559ae8acc0 WARN GST_CAPS gstpad.c:5701:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
0:01:23.953854563 3340 0x559ae8acc0 WARN GST_CAPS gstpad.c:5701:pre_eventfunc_check:<v4l2h264dec0:sink> caps video/x-h264, width=(int)1440, height=(int)1080, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:3:5:1, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high-10, level=(string)4.2 not accepted
0:01:23.973245016 3340 0x559ae87360 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop:<video_source> error: Internal data stream error.
0:01:23.973305161 3340 0x559ae87360 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop:<video_source> error: streaming stopped, reason not-negotiated (-4)
0:01:23.973485688 3340 0x559ae87360 WARN queue gstqueue.c:990:gst_queue_handle_sink_event:<queue4> error: Internal data stream error.
0:01:23.973512130 3340 0x559ae87360 WARN queue gstqueue.c:990:gst_queue_handle_sink_event:<queue4> error: streaming stopped, reason not-negotiated (-4)
(GStreamer error: Internal data stream error.)
