bcm2385-codec not reporting about encoder dropping frame

[lmkq222]

Describe the bug

While implementing v4l2 h264 encoding I have stumbled upon an issue of H264 encoder dropping frames when bitrate overshots intended values. According to Stateful-M2M-Encoder docs I expected to receive CAPTURE buffer with V4L2_BUF_FLAG_ERROR set, but looking at the bcm2385-codec it seems like it silently returns a buffer to the VPU without notifying user space.

Steps to reproduce the behaviour

1. Open V4L2 /dev/video11 encoder
2. Setup H264 encoding 720x576p, 25fps, bitrate 10MBit/s, constrained baseline profile, repeate sequence header
3. Open /dev/urandom and feed random data to the encoder
4. Observe encoder dropping frames and not notifying the user space

Device (s)

Raspberry Pi Zero 2 W

System

• Raspberry Pi reference 2025-02-05 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 9a4a00f082b23c94caa210e7441fdf5d6a341056, stage2
• Nov 26 2024 12:54:19 Copyright (c) 2012 Broadcom version 2ae30f53898ae2f1ba77ff570a92991bedfb0398 (clean) (release) (start)
• Linux raspberrypi 6.6.74+rpt-rpi-v8 1 SMP PREEMPT Debian 1:6.6.74-1+rpt1 (2025-01-27) aarch64 GNU/Linux

Logs

No response

Additional context

No response

[6by9]

Forum thread (such as it was) https://forums.raspberrypi.com/viewtopic.php?t=384108

[6by9]

A colleague has just reminded me of this issue.

Rereading the docs, handling of 0 length buffers as EOS only appears to have applied to some decoders, not encoders, hence the callers to vb2_warn_zero_bytesused are both in cases for the OUTPUT queue, not CAPTURE.

Encoding to a 0 byte length packet is not an error, it's inherent in encoding the bitstream. So actually I think we can just drop the whole clause at https://github.com/raspberrypi/linux/blob/rpi-6.12.y/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c#L1207-L1222. Hopefully clients will behave sensible with that.

I'll give it a go.

[lmkq222]

Thanks for the info, removing whole clause seems like a good call. Also, I would like to ask about another issue that perhaps isn't related to the bcm2835-codec, but to mmal translation layer and has been encountered along with current issue of not reporting frames. Encoding white noise using the same configuration: H264 encoding 720x576p, 25fps, bitrate 10MBit/s, constrained baseline profile, repeate sequence header and header joined with first frame sometimes produces buffers with length < 200 bytes. This buffers likely contain sps and pps headers without frame payload as there is 2 NAL units and decoding them with ffmpeg produces no frame error. This buffers occur regardless of frame type (I-Frame, P-Frame) that's should've been produced on current encoding step. Is there any information about this behaviour, is it expected?

[6by9]

Thanks for the info, removing whole clause seems like a good call.

Time is a little on the short side at the moment. Is there any chance you could give that a quick test to see if it solves your problem? You don't actually say what software you're using to feed /dev/urandom into the encoder, so I'd be needing to create something like a gstreamer pipeline to achieve it.

Also, I would like to ask about another issue that perhaps isn't related to the bcm2835-codec, but to mmal translation layer and has been encountered along with current issue of not reporting frames. Encoding white noise using the same configuration: H264 encoding 720x576p, 25fps, bitrate 10MBit/s, constrained baseline profile, repeate sequence header and header joined with first frame sometimes produces buffers with length < 200 bytes. This buffers likely contain sps and pps headers without frame payload as there is 2 NAL units and decoding them with ffmpeg produces no frame error. This buffers occur regardless of frame type (I-Frame, P-Frame) that's should've been produced on current encoding step. Is there any information about this behaviour, is it expected?

I think that's a comment that the same colleague made. If rate control drops an I-frame due to having run out of bits whilst inline headers are enabled, then the headers are still produced but no I-frame follows it.

It's a little awkward as the codec code inserts the headers before each I-frame, whilst the next layer up deals with combining the headers and frame into one buffer when requested (eg by the V4L2 driver). I'd hope that the next frame (which should still be an I-frame) will also get a set of headers prepended, in which case dropping them if the encoded frame is 0-bytes long shouldn't be impossible, but will require some messing around with the firmware code.

[lmkq222]

Time is a little on the short side at the moment. Is there any chance you could give that a quick test to see if it solves your problem?

I probably won't be able to test this fix any time soon as it isn't a high priority one, but I have tested code that returns buffer with error instead of returning it to the VPU and that seemed to help, but the issue got deprioritized, so not much testing was done. I'm pretty sure that proposed change will help and suggest leaving this issue open for the times when either me or someone else will do proper testing of this change.

You don't actually say what software you're using to feed /dev/urandom into the encoder, so I'd be needing to create something like a gstreamer pipeline to achieve it.

I'm feeding /dev/urandom to the encoder via V4L2 API, apologies for not providing a minimal example, there is a lot of code to stitch together.

It's a little awkward as the codec code inserts the headers before each I-frame, whilst the next layer up deals with combining the headers and frame into one buffer when requested (eg by the V4L2 driver). I'd hope that the next frame (which should still be an I-frame) will also get a set of headers prepended, in which case dropping them if the encoded frame is 0-bytes long shouldn't be impossible, but will require some messing around with the firmware code.

This clarifies some things, if time permits I will try to debug it more thoroughly and report back here.

Once again, thanks for your help!