linux H.264 encoder corruption on repeatable image pattern

Describe the bug

The H.264 M2M encoder breaks down when trying to encode something containing a contrasting repeating (at least partially) pattern, for example, a lot of black text on a white background. Below is an example of the original image and the result in PiKVM.

Source:

Encoded:

Encoding parameters:

V4L2_CID_MPEG_VIDEO_BITRATE = 5000k
V4L2_CID_MPEG_VIDEO_H264_I_PERIOD = 30
V4L2_CID_MPEG_VIDEO_H264_PROFILE = V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_BASELINE
V4L2_CID_MPEG_VIDEO_H264_LEVEL = V4L2_MPEG_VIDEO_H264_LEVEL_4_0 // Also tried 5.1, no changes
V4L2_CID_MPEG_VIDEO_REPEAT_SEQ_HEADER = 1
V4L2_CID_MPEG_VIDEO_H264_MIN_QP = 16
V4L2_CID_MPEG_VIDEO_H264_MAX_QP = 32

On increasing the MIN_QP parameter to 20 or 24, the bug will be reproduced only if the screen is completely filled with text. Probably something breaks inside the encoder because the I-frame does not restore the image. Also corruption does not occur immediately, but after 10-20 seconds after showing the text pattern.

Steps to reproduce the behaviour

Use TC358743 capture and https://github.com/pikvm/ustreamer. Open the first screenshot on the video source. Run ustreamer and record the video:

ustreamer --dv-timings --persistent --format=uyvy --verbose --h264-sink=test
(in the another console)
ustreamer-dump --sink=test -o test.h264
(Ctrl+C after a minute)

Then play file test.h264.

Device (s)

Raspberry Pi 4 Mod. B

System

Arch Linux ARM

Aug 26 2022 14:03:16
Copyright (c) 2012 Broadcom
version 102f1e848393c2112206fadffaaf86db04e98326 (clean) (release) (start)

Linux pikvm 5.15.34-4-rpi-ARCH #1 SMP Tue Jul 5 12:10:16 MSK 2022 armv7l GNU/Linux

Additional context

Initial bug report and investigation

Sep 24 '22 16:09 mdevaev

@6by9 Sorry for bothering, could you take a look at this?

Sep 30 '22 12:09 mdevaev

Could someone please take a look at this? This problem is very disturbing.

Oct 21 '22 22:10 mdevaev

I've fixed this. When the frame is an image of a large amount of small text, the size of the encoded frame increases dramatically and can reach 750kB for 720p and 1.2MB for 1080p. So we just need to increase the buffer size. Here is the fix: #5232

Oct 31 '22 12:10 mdevaev

The userspace fix:

https://github.com/raspberrypi/linux/pull/5232#issuecomment-1297022211
https://github.com/pikvm/ustreamer/commit/f1e9d4568cf8b154ca5aec05b98382d8420e6c1e

Oct 31 '22 13:10 mdevaev