[BUG] MPP decoder passes unsafe PPS scaling-list data to RKVDEC2, leading to kernel Oops in task finalize (RK3588)

[chrisbeeze]

Hardware: RK3588 Kernel: rockchip-linux/kernel 5.10.160 (also seen on 6.1/6.6 lines) GStreamer: 1.18.5

Related kernel issue (driver side): https://github.com/rockchip-linux/kernel/issues/362

Summary

During H.264/H.265 playback a reproducible kernel crash is triggered in the RKVDEC path (mpp_dma_release → mpp_task_finalize → rkvdec*_free_task). The stream contains PPS with scaling-list fields that appear out-of-range / malformed. Note: The analysis below reflects our idea/hypothesis; please verify feasibility and correctness on the MPP side.

Suspected userspace (MPP) problems

1. Decoder parser does not fully validate PPS/SPS scaling-list lengths/offsets before building buffers/offsets that are later mapped into the kernel. This can allow out-of-bounds reads/writes when the driver copies PPS data (see kernel issue above for details).

2. Task lifecycle robustness: on error paths, MPP may submit or finalize a "partially initialized" task (e.g., dma/cmd buffers not fully prepared), which then causes invalid resource releases in the driver.

Why MPP is likely involved

• MPP parses parameter sets and forms the PPS/SPS structures and offsets consumed by the kernel. If malformed fields are not clamped/validated in userspace, the kernel can still be crashed even if it adds checks.
• MPP release notes show nearby fixes (e.g., h264e_pps: add pic_scaling_matrix_present check, h2645d_sei: fix read byte overflow error), suggesting similar bounds checks may be needed on the decoder PPS/ScalingList path.

Minimal reproducer

GStreamer: H.265: gst-launch-1.0 -v rtspsrc location="<RTSP>" protocols=tcp latency=200 !
rtph265depay ! h265parse ! mppvideodec ! fakesink sync=false H.264: gst-launch-1.0 -v rtspsrc location="<RTSP>" protocols=tcp latency=200 !
rtph264depay ! h264parse ! mppvideodec ! fakesink sync=false

Observed behavior

• Kernel Oops in mutex_lock → mpp_dma_release → mpp_task_finalize → rkvdec*_free_task.
• Occurs shortly after PPS is parsed / task prepared.

What would help to fix in MPP

• Add strict validation for PPS/SPS scaling-list presence, lengths, and derived offsets in h264d/h265d parser code before any buffer/offset is exposed to the kernel.
• Guard task finalize paths to tolerate partially constructed tasks (no unconditional finalize).
• Optionally add a contract check to refuse tasks that would map pointers beyond PPS buffers (defensive programming on the userspace side, mirroring kernel checks).

This reflects our current hypothesis; please verify feasibility and correctness on the MPP side. Thank you.

@HermanChen @JeffyCN