Support H.264 and H.265 (HEVC) ffmpeg hardware decode/encode on OrangePi 5

[artem-zinnatullin]

Hi!

I'm trying to get ffmpeg -hwaccel drm to work so both decoding and encoding of H.264 and H.265 (HEVC) videos can happen with hardware acceleration on OrangePi 5 with its Mali G610 GPU.

References:

• https://developer.arm.com/Processors/Mali-G610
• https://www.cpu-monkey.com/en/igpu-arm_mali_g610_mp6

My understanding is that Mali G610 supports hardware accelerated encoding/decoding for H.265 and H.264, however I can't seem to get it fully working.

apt-get update
apt-get install ffmpeg
# This brings tons of packages including mali-g610-firmware

Now let's list relevant available HW decoders:

ffmpeg -decoders | grep -i rkmpp
 V..... av1_rkmpp            av1 (rkmpp) (codec av1)
 V..... h263_rkmpp           h263 (rkmpp) (codec h263)
 V..... h264_rkmpp           h264 (rkmpp) (codec h264)
 V..... hevc_rkmpp           hevc (rkmpp) (codec hevc)
 V..... mpeg1_rkmpp          mpeg1 (rkmpp) (codec mpeg1video)
 V..... mpeg2_rkmpp          mpeg2 (rkmpp) (codec mpeg2video)
 V..... mpeg4_rkmpp          mpeg4 (rkmpp) (codec mpeg4)
 V..... vp8_rkmpp            vp8 (rkmpp) (codec vp8)
 V..... vp9_rkmpp            vp9 (rkmpp) (codec vp9)

Now let's list relevant available HW encoders:

ffmpeg -encoders | grep -i omx
 V..... h264_omx             OpenMAX IL H.264 video encoder (codec h264)
 V..... mpeg4_omx            OpenMAX IL MPEG-4 video encoder (codec mpeg4)

So it appears we have HW H.265 (decoder only??) and HW H.264 decoder and encoder for Mali G610.

---

Let's try to convert a video from H.265 to H.264:

• Input test video: https://test-videos.co.uk/bigbuckbunny/mp4-h265 (I use first 1080p 1MB video)

ffmpeg -hwaccel drm -hwaccel_device /dev/dri/renderD128 -i test-h265.mp4 -c:v h264_omx test-out-h264.mp4

# Some output omitted, here are important bits

# The input stream is H.265 HEVC
Stream #0:0(und): Video: hevc (hev1 / 0x31766568), yuv420p(progressive), 1920x1080, 834 kb/s

# Input decoder picked is HW accelerated and output encoder is also HW accelerated implementations
# otherwise it would've been (hevc (native) -> h264 (libx264)) where "native" and "libx264" indicates CPU implementations
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_rkmpp) -> h264 (h264_omx))

# !!! H.264 output encoder fails due to missing libOMX_Core and ffmpeg errors out !!!
[h264_omx @ 0xaaaac0d2c700] libOMX_Core.so not found
[h264_omx @ 0xaaaac0d2c700] libOmxCore.so not found
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

If I remove HW output encoder, it will successfully run on SW encoder (eating all CPU) and HW input decoder:

ffmpeg -hwaccel drm -hwaccel_device /dev/dri/renderD128 -i test-h265.mp4 test-h264.mp4

So the HW H.265 input decoder seems to work, but HW H.264 output encoder doesn't!

---

Now let's try to convert a video from H.264 to H.265

• Input test video: https://test-videos.co.uk/bigbuckbunny/mp4-h264 (I use first 1080p 1MB video)

ffmpeg -hwaccel drm -hwaccel_device /dev/dri/renderD128 -i test-h264.mp4 -c:v libx265 test-out-h265.mp4

# Some output omitted, here are important bits

# The input stream is H.264
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 832 kb/s

# Input decoder picked is HW accelerated however output  encoder is SW because we don't seem to have HW output encoder built in this ffmpeg build
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_rkmpp) -> hevc (libx265))

# Conversion is very slow and eats all available CPU due to SF output encoding
frame=  167 fps=1.5 q=28.1 size=     256kB time=00:00:02.26 bitrate= 925.3kbits/s dup=0 drop=1 speed=0.02x

---

TL;TR: as of Joshua-Riek/ubuntu-rockchip:v1.19 HW input decoding of H.264 and H.265 (HEVC) works, but HW output encoding of H.264 and H.265 doesn't I think.

• It seems that for H.264 HW output encoding we need libOMX_Core.so in the distro and it's unclear (to me) where can one get it
• It seems that for H.265 (HEVC) HW output encoding we need ffmpeg built with that encoder support (another apt-get ppa for ffmpeg?)

HW decoding/encoding is essential for popular use case of OrangePi 5 boards — Frigate NVR and other similar cases when the board needs to efficiently encode/decode video streams.