RPi 4: H.264 and HEVC hardware decoding

[Mynacol]

Hi, I moved my Kodi setup on my Raspberry Pi 4 from LibreELEC to NixOS. This repository with the hardware-specific settings was very helpful. Unfortunately, one important thing still doesn't work: Hardware video decoding acceleration. Currently, Kodi only software decodes videos, which is choppy at H.264 1080p at best.

I enabled hardware.raspberry-pi."4".fkms-3d.enable already, else Kodi wouldn't even start. If I enable DRM PRIME decoding in Kodi's settings, video output is broken and the log spits a lot of this line

ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: Invalid data found when processing input (-1094995529)

The exact error is different depending on video codec (the above was H.264), but always in the class CDVDVideoCodecDRMPRIME.

The reason I am reporting this here: According to one forum poster, hardware decoding with this setup requires not firmware KMS but "kernel"/"real" KMS. I even tried to add my own dtoverlay based on vc4-kms-v3d-overlay and vc4-kms-v3d-pi4-overlay, but video output after reboot was always broken. The latest version of them is here: vc4-kms.nix, vc4-kms-pi4.nix.

Hence my core question: Has anyone successfully set up hardware video decoding on raspberry pi 4 and NixOS? I'm at a loss and thought one of you could help me. Would "real" KMS help?

[CrunchyFlakes]

I'm stuck with your exact same problem. I have already tried adding the vc4-kms-3d overlay to the config.txt on the firmware partition, but it seems to have no effect (changing config.txt was suggested here: https://nixos.org/guides/installing-nixos-on-a-raspberry-pi.html)

[Mynacol]

I'm somewhat glad I'm not alone with this issue :fist_left:

After poking around once more, I at least solved one problem. Essentially, Kodi needs access to /dev/dma_heap/linux,cma, which NixOS doesn't grant by default. Creating a fitting udev rule now lets me use Kodi with DRM PRIME decoding enabled. The forum post that had the same problem is here and they link to here for udev rules. I just copied all of them over. My NixOS config.nix now has:

  services.udev.extraRules = ''
    # HDMI-CEC
    SUBSYSTEM=="vchiq", GROUP="video", MODE="0660", TAG+="systemd", ENV{SYSTEMD_ALIAS}="/dev/vchiq"
    # permissions from https://github.com/graysky2/kodi-standalone-service/blob/master/arm/udev/99-kodi.rules
#    SUBSYSTEM=="vc-sm",GROUP="video",MODE="0660"
#    SUBSYSTEM=="tty",KERNEL=="tty[0-9]*",GROUP="tty",MODE="0660"

    # https://github.com/RPi-Distro/raspberrypi-sys-mods/blob/master/etc.armhf/udev/rules.d/99-com.rules#L7
    SUBSYSTEM=="input", GROUP="input", MODE="0660"
    SUBSYSTEM=="i2c-dev", GROUP="i2c", MODE="0660"
    SUBSYSTEM=="spidev", GROUP="spi", MODE="0660"
    SUBSYSTEM=="bcm2835-gpiomem", GROUP="gpio", MODE="0660"
    SUBSYSTEM=="rpivid-*", GROUP="video", MODE="0660"
    KERNEL=="vcsm-cma", GROUP="video", MODE="0660"
    SUBSYSTEM=="dma_heap", GROUP="video", MODE="0660"
    SUBSYSTEM=="gpio", GROUP="gpio", MODE="0660"
  '';

And kodi is, among others, in the video group obviously. Should we add these udev rules to nixos-hardware❓

Unfortunately, the DRMPRIME backend is still software, not hardware decoding... No idea why.

Additionally, I noticed (with DRMPRIME SW decoding) a file descriptor leak, lots of anon_inode:sync_file fd's. This leads to frequent Kodi freezes and crashes, great :see_no_evil: I found similar leaks in mesa, but affecting older versions.

[norbertkeri]

Did anyone get this working?