Podman failing to start on UDM SE v3.0.10

[mayankst]

Describe the bug Podan is failing to start on UDM SE v3.0.10 with following error

Oct 18 12:28:33 UDM-SE podman[1465457]: Error: OCI runtime error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

To Reproduce Steps to reproduce the behavior: Start the pi-hole container as service.

ExecStart=/usr/bin/podman run --cidfile=%t/%n.ctr-id --cgroups=no-conmon --pull=always --rm --sdnotify=conmon --replace -d --network dns --name pihole -e "TZ=America/Los Angeles" -v "/mnt/data/etc-pihole/:/etc/pihole/" -v "/mnt/data/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/" --dns=127.0.0.1 --dns=1.1.1.1 --dns=8.8.8.8 --hostname pi.hole -e VIRTUAL_HOST="pi.hole" -e PROXY_LOCATION="pi.hole" -e ServerIP="192.168.4.3" -e IPv6="False" pihole/pihole:latest

UDM Information

• UDM SE
• 3.0.10
• 7.3.73

[chrisdooks]

+1 I think it's because of the upgrade of Debian, it requires Podman v4 to work with the new kernel.

[mayankst]

@chrisdooks Do you know how to build new podman version for unifi? I tried following the instructions in the readme but doesn't seem to work. @boostchicken Can you please help?

If you can provide the instructions to build, then I can build and open a PR.

[chrisdooks]

I'm afraid not, it's above my level of knowledge.

[mayankst]

@boostchicken can you please help. I tried building podman on wsl following your instructions but I am getting compile errors due to missing libseccomp. I tried installing arm64 version of that lib but wasn't successful as dependencies won't install. Main base dependency is libc6:arm64, when I try to install that, apt wants to remove amd64 version of that. So, I am not sure what to do next. Can you please help as pi-hole is completely down due to this incompatibility.

[mayankst]

So, I am able to compile the podman but container still won't start. Keep getting Exit status 1.

[sanderdewit]

I'm interested in this as well. @mayankst can you share the compiling instructions?

[T3LC0]

@mayankst how did you solve the multiarch problem with getting that libc6:arm64 package installed? Trying to compile this myself since support had me upgrade to EA to fix a problem that turned out to not be fixable and I want to run home bridge on my UDM SE with the EA software.

[mayankst]

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

[dquach93]

@mayankst whats the alternative solution you moved to?

[mayankst]

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

[dquach93]

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

Ah i see. Im using this for wpa_supplicant. Didnt see this issue till after updating to 3.0.12 today...

[emilvissing]

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

did you try setting up podman on 3.x using the latest podman/runc/conmon that @boostchicken pushed a couple of days ago?

[TotalGriffLock]

@T3LC0 Yea, I got it to compile but unfortunately looks like Unifi have borked the any ability to run containers in v3 of the OS. There are some major dependencies missing from the OS which are causing runc to not work. At this point I have given up (spent over a month on this researching and debugging) and moved on to alternative solution.

did you try setting up podman on 3.x using the latest podman/runc/conmon that @boostchicken pushed a couple of days ago?

Can confirm same behaviour here on a UDM-SE on v3.0.13 and that's with @boostchicken latest published podman. It would seem that the BPF_CGROUP_DEVICE call has been removed from the new kernel on v3 of the OS.

[kennylerma]

Just noting I have the same issue :( I grabbed the latest podman for UDM Pro SE and still the same start error.

[boostchicken]

https://github.com/fabianishere/udm-kernel-tools

You need that for the UDM-SE

[simkin]

https://github.com/fabianishere/udm-kernel-tools

You need that for the UDM-SE

Kindly elborate as I have some doubts this will work: https://github.com/fabianishere/udm-kernel-tools/issues/82 If it does work, which custom kernel must be loaded ? Edge?

[mazzy89]

@boostchicken the project is not yet ready for UDM Pro SE.

[mazzy89]

Lookign at the kernel file what it has been stripped out by the stock kernel is the CONFIG_BPF_SYSCALL kernel parameter. it must be set to y because Podman same as other containers runtime use nowadays ebpf to hooks programs into the kernel. The only way here is to have a custom kernel into our UDM SE and to achieve this, we need to bring the support for UDM SE into here https://github.com/fabianishere/udm-kernel-tools and have installed into our UDM SE the edge version. I will try to check what it is needed to be done there.

[andrewmiskell]

@mayankst whats the alternative solution you moved to?

Running Ad Guard home on home assistant. Could have done pi-hole on rPi as well, but I already have the home assistant so it was easier.

Ah i see. Im using this for wpa_supplicant. Didnt see this issue till after updating to 3.0.12 today...

On 2.x firmwares, you can just install the Debian package for wpa_supplicant instead of running it in a container so you can break the reliance on podman for that. (Assuming you're using it like I am with AT&T Fiber).

Haven't tried it on 3.x yet (I'm using a UDM and just got the 2.x firmware yesterday).

[SirUli]

Apparently rootless containers fail. e.g. the https://hub.docker.com/r/tusc/chrony-udm still works for me.

[foxcris]

I am on UDM SE v3.0.13. There you can install podman directly via apt from the debian repository. Have you tried to use that package? Installation runs without any errors but i haven't tried to run a container yet. Here the output of podman info:

host:
  arch: arm64
  buildahVersion: 1.19.6
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    distribution: debian
    version: "11"
  eventLogger: journald
  hostname: UDM-SE
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.19.152-ui-alpine
  linkmode: dynamic
  memFree: 161476608
  memTotal: 4137881600
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 7509372928
  swapTotal: 7516188672
  uptime: 191h 36m 17.01s (Approximately 7.96 days)
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus: {}
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.15
  OsArch: linux/arm64
  Version: 3.0.1
```

[mazzy89]

@foxcris it won't work. This is not a problem of podman. The problem is in the OS kernel which come with the newer firmware version.

[foxcris]

@mazzy89 You are right. Just tried the hello world container and i am getting:

podman run hello-world
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 7050e35b49f5 done
Copying config 46331d942d done
Writing manifest to image destination
Storing signatures
Error: open /proc/self/uid_map: no such file or directory

[bennettp123]

@foxcris I think the issue is probably crun—it requires the kernel to be compiles with CONFIG_USER_NS, but the UDM kernels aren't.

What happens if you switch to the older runc ociRuntime instead?

apt-get remove crun
apt-get install runc

[jimmy-ungerman]

What happens if you switch to the older runc ociRuntime instead?

Just tried this on my UDM SE v3.0.13 and still the same issue.

[foxcris]

@bennettp123 Same error when i use runc instaed of crun.

[foxcris]

Isn't it an option to run a standard debian linux kernel?

[frasderp]

@foxcris any progress on this?

[foxcris]

No. I didn't had time to try to use a standard Debian kernel. Don't want to brick my udm se. Till now I can install podman but it's not working due to the missing kernel features.

[cftechwiz]

FWIW - I did actually brick my UDM on one occasion when I accidentally executed an rm -rf at the wrong subfolder and removed a good chunk of the OS. Turns out there is a neat trick to bring it back to stock from an embedded image.
The instructions I got from Support were as follows:

1. Remove the unit from your network and disconnect the cables from the unit.
2. Press down the reset button for 40+ seconds without power and cables.
3. Release the reset button and power the unit.
4. Again press the reset button for 15+ seconds.

I did this and it worked like a charm.