(dev) Podman 5.0.3 dependencies breaks abroot

[mirkobrombin]

Our latest sync introduced Podman 5.0.3 breaking abroot:

STEP 1/5: FROM ghcr.io/vanilla-os/nvidia@sha256:7caffbd2cc72bfc78467828be6ec98e8ea51522d442dcd08b7d4a77ee7b8c9b9
STEP 2/5: LABEL ABRoot.BaseImageDigest=sha256:7caffbd2cc72bfc78467828be6ec98e8ea51522d442dcd08b7d4a77ee7b8c9b9
STEP 3/5: LABEL maintainer='Generated by ABRoot'
STEP 4/5: LABEL ABRoot.root=vos-a
STEP 5/5: RUN true
[INFO  netavark::firewall] Using iptables firewall driver
error running container: did not get container start message from parent: EOF
(Verbose) OciExportRootFs:err(7.000000):building at STEP "RUN true": setup network: netavark (exit code 1): No such file or directory (os error 2)
(Verbose) ABSystem.RunOperation:err(4.200000):building at STEP "RUN true": setup network: netavark (exit code 1): No such file or directory (os error 2)
(Verbose) Partition.Unmount:info:running...
(Verbose) Partition.Unmount:info:successfully unmounted/part-future/
(Verbose) ABSystem.UnlockUpgrade:info:lock file removed
  ERROR   building at STEP "RUN true": setup network: netavark (exit code 1): No such file or directory (os error 2)
Error: building at STEP "RUN true": setup network: netavark (exit code 1): No such file or directory (os error 2)
Usage:
  abroot upgrade [flags]

Examples:
abroot upgrade

Flags:
  -c, --check-only   check for updates but do not apply them
  -d, --dry-run      perform a dry run of the operation
  -f, --force        force update even if the system is up to date
  -h, --help         help for upgrade

Global Flags:
  -V, --verbose   show more detailed output

this might not be related to podman itself but since prometheus uses the same configurations and libraries, might be related to one of its configurations and/or dependencies.

What I tried

Since logs of the containers' projects are always confusing and everything but not verbose, I assumed netavark was not being found, then I trying to explicitly install it but abroot was still failing, then I tried to link it to /usr/bin to make it discoverable (it gets installed in /usr/lib/podman/) but still no luck.

Ideas

• switch to a different networking backend in /etc/containers/containers.conf
• downgrade podman manually
• upgrade prometheus dependencies (this might introduce more issues than those it can solve)

[axtloss]

We just have to install netavarak in the image, it's an optional dependency so we have to install it ourselves

[mirkobrombin]

We just have to install netavarak in the image, it's an optional dependency so we have to install it ourselves

did it already, the issue persist: https://github.com/Vanilla-OS/core-image/commit/e57401ab50f6597439f0af00e802bf60f185febb

[fluoriteByte]

have you tried the versions before 5.0.3+ds1-5 since that version seems to add netavark as a hard dependency

[fluoriteByte]

i would suggest starting with 5.0.3+ds1-1 and see if that version works

[axtloss]

podman switched to netavark as the default backend with podman 5, any packages on debian that do not depend on netavark are broken packages, and wouldn't resolve the issue.

[axtloss]

We probably need to explicitly install iptables and aardvark-dns, not only netavark

[mirkobrombin]

iptables is there already, not sure about aardvark-dns

[jardon]

Is this still an issue? I see that podman version 5.2.1 is in debian unstable

[mirkobrombin]

Is this still an issue? I see that podman version 5.2.1 is in debian unstable

it's still an issue because we are not sure it's completely podman's fault (since we don't use it in abroot)

[jardon]

i really wouldnt mind looking at this, but i dont have enough context for whats going on here to know how to reproduce it.

[mirkobrombin]

To reproduce it, just change the tag image in the abroot config (abroot config-editor) pointing it to dev, then upgrade (abroot upgrade -f), restart and then force upgrade again to see the error.

[jardon]

Could be related to this

The linked issue is that the $PATH is missing the directory to find iptables and has the same error. Haven't confirmed that yet or what the fix would be.

Edit: Also there's this post but it has no responses :/

[mirkobrombin]

Found the issue. The iptables binary is not installed, not sure why but even if the iptables package was marked as installed, the iptables binary is not being placed in /usr/bin nor /usr/sbin. Forcing re-installing it fixed the issue.