Podman build failure: Connection reset by peer

[igsilya]

Description

Originally created as a https://github.com/actions/runner/issues/1994 . It was suggested in the thread to create a bug for virtual environment, but that didn't happen, IIUC. So, doing that now.

The issue affects multiple projects. Basically, everyone who tries to build podman images in GHA. Freshly failed job: https://github.com/ovn-org/ovn-fake-multinode/actions/runs/3104887855/jobs/5029867767

Quoting the original reporter @ebattat :

Starting to get the following error from 7/12 (Github actions change) when running podman build inside ubuntu-latest container: error running container: error from /usr/bin/crun creating container for [/bin/sh -c dnf update -y --nobest]: sd-bus call: Connection reset by peer

Example for our failed build: The full GitHub actions link The same workflow run successfully several days ago, link

Steps to reproduce

name: Podman test on: workflow_dispatch:

jobs: bundle: runs-on: ubuntu-latest steps: - name: Create container run: | cat <<EOF | sudo podman build -t test-container -f - . FROM fedora:36 RUN echo test EOF Output:

STEP 1/2: FROM fedora:36 Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull registry.fedoraproject.org/fedora:36... Getting image source signatures Copying blob sha256:75f075168a24344c81b41a4877ffc6005723793ced6424f20b5f3ee9d6a66b57 Copying blob sha256:75f075168a24344c81b41a4877ffc6005723793ced6424f20b5f3ee9d6a66b57 Copying config sha256:3a66698e604003f7822a0c73e9da50e090fda9a99fe1f2e1e2e7fe796cc803d5 Writing manifest to image destination Storing signatures STEP 2/2: RUN echo test error running container: error from /usr/bin/crun creating container for [/bin/sh -c echo test]: sd-bus call: Connection reset by peer : exit status 1

For more detail

**Workaround for GitHub actions till fix: ** sudo -E XDG_RUNTIME_DIR= podman build -t test-container -f - .

Platforms affected

• [ ] Azure DevOps
• [X] GitHub Actions

Runner images affected

• [ ] Ubuntu 18.04
• [X] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] macOS 10.15
• [ ] macOS 11
• [ ] macOS 12
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

Version: 20220905.1 https://github.com/ovn-org/ovn-fake-multinode/actions/runs/3104887855/jobs/5029867767

Is it regression?

It is a regression. Successful build from the original reporter: https://github.com/redhat-performance/benchmark-runner/runs/7229627245?check_suite_focus=true

Expected behavior

Container is able to be built.

Actual behavior

Containers are not built with: error running container: error from /usr/bin/crun creating container for [/bin/sh -c echo test]: sd-bus call: Connection reset by peer: exit status 1

Repro steps

Run any job that builds a podman container.

[igorboskovic3]

@igsilya Thank you for reporting this one. We will take a look and get back to you.

[al-cheb]

Hey @igsilya. Could you please try to set XDG_RUNTIME_DIR to emtpy value?

env:
      XDG_RUNTIME_DIR: ''

[igsilya]

@al-cheb that workaround works, yes. But it is a workaround.

[al-cheb]

@al-cheb that workaround works, yes. But it is a workaround.

You can reproduce the same behavior on a local vm:

user@ubnt01:~/ovn-fake-multinode$ id -a
uid=1000(user) gid=1000(user) groups=1000(user),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),110(lxd),999(docker)
user@ubnt01:~/ovn-fake-multinode$ echo $XDG_RUNTIME_DIR
/run/user/1000
user@ubnt01:~/ovn-fake-multinode$ sudo -E ./ovn_cluster.sh build
error running container: error from /usr/bin/crun creating container for [/bin/sh -c /install_pkg.sh $OS_IMAGE]: sd-bus call: Transport endpoint is not connected

sudo -E preserves a user environment. These point to directories with UID names, which not intended to be used by another user.

[igsilya]

sudo -E preserves a user environment. These point to directories with UID names, which not intended to be used by another user.

Yes, but we need that environment to not pass all the other variables manually.

[ebattat]

Hi @al-cheb, @igsilya

I agreed, I think that this issue should be solved because it worked properly in the past:

Actual: sudo -E XDG_RUNTIME_DIR= podman build -t test-container -f - .
Expected:  podman build -t test-container -f - .

[al-cheb]

Hi @al-cheb, @igsilya

I agreed, I think that this issue should be solved because it worked properly in the past:

Actual: sudo -E XDG_RUNTIME_DIR= podman build -t test-container -f - .
Expected:  podman build -t test-container -f - .

@ebattat, I can reproduce the same behavior on my local vm:

user@ubnt01:~$ id
uid=1000(user) gid=1000(user) groups=1000(user),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),110(lxd),999(docker)
user@ubnt01:~$ sudo -E podman build -t test-container .
STEP 1/2: FROM fedora:36
STEP 2/2: RUN echo test
error running container: error from /usr/bin/crun creating container for [/bin/sh -c echo test]: sd-bus call: Transport endpoint is not connected
: exit status 1

[erik-bershel]

@igsilya The problem is reproducible in any environment and is related to the operation of the application, not related to compatibility or version. Therefore, the issue is not related to the operation of the runner images. I have to close the issue. If you have any other questions or concerns, feel free to contact us at any time.