runc: "/sys" caused: mount through procfd: operation not permitted: OCI permission denied, but crun works.

[jianzhangbjz]

Description

Podman runc failed at the following, but crun works. Hope the runc can support it as the crun done. Original issue https://github.com/containers/podman/discussions/19524

[cloud-user@preserve-olm-env2 jian]$ podman run --rm -ti --entrypoint /bin/bash  registry.ci.openshift.org/ocp/4.14@sha256:8235c041ce1cb343b27301743414e31b7ab0fa9c57a0217fba4cd892d32e3e42
Error: runc: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:75: mounting "sysfs" to rootfs at "/sys" caused: mount through procfd: operation not permitted: OCI permission denied
[cloud-user@preserve-olm-env2 jian]$ 
[cloud-user@preserve-olm-env2 jian]$ podman --runtime crun run --rm -ti --entrypoint /bin/bash  registry.ci.openshift.org/ocp/4.14@sha256:8235c041ce1cb343b27301743414e31b7ab0fa9c57a0217fba4cd892d32e3e42
[root@8ca488ca7e98 /]# 

Steps to reproduce the issue

Describe the results you received and expected

/sys/fs/cgroup has a read only bind mount so the kernel blocks mounting a fresh sys. The error from the kernel is expected. crun has a fallback in this case, that is the difference with runc.

INFO[0000] Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for cpu: mkdir /sys/fs/cgroup/cpu/conmon: permission denied 
DEBU[0000] Received: -1 

What version of runc are you using?

[cloud-user@preserve-olm-env2 ~]$ runc --version runc version 1.0.3 spec: 1.0.2-dev go: go1.16.7 libseccomp: 2.5.1

Host OS information

[cloud-user@preserve-olm-env2 ~]$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="8.4 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.4"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.4"

Host kernel information

[cloud-user@preserve-olm-env2 ~]$ uname -a 
Linux preserve-olm-env2 4.18.0-287.el8.dt4.x86_64 #1 SMP Thu Feb 18 13:31:55 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

[rata]

@jianzhangbjz do you happent to have the config.json handy?

IIRC crun does a bind-mount of /sys when it fails. I don't think that is according to the spec, but it might make sense to do it here anyways IF indeed that is the difference and fixes the issue.

What do others think?

[jianzhangbjz]

Hi @rata , sorry, which config.json file? Thanks! Here is the original discussion: https://github.com/containers/podman/discussions/19524

[rata]

@jianzhangbjz when podman calls runc, it creates a config.json file with the configuration runc uses.

One hack to get it can be this: create a /usr/local/sbin/runc script with this:

#!/bin/bash

echo "Getting para ${8}"  >> /tmp/rata.log
if [ "${8}" = "--bundle" ]; then
       echo "Getting config.json"  >> /tmp/rata.log
       mkdir -p /tmp/rata-debug-k8s/
       cp -ar "${9}" "/tmp/rata-debug-k8s/$$/"
       echo "Getting param: ${9}" >> /tmp/rata.log
fi

exec <path-to-runc> --debug "$@"

This will copy the config.json and other files to /tmp/rata-debug-k8s/. Can you get that file for the failing container?

And ideally, can you run a more recent runc version, just in case? (I doubt this has changed, but what you are using is very old).

[rata]

Friendly ping?