gvisor
gvisor copied to clipboard
google
bug: gvisor cannot join a pod when running containers as runtime
Description
After creating and running a podman pod with gvisor, trying to use gvisor to run a new container added to the pod has a problem
Additional information: https://github.com/containers/podman/issues/15223
Describe the results expected:
!... Hello Podman World ...!
.--"--.
/ - - \
/ (O) (O) \
~~~| -=(,Y,)=- |
.---. /` \ |~~
~/ o o \~~~~.----. ~~
| =(X)= |~ / (O (O) \
~~~~~~~ ~| =(Y_)=- |
~~~~ ~~~| U |~~
Project: https://github.com/containers/podman
Website: https://podman.io
Documents: https://docs.podman.io
Twitter: @Podman_io
Describe the results received:
Error: OCI runtime error: runsc: panic: interface conversion: cgroup.Cgroup is *cgroup.cgroupSystemd, not *cgroup.cgroupV2 [recovered]
panic: interface conversion: cgroup.Cgroup is *cgroup.cgroupSystemd, not *cgroup.cgroupV2
goroutine 1 [running]:
encoding/json.(*encodeState).marshal.func1()
GOROOT/src/encoding/json/encode.go:328 +0x6e
panic({0xf41d80, 0xc0003c1ce0})
GOROOT/src/runtime/panic.go:838 +0x207
gvisor.dev/gvisor/runsc/cgroup.(*CgroupJSON).MarshalJSON(0xc000124620)
runsc/cgroup/cgroup.go:474 +0x1a5
encoding/json.addrMarshalerEncoder(0xc00053c400, {0xf8ba60?, 0xc000124620?, 0x523087?}, {0x28?, 0xc4?})
GOROOT/src/encoding/json/encode.go:496 +0x102
encoding/json.condAddrEncoder.encode({0x1109978?, 0xc000107050?}, 0x500475?, {0xf8ba60?, 0xc000124620?, 0x2?}, {0xf?, 0x0?})
GOROOT/src/encoding/json/encode.go:960 +0x4a
encoding/json.structEncoder.encode({{{0xc000146900?, 0x40eb70?, 0x7fecc34b4888?}, 0xc000107260?}}, 0xc00053c400, {0x10470c0?, 0xc0001245a0?, 0xc0000a70e0?}, {0x0, 0x1})
GOROOT/src/encoding/json/encode.go:761 +0x1f4
encoding/json.ptrEncoder.encode({0xc00050c068?}, 0xc00053c400, {0x106ea20?, 0xc0001245a0?, 0x106ea20?}, {0x84?, 0x31?})
GOROOT/src/encoding/json/encode.go:945 +0x25e
encoding/json.(*encodeState).reflectValue(0xc000587178?, {0x106ea20?, 0xc0001245a0?, 0x1b94060?}, {0x0?, 0x73?})
GOROOT/src/encoding/json/encode.go:360 +0x78
encoding/json.(*encodeState).marshal(0xc0000a7060?, {0x106ea20?, 0xc0001245a0?}, {0x30?, 0x70?})
GOROOT/src/encoding/json/encode.go:332 +0xfa
encoding/json.Marshal({0x106ea20, 0xc0001245a0})
GOROOT/src/encoding/json/encode.go:161 +0x45
gvisor.dev/gvisor/runsc/container.(*StateFile).saveLocked(0xc000124638, {0x106ea20, 0xc0001245a0})
runsc/container/state_file.go:315 +0x5d
gvisor.dev/gvisor/runsc/container.(*Container).saveLocked(0xc0001245a0)
runsc/container/container.go:801 +0xa5
gvisor.dev/gvisor/runsc/container.New(0xc00048c000, {{0x7ffd8e32bef0, 0x40}, 0xc0000c4e00, {0x7ffd8e32bdeb, 0x78}, {0x0, 0x0}, {0x7ffd8e32be6f, 0x80}, ...})
runsc/container/container.go:329 +0x10f3
gvisor.dev/gvisor/runsc/cmd.(*Create).Execute(0xc0000c2d40, {0xc0000ac970?, 0xc0003e9998?}, 0x5352c6?, {0xc0000d4fe0?, 0x7fecea2f1b00?, 0x20?})
runsc/cmd/create.go:112 +0x265
github.com/google/subcommands.(*Commander).Execute(0xc0000c4000, {0x12ca7e8, 0xc0000b0000}, {0xc0000d4fe0, 0x2, 0x2})
external/com_github_google_subcommands/subcommands.go:200 +0x3bc
github.com/google/subcommands.Execute(...)
external/com_github_google_subcommands/subcommands.go:481
gvisor.dev/gvisor/runsc/cli.Main({0x12ba110, 0x12})
runsc/cli/main.go:240 +0x5486
main.main()
runsc/main.go:23 +0x27
Steps to reproduce
- run:
sudo podman --runtime runsc --runtime-flag debug --runtime-flag debug-log=/tmp/trsc-pod.log pod create --infra-name trsc-pod-infra --hostname trsc-pod-host --name trsc-pod --replace && sudo podman --runtime crun --runtime-flag debug --runtime-flag debug-log=/tmp/trsc-pod.log.2 pod start trsc-pod
- run:
sudo podman --runtime runsc --runtime-flag debug --runtime-flag debug-log=/tmp/trsc-container-t.log run --name trsc-container-t --pod trsc-pod --replace quay.io/podman/hello
runsc version
runsc version release-20220808.0
spec: 1.0.2-dev
docker version (if using docker)
# podman info
host:
arch: amd64
buildahVersion: 1.27.0
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.4-1
path: /usr/bin/conmon
version: 'conmon version 2.1.4, commit: bd1459a3ffbb13eb552cc9af213e1f56f31ba2ee'
cpuUtilization:
idlePercent: 77.83
systemPercent: 5.7
userPercent: 16.48
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: arch
idMappings:
gidmap: null
uidmap: null
kernel: 5.19.6-zen1-1-zen
linkmode: dynamic
logDriver: journald
memFree: 152612864
memTotal: 6599462912
networkBackend: cni
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.5-2
path: /usr/bin/crun
version: |-
crun version 1.5
commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 7566929920
swapTotal: 9741783040
uptime: 52h 19m 49.00s (Approximately 2.17 days)
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.fedoraproject.org
- quay.io
- registry.access.redhat.com
- docker.io
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 12
paused: 0
running: 6
stopped: 6
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev
graphRoot: /var/lib/containers/storage
graphRootAllocated: 67006668800
graphRootUsed: 52119412736
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "true"
imageCopyTmpDir: /var/tmp
imageStore:
number: 21
runRoot: /var/run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 4.2.0
Built: 1660386415
BuiltTime: Sat Aug 13 18:26:55 2022
GitCommit: 7fe5a419cfd2880df2028ad3d7fd9378a88a04f4-dirty
GoVersion: go1.19
Os: linux
OsArch: linux/amd64
Version: 4.2.0
uname
Linux arch 5.19.6-zen1-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Wed, 31 Aug 2022 22:09:42 +0000 x86_64 GNU/Linux
kubectl (if using Kubernetes)
No response
repo state (if built from source)
No response
runsc debug logs (if available)
trsc-container-t.log:
I0904 22:00:13.597976 127613 main.go:213] ***************************
I0904 22:00:13.599555 127613 main.go:214] Args: [/usr/bin/runsc --systemd-cgroup --debug --debug-log=/tmp/trsc-container-t.log --log-format=json --log /var/run/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/oci-log create --bundle /var/lib/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata --pid-file /var/run/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/pidfile 4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2]
I0904 22:00:13.599687 127613 main.go:215] Version release-20220808.0
I0904 22:00:13.599751 127613 main.go:216] GOOS: linux
I0904 22:00:13.599839 127613 main.go:217] GOARCH: amd64
I0904 22:00:13.599922 127613 main.go:218] PID: 127613
I0904 22:00:13.600009 127613 main.go:219] UID: 0, GID: 0
I0904 22:00:13.600122 127613 main.go:220] Configuration:
I0904 22:00:13.600938 127613 main.go:221] RootDir: /var/run/runsc
I0904 22:00:13.600989 127613 main.go:222] Platform: ptrace
I0904 22:00:13.601042 127613 main.go:223] FileAccess: exclusive, overlay: false
I0904 22:00:13.601179 127613 main.go:224] Network: sandbox, logging: false
I0904 22:00:13.601302 127613 main.go:225] Strace: false, max size: 1024, syscalls:
I0904 22:00:13.601403 127613 main.go:226] LISAFS: false
I0904 22:00:13.601458 127613 main.go:227] Debug: true
I0904 22:00:13.601511 127613 main.go:228] Systemd: true
I0904 22:00:13.601563 127613 main.go:229] ***************************
W0904 22:00:13.605479 127613 specutils.go:113] noNewPrivileges ignored. PR_SET_NO_NEW_PRIVS is assumed to always be set.
D0904 22:00:13.608852 127613 specutils.go:75] Spec:
{
"ociVersion": "1.0.2-dev",
"process": {
"user": {
"uid": 1000,
"gid": 0,
"umask": 18
},
"args": [
"/usr/local/bin/podman_hello_world"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"container=podman",
"HOSTNAME=trsc-pod-host"
],
"cwd": "/",
"rlimits": [
{
"type": "RLIMIT_NOFILE",
"hard": 1048576,
"soft": 1048576
},
{
"type": "RLIMIT_NPROC",
"hard": 4194304,
"soft": 4194304
}
]
},
"root": {
"path": "/var/lib/containers/storage/overlay/94ae7f7dc39150bc76b9c69326f281dfa3f8374e3adb01cb5db3e285eb9d03ec/merged"
},
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "/var/lib/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/proc",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/dev",
"type": "tmpfs",
"source": "/var/lib/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/tmpfs",
"options": [
"nosuid",
"noexec",
"strictatime",
"mode=755",
"size=65536k"
]
},
{
"destination": "/sys",
"type": "sysfs",
"source": "/var/lib/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/sysfs",
"options": [
"nosuid",
"noexec",
"nodev",
"ro"
]
},
{
"destination": "/dev/pts",
"type": "devpts",
"source": "/var/lib/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/devpts",
"options": [
"nosuid",
"noexec",
"newinstance",
"ptmxmode=0666",
"mode=0620",
"gid=5"
]
},
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "/var/lib/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/mqueue",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/etc/resolv.conf",
"type": "bind",
"source": "/var/run/containers/storage/overlay-containers/256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6/userdata/resolv.conf",
"options": [
"bind",
"rprivate"
]
},
{
"destination": "/etc/hosts",
"type": "bind",
"source": "/var/run/containers/storage/overlay-containers/256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6/userdata/hosts",
"options": [
"bind",
"rprivate"
]
},
{
"destination": "/dev/shm",
"type": "bind",
"source": "/var/lib/containers/storage/overlay-containers/256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6/userdata/shm",
"options": [
"bind",
"rprivate",
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/etc/hostname",
"type": "bind",
"source": "/var/run/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/hostname",
"options": [
"bind",
"rprivate"
]
},
{
"destination": "/run/.containerenv",
"type": "bind",
"source": "/var/run/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/.containerenv",
"options": [
"bind",
"rprivate"
]
},
{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "/var/lib/containers/storage/overlay-containers/4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2/userdata/cgroup",
"options": [
"rprivate",
"nosuid",
"noexec",
"nodev",
"relatime",
"ro"
]
}
],
"annotations": {
"io.container.manager": "libpod",
"io.kubernetes.cri-o.ContainerType": "container",
"io.kubernetes.cri-o.Created": "2022-09-04T22:00:13.370706694+08:00",
"io.kubernetes.cri-o.SandboxID": "256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6",
"io.kubernetes.cri-o.TTY": "false",
"io.podman.annotations.autoremove": "FALSE",
"io.podman.annotations.init": "FALSE",
"io.podman.annotations.privileged": "FALSE",
"io.podman.annotations.publish-all": "FALSE",
"org.opencontainers.image.base.digest": "sha256:ddee1d549e109453af1627c06074495c7eb21ff123f862f1b91f4019b31d924b",
"org.opencontainers.image.base.name": "",
"org.opencontainers.image.stopSignal": "15"
},
"linux": {
"resources": {
"pids": {
"limit": 2048
}
},
"cgroupsPath": "machine-libpod_pod_ec33dac132fe882d7770691e886cd6960d65270da6b15bc2bca36d438e3b0573.slice:libpod:4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2",
"namespaces": [
{
"type": "pid"
},
{
"type": "network",
"path": "/proc/127514/ns/net"
},
{
"type": "ipc",
"path": "/proc/127514/ns/ipc"
},
{
"type": "uts",
"path": "/proc/127514/ns/uts"
},
{
"type": "mount"
},
{
"type": "cgroup"
}
]
}
}
D0904 22:00:13.610418 127613 container.go:180] Create container, cid: 4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2, rootDir: "/var/run/runsc"
D0904 22:00:13.610698 127613 container.go:289] Creating new container, cid: 4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2, sandbox: 256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6
D0904 22:00:13.610726 127613 state_file.go:57] Load container, rootDir: "/var/run/runsc", id: {SandboxID:256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6 ContainerID:256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6}, opts: {Exact:true SkipCheck:false RootContainer:false}
D0904 22:00:13.617319 127613 container.go:582] Signal container, cid: 256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6, signal: signal 0 (0)
D0904 22:00:13.617362 127613 sandbox.go:1020] Signal sandbox "256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6"
D0904 22:00:13.617382 127613 sandbox.go:522] Connecting to sandbox "256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6"
D0904 22:00:13.617656 127613 urpc.go:568] urpc: successfully marshalled 144 bytes.
D0904 22:00:13.620523 127613 urpc.go:611] urpc: unmarshal success.
D0904 22:00:13.628151 127613 cgroup.go:412] New cgroup for pid: self, *cgroup.cgroupSystemd: &{cgroupV2:{Mountpoint:/sys/fs/cgroup Path:/machine.slice/machine-libpod_pod_ec33dac132fe882d7770691e886cd6960d65270da6b15bc2bca36d438e3b0573.slice/libpod-4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2.scope Controllers:[cpuset cpu io memory hugetlb pids rdma misc] Own:[]} Name:4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2 Parent:machine-libpod_pod_ec33dac132fe882d7770691e886cd6960d65270da6b15bc2bca36d438e3b0573.slice ScopePrefix:libpod properties:[] dbusConn:0xc0004e0200}
D0904 22:00:13.628191 127613 systemd.go:98] Installing systemd cgroup resource controller under machine-libpod_pod_ec33dac132fe882d7770691e886cd6960d65270da6b15bc2bca36d438e3b0573.slice
D0904 22:00:13.628206 127613 sandbox.go:250] Create sub-container "4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2" in sandbox "256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6", PID: 127514
D0904 22:00:13.628219 127613 sandbox.go:522] Connecting to sandbox "256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6"
D0904 22:00:13.628450 127613 urpc.go:568] urpc: successfully marshalled 129 bytes.
D0904 22:00:13.628689 127613 urpc.go:611] urpc: unmarshal success.
D0904 22:00:13.628732 127613 container.go:800] Save container, cid: 4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2
D0904 22:00:13.629195 127613 container.go:729] Destroy container, cid: 4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2
D0904 22:00:13.629245 127613 container.go:814] Destroying container, cid: 4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2
D0904 22:00:13.629260 127613 sandbox.go:1369] Destroying container, cid: 4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2, sandbox: 256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6
D0904 22:00:13.629275 127613 sandbox.go:522] Connecting to sandbox "256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6"
D0904 22:00:13.629456 127613 urpc.go:568] urpc: successfully marshalled 122 bytes.
D0904 22:00:13.629668 127613 urpc.go:611] urpc: unmarshal success.
D0904 22:00:13.629718 127613 cgroup_v2.go:175] Deleting cgroup "/sys/fs/cgroup/machine.slice/machine-libpod_pod_ec33dac132fe882d7770691e886cd6960d65270da6b15bc2bca36d438e3b0573.slice/libpod-4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2.scope"
D0904 22:00:13.630289 127613 state_file.go:57] Load container, rootDir: "/var/run/runsc", id: {SandboxID:256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6 ContainerID:256763062fa628fb8404c097b5d5899805d060f9294d312a792345a867b26ef6}, opts: {Exact:true SkipCheck:true RootContainer:false}
I0904 22:00:13.688109 127620 main.go:213] ***************************
I0904 22:00:13.688188 127620 main.go:214] Args: [/usr/bin/runsc --debug --debug-log=/tmp/trsc-container-t.log delete --force 4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2]
I0904 22:00:13.688273 127620 main.go:215] Version release-20220808.0
I0904 22:00:13.688337 127620 main.go:216] GOOS: linux
I0904 22:00:13.688390 127620 main.go:217] GOARCH: amd64
I0904 22:00:13.688443 127620 main.go:218] PID: 127620
I0904 22:00:13.688497 127620 main.go:219] UID: 0, GID: 0
I0904 22:00:13.688550 127620 main.go:220] Configuration:
I0904 22:00:13.688608 127620 main.go:221] RootDir: /var/run/runsc
I0904 22:00:13.688684 127620 main.go:222] Platform: ptrace
I0904 22:00:13.688738 127620 main.go:223] FileAccess: exclusive, overlay: false
I0904 22:00:13.688799 127620 main.go:224] Network: sandbox, logging: false
I0904 22:00:13.688856 127620 main.go:225] Strace: false, max size: 1024, syscalls:
I0904 22:00:13.688909 127620 main.go:226] LISAFS: false
I0904 22:00:13.689008 127620 main.go:227] Debug: true
I0904 22:00:13.689062 127620 main.go:228] Systemd: false
I0904 22:00:13.689114 127620 main.go:229] ***************************
D0904 22:00:13.689187 127620 state_file.go:57] Load container, rootDir: "/var/run/runsc", id: {SandboxID: ContainerID:4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2}, opts: {Exact:false SkipCheck:false RootContainer:false}
W0904 22:00:13.689331 127620 delete.go:75] couldn't find container "4cbe07b149175cff9d366899f1bd26c78b9a51637a49d2d77f95c66fc0d38cc2": file does not exist
I0904 22:00:13.689363 127620 main.go:245] Exiting with status: 0
/tmp/trsc-pod.log.2:
https://fars.ee/3EoR
@manninglucas: is this addressed with the fixes to the cgroups bug we found?
Yes, should be fixed in release 20220905.0 by 23b21af6d631c6574901628ea12ec1e7f7e2324d. unknowndevQwQ please let us know if the problem persists after upgrading to that release.
Yes, should be fixed in release 20220905.0 by 23b21af. unknowndevQwQ please let us know if the problem persists after upgrading to that release.
Error: failed to connect to container's attach socket: /var/lib/containers/storage/overlay-containers/8f03597426b9d20868e960383094eedd2a6eecd2c1b3e2c25026ab9172a3047e/userdata/attach: no such file or directory
and: https://github.com/containers/podman/issues/15223#issuecomment-1208068034
trsc-container-t.log.txt
trsc-pod.log.2.txt
podman --log-level trace --runtime runsc --runtime-flag debug --runtime-flag debug-log=/tmp/trsc-container-t.log run --name trsc-container-t --pod trsc-pod --replace quay.io/podman/hello:
INFO[0000] podman filtering at log level trace
DEBU[0000] Called run.PersistentPreRunE(podman --log-level trace --runtime runsc --runtime-flag debug --runtime-flag debug-log=/tmp/trsc-container-t.log run --name trsc-container-t --pod trsc-pod --replace quay.io/podman/hello)
TRAC[0000] Reading configuration file "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
TRAC[0000] &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.49.1 Annotations:[] BaseHostsFile: CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[nproc=4194304:4194304] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:journald LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/libpod/events/events.log EventsLogFileMaxSize:1000000 EventsLogger:journald graphRoot:/var/lib/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/lib/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/var/lib/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/libpod VolumePath:/var/lib/containers/storage/volumes VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:10.88.0.0/16 DefaultSubnetPools:[{Base:10.89.0.0/16 Size:24} {Base:10.90.0.0/15 Size:24} {Base:10.92.0.0/14 Size:24} {Base:10.96.0.0/11 Size:24} {Base:10.128.0.0/9 Size:24}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}
TRAC[0000] Reading configuration file "/etc/containers/containers.conf"
DEBU[0000] Merged system config "/etc/containers/containers.conf"
TRAC[0000] &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.49.1 Annotations:[] BaseHostsFile: CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[nproc=4194304:4194304] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:journald LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/libpod/events/events.log EventsLogFileMaxSize:1000000 EventsLogger:journald graphRoot:/var/lib/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/lib/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/var/lib/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/libpod VolumePath:/var/lib/containers/storage/volumes VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:10.88.0.0/16 DefaultSubnetPools:[{Base:10.89.0.0/16 Size:24} {Base:10.90.0.0/15 Size:24} {Base:10.92.0.0/14 Size:24} {Base:10.96.0.0/11 Size:24} {Base:10.128.0.0/9 Size:24}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/containers/storage" with "/var/run/containers/storage" from database
DEBU[0000] Overriding tmp dir "/run/libpod" with "/var/run/libpod" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is being used
DEBU[0000] Cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend journald
TRAC[0000] found runtime "/usr/bin/runsc"
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
TRAC[0000] found runtime "/usr/bin/crun"
TRAC[0000] found runtime "/usr/bin/runc"
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
TRAC[0000] found runtime "/usr/bin/kata-runtime"
DEBU[0000] Using OCI runtime "/usr/bin/runsc"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Pulling image quay.io/podman/hello (policy: missing)
DEBU[0000] Looking up image "quay.io/podman/hello" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Found image "quay.io/podman/hello" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29)
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Looking up image "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29)
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Error looking up container "trsc-container-t": no container with name or ID "trsc-container-t" found: no such container
DEBU[0000] Looking up image "quay.io/podman/hello" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Found image "quay.io/podman/hello" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29)
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Inspecting image f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Inspecting image f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29
DEBU[0000] Inspecting image f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29
DEBU[0000] Inspecting image f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29
DEBU[0000] using systemd mode: false
DEBU[0000] adding container to pod trsc-pod
DEBU[0000] setting container name trsc-container-t
DEBU[0000] Loading seccomp profile from "/etc/containers/seccomp.json"
INFO[0000] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host
DEBU[0000] Allocated lock 4 for container 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are supported
DEBU[0000] Created container "3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16"
DEBU[0000] Container "3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16" has work directory "/var/lib/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata"
DEBU[0000] Container "3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16" has run directory "/var/run/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata"
DEBU[0000] Not attaching to stdin
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is being used
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/O7ADVJEAG4YTOOUP4WMCQXKT5U,upperdir=/var/lib/containers/storage/overlay/8a49fcb0aa466eca5ba8a9fb3a9fa02395afff372c659490854d93bf3b6343da/diff,workdir=/var/lib/containers/storage/overlay/8a49fcb0aa466eca5ba8a9fb3a9fa02395afff372c659490854d93bf3b6343da/work,nodev
DEBU[0000] Mounted container "3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16" at "/var/lib/containers/storage/overlay/8a49fcb0aa466eca5ba8a9fb3a9fa02395afff372c659490854d93bf3b6343da/merged"
DEBU[0000] Created root filesystem for container 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16 at /var/lib/containers/storage/overlay/8a49fcb0aa466eca5ba8a9fb3a9fa02395afff372c659490854d93bf3b6343da/merged
DEBU[0000] Modifying container 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16 /etc/passwd
DEBU[0000] Modifying container 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16 /etc/group
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# Configuration file for default mounts in containers (see man 5"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# containers-mounts.conf for further information)"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: ""
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting Cgroups for container 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16 to machine-libpod_pod_9caddd7ec1d5a4392cc530fe1eb91620062e42420dbfdead59258a75df326a12.slice:libpod:3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/var/lib/containers/storage/overlay/8a49fcb0aa466eca5ba8a9fb3a9fa02395afff372c659490854d93bf3b6343da/merged"
DEBU[0000] Created OCI spec for container 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16 at /var/lib/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata/config.json
DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -c 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16 -u 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16 -r /usr/bin/runsc -b /var/lib/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata -p /var/run/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata/pidfile -n trsc-container-t --exit-dir /var/run/libpod/exits --full-attach --runtime-arg --debug --runtime-arg --debug-log=/tmp/trsc-container-t.log -s -l journald --log-level trace --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg trace --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg runsc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16]"
INFO[0000] Running conmon under slice machine-libpod_pod_9caddd7ec1d5a4392cc530fe1eb91620062e42420dbfdead59258a75df326a12.slice and unitName libpod-conmon-3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16.scope
DEBU[0000] Received: 417712
INFO[0000] Got Conmon PID as 417803
DEBU[0000] Created container 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16 in OCI runtime
DEBU[0000] Attaching to container 3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16
DEBU[0000] ExitCode msg: "failed to connect to container's attach socket: /var/lib/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata/attach: no such file or directory"
Error: failed to connect to container's attach socket: /var/lib/containers/storage/overlay-containers/3160556fd5b609291c2a92a89e8f46d3759473f101d319ba91847e34f7e5cc16/userdata/attach: no such file or directory
Where did runsc put the attach file?
Do you have a new stack trace similar to your original post? There's definitely still an issue but I think it's different from the original problem.
@unknowndevQwQ : are the passed flags also the only flags, or is there a config file as well? If so, what flags are set in the config file?
@unknowndevQwQ : are the passed flags also the only flags, or is there a config file as well? If so, what flags are set in the config file?
Is the flag mentioned here just the flag that podman passes to runsc?
Docker has the /etc/docker/daemon.json file which you can pass flags to the runtime as "runtimeArgs". I'm wondering if there is something similar in podman or if the arguments you pass via --runtime-flag are the only arguments.
We suspect this is a cgroups issue...we've seen this recently moving to cgroupsv2 in other services. There's just a lot to test on different systems and it is hard to setup test envs for each.
What's the output of these commands?
runsc flags 2>&1 | grep cgroup
mount | grep cgroup
Docker has the /etc/docker/daemon.json file which you can pass flags to the runtime as "runtimeArgs". I'm wondering if there is something similar in podman or if the arguments you pass via
--runtime-flagare the only arguments.We suspect this is a cgroups issue...we've seen this recently moving to cgroupsv2 in other services. There's just a lot to test on different systems and it is hard to setup test envs for each.
What's the output of these commands?
runsc flags 2>&1 | grep cgroupmount | grep cgroup
runsc flags 2>&1 | grep cgroup
-cgroupfs
Automatically mount cgroupfs.
-ignore-cgroups
don't configure cgroups.
-systemd-cgroup
EXPERIMENTAL. Use systemd for cgroups.
mount -t cgroup2:
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
I'm wondering if there is something similar in podman or if the arguments you pass via
--runtime-flagare the only arguments.
podman does not yet implement engine.runtime_flags, and yes, --runtime-flag is the only way to pass the runsc argument
We suspect this is a cgroups issue...we've seen this recently moving to cgroupsv2 in other services. There's just a lot to test on different systems and it is hard to setup test envs for each.
This is not a cgroup issue, after I tried adding systemd.unified_cgroup_hierarchy=0 and making sure I was using cgroup v1 the problem persisted
What's the output of these commands?
runsc flags 2>&1 | grep cgroupmount | grep cgroup
$ runsc flags 2>&1 | grep cgroup
-cgroupfs
Automatically mount cgroupfs.
-ignore-cgroups
don't configure cgroups.
-systemd-cgroup
EXPERIMENTAL. Use systemd for cgroups.
$ mount -t cgroup
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
$ mount -t cgroup2
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
$ runsc --version
runsc version release-20220905.0
spec: 1.0.2-dev
/tmp/trsc-pod.log.2: https://fars.ee/o_dx
/tmp/trsc-container-t.log: https://fars.ee/EX5A
podman log:
INFO[0000] podman filtering at log level trace
DEBU[0000] Called run.PersistentPreRunE(podman --log-level trace --runtime runsc --runtime-flag debug --runtime-flag debug-log=/tmp/trsc-container-t.log run --name trsc-container-t --pod trsc-pod --replace quay.io/podman/hello)
TRAC[0000] Reading configuration file "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
TRAC[0000] &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.49.1 Annotations:[] BaseHostsFile: CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[nproc=4194304:4194304] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:journald LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/libpod/events/events.log EventsLogFileMaxSize:1000000 EventsLogger:journald graphRoot:/var/lib/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/lib/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/var/lib/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/libpod VolumePath:/var/lib/containers/storage/volumes VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:10.88.0.0/16 DefaultSubnetPools:[{Base:10.89.0.0/16 Size:24} {Base:10.90.0.0/15 Size:24} {Base:10.92.0.0/14 Size:24} {Base:10.96.0.0/11 Size:24} {Base:10.128.0.0/9 Size:24}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}
TRAC[0000] Reading configuration file "/etc/containers/containers.conf"
DEBU[0000] Merged system config "/etc/containers/containers.conf"
TRAC[0000] &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.49.1 Annotations:[] BaseHostsFile: CgroupNS:host Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[nproc=4194304:4194304] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:false Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:journald LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/libpod/events/events.log EventsLogFileMaxSize:1000000 EventsLogger:journald graphRoot:/var/lib/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/lib/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/var/lib/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/libpod VolumePath:/var/lib/containers/storage/volumes VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:10.88.0.0/16 DefaultSubnetPools:[{Base:10.89.0.0/16 Size:24} {Base:10.90.0.0/15 Size:24} {Base:10.92.0.0/14 Size:24} {Base:10.96.0.0/11 Size:24} {Base:10.128.0.0/9 Size:24}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/containers/storage" with "/var/run/containers/storage" from database
DEBU[0000] Overriding tmp dir "/run/libpod" with "/var/run/libpod" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is being used
DEBU[0000] Cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend journald
TRAC[0000] found runtime "/usr/bin/runsc"
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
TRAC[0000] found runtime "/usr/bin/crun"
TRAC[0000] found runtime "/usr/bin/runc"
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
TRAC[0000] found runtime "/usr/bin/kata-runtime"
DEBU[0000] Using OCI runtime "/usr/bin/runsc"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Pulling image quay.io/podman/hello (policy: missing)
DEBU[0000] Looking up image "quay.io/podman/hello" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Found image "quay.io/podman/hello" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29)
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Looking up image "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello:latest" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29)
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Error looking up container "trsc-container-t": no container with name or ID "trsc-container-t" found: no such container
DEBU[0000] Looking up image "quay.io/podman/hello" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "quay.io/podman/hello:latest" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Found image "quay.io/podman/hello" as "quay.io/podman/hello:latest" in local containers storage
DEBU[0000] Found image "quay.io/podman/hello" as "quay.io/podman/hello:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29)
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Inspecting image f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Inspecting image f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29
DEBU[0000] Inspecting image f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29
DEBU[0000] Inspecting image f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29
DEBU[0000] using systemd mode: false
DEBU[0000] adding container to pod trsc-pod
DEBU[0000] setting container name trsc-container-t
DEBU[0000] Loading seccomp profile from "/etc/containers/seccomp.json"
INFO[0000] Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host
DEBU[0000] Allocated lock 4 for container fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev]@f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] exporting opaque data as blob "sha256:f67d6ab14868b6ad34d7b67c9118506facc3ad0a7a3833e683dc71950534ee29"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are supported
DEBU[0000] Created container "fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702"
DEBU[0000] Container "fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702" has work directory "/var/lib/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata"
DEBU[0000] Container "fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702" has run directory "/var/run/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata"
DEBU[0000] Not attaching to stdin
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is being used
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/O7ADVJEAG4YTOOUP4WMCQXKT5U,upperdir=/var/lib/containers/storage/overlay/49bb45cda53eec6429b99e635d00dd7057f5cf78b0b3087860c6fee7c642eb14/diff,workdir=/var/lib/containers/storage/overlay/49bb45cda53eec6429b99e635d00dd7057f5cf78b0b3087860c6fee7c642eb14/work,nodev
DEBU[0000] Mounted container "fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702" at "/var/lib/containers/storage/overlay/49bb45cda53eec6429b99e635d00dd7057f5cf78b0b3087860c6fee7c642eb14/merged"
DEBU[0000] Created root filesystem for container fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702 at /var/lib/containers/storage/overlay/49bb45cda53eec6429b99e635d00dd7057f5cf78b0b3087860c6fee7c642eb14/merged
DEBU[0000] Modifying container fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702 /etc/passwd
DEBU[0000] Modifying container fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702 /etc/group
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# Configuration file for default mounts in containers (see man 5"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: "# containers-mounts.conf for further information)"
DEBU[0000] Skipping unrecognized mount in /etc/containers/mounts.conf: ""
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting Cgroups for container fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702 to machine-libpod_pod_b67ec6b44c7552cd88eb69d1ca108b18d6dba3d6e632eb057fcbbf70e5cfc094.slice:libpod:fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/var/lib/containers/storage/overlay/49bb45cda53eec6429b99e635d00dd7057f5cf78b0b3087860c6fee7c642eb14/merged"
DEBU[0000] Created OCI spec for container fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702 at /var/lib/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata/config.json
DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -c fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702 -u fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702 -r /usr/bin/runsc -b /var/lib/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata -p /var/run/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata/pidfile -n trsc-container-t --exit-dir /var/run/libpod/exits --full-attach --runtime-arg --debug --runtime-arg --debug-log=/tmp/trsc-container-t.log -s -l journald --log-level trace --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg trace --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg runsc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702]"
INFO[0000] Running conmon under slice machine-libpod_pod_b67ec6b44c7552cd88eb69d1ca108b18d6dba3d6e632eb057fcbbf70e5cfc094.slice and unitName libpod-conmon-fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702.scope
DEBU[0000] Received: 5846
INFO[0000] Got Conmon PID as 5933
DEBU[0000] Created container fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702 in OCI runtime
DEBU[0000] Attaching to container fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702
DEBU[0000] ExitCode msg: "failed to connect to container's attach socket: /var/lib/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata/attach: no such file or directory"
Error: failed to connect to container's attach socket: /var/lib/containers/storage/overlay-containers/fdbff858235cd4242ee576f913f619c9ade6fb81bec1ce8105040dffd8ffc702/userdata/attach: no such file or directory
Run sysdig "(proc.name in (conmon,podman,runsc,runsc-gofer,exe) and evt.type in (open, execve, openat))", never found runsc or runsc-gofer to create the attach file expected by podman
3203155 15:29:28.649712669 3 podman (9043.9045) > openat dirfd=-100(AT_FDCWD) name=/var/lib/containers/storage/libpod/bolt_state.db flags=4103(O_CREAT|O_RDWR|O_CLOEXEC) mode=0600
3203158 15:29:28.649726707 3 podman (9043.9045) < openat fd=7(<f>/var/lib/containers/storage/libpod/bolt_state.db) dirfd=-100(AT_FDCWD) name=/var/lib/containers/storage/libpod/bolt_state.db flags=4103(O_CREAT|O_RDWR|O_CLOEXEC) mode=0600 dev=802
3203361 15:29:28.651246781 3 podman (9043.9052) > openat dirfd=-100(AT_FDCWD) name=/var/lib/containers/storage/overlay-containers/f9bfa6352516263c6d163b19c46b7a55bff2f6b56a2f364769bb8c430e242c34/userdata/attach flags=1(O_RDONLY) mode=0
3203366 15:29:28.651264496 3 podman (9043.9052) < openat fd=-2(ENOENT) dirfd=-100(AT_FDCWD) name=/var/lib/containers/storage/overlay-containers/f9bfa6352516263c6d163b19c46b7a55bff2f6b56a2f364769bb8c430e242c34/userdata/attach flags=1(O_RDONLY) mode=0 dev=0
3203987 15:29:28.655377536 1 podman (9060.9060) > openat dirfd=-100(AT_FDCWD) name=/proc/self/fd flags=13377(O_DIRECTORY|O_NONBLOCK|O_RDONLY|O_CLOEXEC|O_TMPFILE) mode=0
3203992 15:29:28.655398326 1 podman (9060.9060) < openat fd=3(<d>/proc/self/fd) dirfd=-100(AT_FDCWD) name=/proc/self/fd flags=13377(O_DIRECTORY|O_NONBLOCK|O_RDONLY|O_CLOEXEC|O_TMPFILE) mode=0 dev=14
3185896 15:29:28.537401014 2 conmon (9054.9054) > openat dirfd=-100(AT_FDCWD) name=/var/lib/containers/storage/overlay-containers/f9bfa6352516263c6d163b19c46b7a55bff2f6b56a2f364769bb8c430e242c34/userdata flags=1(O_RDONLY) mode=0
3185897 15:29:28.537412275 2 conmon (9054.9054) < openat fd=11(<f>/var/lib/containers/storage/overlay-containers/f9bfa6352516263c6d163b19c46b7a55bff2f6b56a2f364769bb8c430e242c34/userdata) dirfd=-100(AT_FDCWD) name=/var/lib/containers/storage/overlay-containers/f9bfa6352516263c6d163b19c46b7a55bff2f6b56a2f364769bb8c430e242c34/userdata flags=1(O_RDONLY) mode=0 dev=802
3185898 15:29:28.537435641 2 conmon (9054.9054) > socket domain=1(AF_LOCAL) type=526341 proto=0
3185899 15:29:28.537444170 2 conmon (9054.9054) < socket fd=12(<u>)
3185912 15:29:28.537552515 2 conmon (9054.9054) > listen fd=12(<u>/proc/self/fd/11/attach) backlog=10
3185913 15:29:28.537553747 2 conmon (9054.9054) < listen res=0
3185924 15:29:28.537658760 2 conmon (9054.9054) > openat dirfd=-100(AT_FDCWD) name=/var/lib/containers/storage/overlay-containers/f9bfa6352516263c6d163b19c46b7a55bff2f6b56a2f364769bb8c430e242c34/userdata/ctl flags=4161(O_NONBLOCK|O_RDONLY|O_CLOEXEC) mode=0
3185925 15:29:28.537673607 2 conmon (9054.9054) < openat fd=11(<f>/var/lib/containers/storage/overlay-containers/f9bfa6352516263c6d163b19c46b7a55bff2f6b56a2f364769bb8c430e242c34/userdata/ctl) dirfd=-100(AT_FDCWD) name=/var/lib/containers/storage/overlay-containers/f9bfa6352516263c6d163b19c46b7a55bff2f6b56a2f364769bb8c430e242c34/userdata/ctl flags=4161(O_NONBLOCK|O_RDONLY|O_CLOEXEC) mode=0 dev=802
I don't understand why this is @rhatdan
@unknowndevQwQ: I'm sorry to have missed this. Two months missed is pretty bad...sorry.
I don't know what's going on, but podman just hasn't been on our radar.
There is a bug w/ some podman info: https://github.com/google/gvisor/issues/311, more specifically this comment: https://github.com/google/gvisor/issues/311#issuecomment-1121668954
If not, LMK and I'll see if I can carve out some time to play with podman.
Flagging @manninglucas
@unknowndevQwQ: I'm sorry to have missed this. Two months missed is pretty bad...sorry.
I don't know what's going on, but podman just hasn't been on our radar.
There is a bug w/ some podman info: #311, more specifically this comment: #311 (comment)
If not, LMK and I'll see if I can carve out some time to play with podman.
Flagging @manninglucas
The current issue is not a rootless container and should not be related to #311
A friendly reminder that this issue had no activity for 120 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
$ runsc --version
runsc version release-20231204.0
spec: 1.1.0-rc.1
$ podman version
Client: Podman Engine
Version: 4.8.1
API Version: 4.8.1
Go Version: go1.21.4
Git Commit: ef6e5ac40302770027e7598ee09d177454adcd2f-dirty
Built: Wed Dec 6 01:09:26 2023
OS/Arch: linux/amd64
Works fine in this version, Close