gvisor Pods stuck in Terminating state due to process not being killed

Pods stuck in Terminating state due to process not being killed

Open markusthoemmes opened this issue 1 year ago • 54 comments

Description

We're seeing pods getting stuck in a Terminating state permanently. Any commands done via crictl or ctr fail in the same way as the containerd and kubelet logs suggest (see below). It's only if we manually kill the sandbox PID (as surfaced by crictl inspect and crictl inspecp) that the deletion process unwedges. I think what's happening is:

The deletion command is sent
We try to kill the containers. For some reason the signal is not received, thus the processes are not shutting down. But apparently either containerd or runsc don't doublecheck that.
Subsequently trying to kill the container via containerd or crictl results in the "resource is busy" error, because correctly, the rootfs is still in use by the container's processes. (I've proven this by doing an lsof onto the rootfs and following the hierarchy of the using PID back up to 1101026, the PID of the container/sandbox)
The former process also deleted the runsc state file of the container (not the surrounding pod) and thus we see the "file does not exist" errors subsequently.

I'm a bit at a loss for further debugging to be honest and I don't know if gvisor or containerd is at fault here either. It's happening in a live system fairly regularly but we don't have a reproducer yet either. I'd appreciate any thoughts for further information gathering to pinpoint exactly what is going on.

Pod state

  containerStatuses:
  - containerID: containerd://4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31
    lastState: {}
    ready: false
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-12-14T01:20:30Z"

container state in crictl

Note that CONTAINER_RUNNING here is inconsistent with STOPPED in containerd state.

$ crictl inspect 4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31
...
    "state": "CONTAINER_RUNNING",
    "createdAt": "2023-12-14T01:20:30.455515622Z",
    "startedAt": "2023-12-14T01:20:30.79432126Z",
    "finishedAt": "0001-01-01T00:00:00Z",
...
    "sandboxID": "8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2",
    "pid": 1101026,
    "removing": false,
    "snapshotKey": "4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31",
    "snapshotter": "overlayfs",
    "runtimeType": "io.containerd.runsc.v1",
    "runtimeOptions": {
      "type_url": "io.containerd.runsc.v1.options",
      "config_path": "/etc/containerd/runsc-systrap-nodirectfs.toml"
    },
...

pod state in crictl

$ crictl inspectp 8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2
...
    "state": "SANDBOX_READY",
    "createdAt": "2023-12-14T01:20:28.615187802Z",
...
    "pid": 1101026,
    "processStatus": "running",
    "netNamespaceClosed": false,
    "image": "registry.k8s.io/pause:3.8",
    "snapshotKey": "8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2",
    "snapshotter": "overlayfs",
    "runtimeHandler": "runsc-systrap-nodirectfs",
    "runtimeType": "io.containerd.runsc.v1",
    "runtimeOptions": {
      "type_url": "io.containerd.runsc.v1.options",
      "config_path": "/etc/containerd/runsc-systrap-nodirectfs.toml"
    },
...

container state in ctr

$ ctr -n k8s.io t ls | grep 4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31
4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31    1101026    STOPPED

pod state in ctr

$ ctr -n k8s.io t ls | grep 8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2
8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2    1101026    RUNNING

Logs grepping for the pod

$ journalctl -u containerd | grep 8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2
Dec 14 01:20:28 fancy-machine containerd[601]: time="2023-12-14T01:20:28.884078779Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2 pid=1101010
Dec 14 01:20:29 fancy-machine containerd[601]: time="2023-12-14T01:20:29.234943041Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:foo-docker-5479647b89-xvgmx,Uid:0529db22-4a0b-467d-b53e-7cccbec7aab1,Namespace:app-c3640d47-fa78-412d-9416-8be5c2f56b63,Attempt:0,} returns sandbox id \"8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2\""
Dec 14 01:20:30 fancy-machine containerd[601]: time="2023-12-14T01:20:30.404825455Z" level=info msg="CreateContainer within sandbox \"8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2\" for container &ContainerMetadata{Name:foo-docker,Attempt:0,}"
Dec 14 01:20:30 fancy-machine containerd[601]: time="2023-12-14T01:20:30.456349292Z" level=info msg="CreateContainer within sandbox \"8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2\" for &ContainerMetadata{Name:foo-docker,Attempt:0,} returns container id \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\""
Dec 14 02:23:46 fancy-machine containerd[601]: time="2023-12-14T02:23:46.134770468Z" level=info msg="StopPodSandbox for \"8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2\""
Dec 14 02:38:46 fancy-machine containerd[601]: time="2023-12-14T02:38:46.135031132Z" level=error msg="StopPodSandbox for \"8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2\" failed" error="rpc error: code = DeadlineExceeded desc = failed to stop container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\": an error occurs during waiting for container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" to be killed: wait container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\": context deadline exceeded"
Dec 14 02:54:16 fancy-machine containerd[601]: time="2023-12-14T02:54:16.347677617Z" level=info msg="StopPodSandbox for \"8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2\""
Dec 14 03:09:16 fancy-machine containerd[601]: time="2023-12-14T03:09:16.348232807Z" level=error msg="StopPodSandbox for \"8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2\" failed" error="rpc error: code = Canceled desc = failed to stop container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\": an error occurs during waiting for container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" to be killed: wait container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\": context canceled"

Logs grepping for the container

$ journalctl -u containerd | grep 4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31
Dec 14 01:20:30 fancy-machine containerd[601]: time="2023-12-14T01:20:30.456349292Z" level=info msg="CreateContainer within sandbox \"8dafd41d1e71a539c2f537cb5412be73cece506ee6766ed9a2ff68566fac8dd2\" for &ContainerMetadata{Name:foo-docker,Attempt:0,} returns container id \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\""
Dec 14 01:20:30 fancy-machine containerd[601]: time="2023-12-14T01:20:30.456934572Z" level=info msg="StartContainer for \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\""
Dec 14 01:20:30 fancy-machine containerd[601]: time="2023-12-14T01:20:30.523911876Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31 pid=1101107
Dec 14 01:20:30 fancy-machine containerd[601]: time="2023-12-14T01:20:30.795747481Z" level=info msg="StartContainer for \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" returns successfully"
Dec 14 02:08:16 fancy-machine containerd[601]: time="2023-12-14T02:08:16.134427206Z" level=info msg="StopContainer for \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" with timeout 30 (s)"
Dec 14 02:08:16 fancy-machine containerd[601]: time="2023-12-14T02:08:16.164121746Z" level=info msg="Stop container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" with signal terminated"
Dec 14 02:08:18 fancy-machine containerd[601]: time="2023-12-14T02:08:18.966779349Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 14 02:08:18 fancy-machine containerd[601]: time="2023-12-14T02:08:18.968217399Z" level=error msg="failed to handle container TaskExit event container_id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" pid:1101026 exit_status:143 exited_at:{seconds:1702519696 nanos:298254231}" error="failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31/rootfs: device or resource busy: unknown"
Dec 14 02:08:19 fancy-machine containerd[601]: time="2023-12-14T02:08:19.453267097Z" level=error msg="collecting metrics for 4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31" error="cannot stat a stopped container: unknown"
Dec 14 02:08:20 fancy-machine containerd[601]: time="2023-12-14T02:08:20.631148165Z" level=info msg="TaskExit event container_id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" pid:1101026 exit_status:143 exited_at:{seconds:1702519696 nanos:298254231}"
Dec 14 02:08:20 fancy-machine containerd[601]: time="2023-12-14T02:08:20.674609167Z" level=warning msg="Ignoring error killing container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 14 02:08:23 fancy-machine containerd[601]: time="2023-12-14T02:08:23.206429065Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 14 02:08:23 fancy-machine containerd[601]: time="2023-12-14T02:08:23.206693383Z" level=error msg="Failed to handle backOff event container_id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" pid:1101026 exit_status:143 exited_at:{seconds:1702519696 nanos:298254231} for 4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31/rootfs: device or resource busy: unknown"
Dec 14 02:08:25 fancy-machine containerd[601]: time="2023-12-14T02:08:25.630658794Z" level=info msg="TaskExit event container_id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" pid:1101026 exit_status:143 exited_at:{seconds:1702519696 nanos:298254231}"
Dec 14 02:08:25 fancy-machine containerd[601]: time="2023-12-14T02:08:25.661725410Z" level=warning msg="Ignoring error killing container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 14 02:08:28 fancy-machine containerd[601]: time="2023-12-14T02:08:28.204892129Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 14 02:08:28 fancy-machine containerd[601]: time="2023-12-14T02:08:28.205118099Z" level=error msg="Failed to handle backOff event container_id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" pid:1101026 exit_status:143 exited_at:{seconds:1702519696 nanos:298254231} for 4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31/rootfs: device or resource busy: unknown"
Dec 14 02:08:29 fancy-machine containerd[601]: time="2023-12-14T02:08:29.861634909Z" level=error msg="collecting metrics for 4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31" error="cannot stat a stopped container: unknown"
Dec 14 02:08:32 fancy-machine containerd[601]: time="2023-12-14T02:08:32.630224824Z" level=info msg="TaskExit event container_id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" id:\"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\" pid:1101026 exit_status:143 exited_at:{seconds:1702519696 nanos:298254231}"
Dec 14 02:08:32 fancy-machine containerd[601]: time="2023-12-14T02:08:32.666238678Z" level=warning msg="Ignoring error killing container \"4ae6bba644657639c2ebf3776dfbd4d1c4ebcfbb810543caddb14f8de83b2b31\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"

Steps to reproduce

None yet, sadly.

runsc version

runsc version release-20231113.0-41-g9c6f50d59e02
spec: 1.1.0-rc.1

[runsc_config]
systemd-cgroup = "true"
platform = "systrap"
directfs = "false"

containerd:
Client:
  Version:  v1.7.8
  Revision: 8e4b0bde866788eec76735cc77c4720144248fb7
  Go version: go1.20.10

Server:
  Version:  v1.7.8
  Revision: 8e4b0bde866788eec76735cc77c4720144248fb7
  UUID: 5b676e4a-6f9e-4208-871e-e11edad4c8b8

docker version (if using docker)

No response

uname

6.1.0-12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux

kubectl (if using Kubernetes)

v1.28.2

repo state (if built from source)

No response

runsc debug logs (if available)

No response

Dec 21 '23 13:12 markusthoemmes

From the logs and following a bit more of what the shim does etc, it looks like the shim thought it had actually killed the correct PID and it even prints an exit code of 143, which would've been a SIGTERM exit. Buuuut, the respective process was definitely still running. I'm puzzled.

In another instance of the same issue, containerd did not print an exit code, but again the correct PID and a exited_at timestamp.

Dec 22 '23 09:12 markusthoemmes

Here some more information. This time, we didn't get the umount issues (which I believe are only a followup problem on the "kill not being effective" anyway) but I got the chance to manually run runsc kill with debug logging enabled. It doesn't really tell me anything useful though. In this case, containerd did not think that the process was stopped. Both runsc kill (first to the container, then to the sandbox directly) did nothing to actually stop either container or sandbox. It looked like the runsc-gofer went away though.

$ journalctl -u containerd | grep ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478
Dec 20 11:32:21 fancy-machine containerd[599]: time="2023-12-20T11:32:21.640759516Z" level=info msg="CreateContainer within sandbox \"52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707\" for &ContainerMetadata{Name:service,Attempt:0,} returns container id \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\""
Dec 20 11:32:21 fancy-machine containerd[599]: time="2023-12-20T11:32:21.641417411Z" level=info msg="StartContainer for \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\""
Dec 20 11:32:21 fancy-machine containerd[599]: time="2023-12-20T11:32:21.736554591Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478 pid=884345
Dec 20 11:32:22 fancy-machine containerd[599]: time="2023-12-20T11:32:22.050765688Z" level=info msg="StartContainer for \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" returns successfully"
Dec 22 09:20:41 fancy-machine containerd[599]: time="2023-12-22T09:20:41.057453474Z" level=info msg="StopContainer for \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" with timeout 30 (s)"
Dec 22 09:20:41 fancy-machine containerd[599]: time="2023-12-22T09:20:41.102373039Z" level=info msg="Stop container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" with signal terminated"
Dec 22 09:21:11 fancy-machine containerd[599]: time="2023-12-22T09:21:11.182849104Z" level=info msg="Kill container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\""
Dec 22 09:36:11 fancy-machine containerd[599]: time="2023-12-22T09:36:11.057405548Z" level=error msg="StopContainer for \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" to be killed: wait container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\": context canceled"
Dec 22 09:36:11 fancy-machine containerd[599]: time="2023-12-22T09:36:11.204889744Z" level=info msg="Kill container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\""
Dec 22 09:51:11 fancy-machine containerd[599]: time="2023-12-22T09:51:11.057962413Z" level=error msg="StopPodSandbox for \"52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707\" failed" error="rpc error: code = Canceled desc = failed to stop container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\": an error occurs during waiting for container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" to be killed: wait container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\": context canceled"
Dec 22 09:51:11 fancy-machine containerd[599]: time="2023-12-22T09:51:11.739212354Z" level=info msg="StopContainer for \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" with timeout 30 (s)"
Dec 22 09:51:11 fancy-machine containerd[599]: time="2023-12-22T09:51:11.794469456Z" level=info msg="Skipping the sending of signal terminated to container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" because a prior stop with timeout>0 request already sent the signal"
Dec 22 09:51:41 fancy-machine containerd[599]: time="2023-12-22T09:51:41.795213941Z" level=info msg="Kill container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\""
Dec 22 10:06:41 fancy-machine containerd[599]: time="2023-12-22T10:06:41.739067786Z" level=error msg="StopContainer for \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" to be killed: wait container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\": context canceled"
Dec 22 10:06:41 fancy-machine containerd[599]: time="2023-12-22T10:06:41.792177189Z" level=info msg="Kill container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\""

$ journalctl -u containerd | grep 52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707
Dec 20 11:32:21 fancy-machine containerd[599]: time="2023-12-20T11:32:21.105809959Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707 pid=884178
Dec 20 11:32:21 fancy-machine containerd[599]: time="2023-12-20T11:32:21.555317685Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:service-c99ddb4d9-nsjt6,Uid:246d774a-ddc2-42db-9243-399e6089a049,Namespace:app-b077b29d-1ed1-4cd4-a85a-50dda755e4b1,Attempt:0,} returns sandbox id \"52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707\""
Dec 20 11:32:21 fancy-machine containerd[599]: time="2023-12-20T11:32:21.568671853Z" level=info msg="CreateContainer within sandbox \"52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707\" for container &ContainerMetadata{Name:service,Attempt:0,}"
Dec 20 11:32:21 fancy-machine containerd[599]: time="2023-12-20T11:32:21.640759516Z" level=info msg="CreateContainer within sandbox \"52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707\" for &ContainerMetadata{Name:service,Attempt:0,} returns container id \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\""
Dec 22 09:36:11 fancy-machine containerd[599]: time="2023-12-22T09:36:11.057573038Z" level=info msg="StopPodSandbox for \"52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707\""
Dec 22 09:51:11 fancy-machine containerd[599]: time="2023-12-22T09:51:11.057962413Z" level=error msg="StopPodSandbox for \"52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707\" failed" error="rpc error: code = Canceled desc = failed to stop container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\": an error occurs during waiting for container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\" to be killed: wait container \"ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478\": context canceled"
Dec 22 10:06:41 fancy-machine containerd[599]: time="2023-12-22T10:06:41.739454662Z" level=info msg="StopPodSandbox for \"52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707\""

I1222 10:05:24.744173  2930321 main.go:189] ***************************
I1222 10:05:24.744217  2930321 main.go:190] Args: [runsc --debug --debug-log=./ --root=/run/containerd/runsc/k8s.io kill ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478]
I1222 10:05:24.744230  2930321 main.go:191] Version release-20231113.0-41-g9c6f50d59e02
I1222 10:05:24.744236  2930321 main.go:192] GOOS: linux
I1222 10:05:24.744241  2930321 main.go:193] GOARCH: amd64
I1222 10:05:24.744247  2930321 main.go:194] PID: 2930321
I1222 10:05:24.744253  2930321 main.go:195] UID: 0, GID: 0
I1222 10:05:24.744258  2930321 main.go:196] Configuration:
I1222 10:05:24.744264  2930321 main.go:197] 		RootDir: /run/containerd/runsc/k8s.io
I1222 10:05:24.744269  2930321 main.go:198] 		Platform: systrap
I1222 10:05:24.744274  2930321 main.go:199] 		FileAccess: exclusive
I1222 10:05:24.744281  2930321 main.go:200] 		Directfs: true
I1222 10:05:24.744286  2930321 main.go:201] 		Overlay: root:self
I1222 10:05:24.744293  2930321 main.go:202] 		Network: sandbox, logging: false
I1222 10:05:24.744306  2930321 main.go:203] 		Strace: false, max size: 1024, syscalls:
I1222 10:05:24.744318  2930321 main.go:204] 		IOURING: false
I1222 10:05:24.744328  2930321 main.go:205] 		Debug: true
I1222 10:05:24.744336  2930321 main.go:206] 		Systemd: false
I1222 10:05:24.744341  2930321 main.go:207] ***************************
D1222 10:05:24.744370  2930321 state_file.go:78] Load container, rootDir: "/run/containerd/runsc/k8s.io", id: {SandboxID: ContainerID:ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478}, opts: {Exact:false SkipCheck:false TryLock:false RootContainer:false}
D1222 10:05:24.746033  2930321 container.go:673] Signal container, cid: ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478, signal: signal 0 (0)
D1222 10:05:24.746061  2930321 sandbox.go:1211] Signal sandbox "52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707"
D1222 10:05:24.746068  2930321 sandbox.go:613] Connecting to sandbox "52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707"
D1222 10:05:24.746161  2930321 urpc.go:568] urpc: successfully marshalled 144 bytes.
D1222 10:05:24.801110  2930321 urpc.go:611] urpc: unmarshal success.
D1222 10:05:24.801174  2930321 container.go:673] Signal container, cid: ae3a2430d446f8ca4bdaee18e4a29d6bddaff5687a34056e4584ffda62378478, signal: terminated (15)
D1222 10:05:24.801194  2930321 sandbox.go:1211] Signal sandbox "52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707"
D1222 10:05:24.801204  2930321 sandbox.go:613] Connecting to sandbox "52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707"
D1222 10:05:24.801266  2930321 urpc.go:568] urpc: successfully marshalled 145 bytes.
D1222 10:05:24.802085  2930321 urpc.go:611] urpc: unmarshal success.
I1222 10:05:24.802117  2930321 main.go:224] Exiting with status: 0

I1222 10:11:04.736228  2934612 main.go:189] ***************************
I1222 10:11:04.736301  2934612 main.go:190] Args: [runsc --debug --debug-log=./ --root=/run/containerd/runsc/k8s.io kill 52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707]
I1222 10:11:04.736318  2934612 main.go:191] Version release-20231113.0-41-g9c6f50d59e02
I1222 10:11:04.736326  2934612 main.go:192] GOOS: linux
I1222 10:11:04.736333  2934612 main.go:193] GOARCH: amd64
I1222 10:11:04.736341  2934612 main.go:194] PID: 2934612
I1222 10:11:04.736350  2934612 main.go:195] UID: 0, GID: 0
I1222 10:11:04.736359  2934612 main.go:196] Configuration:
I1222 10:11:04.736366  2934612 main.go:197] 		RootDir: /run/containerd/runsc/k8s.io
I1222 10:11:04.736374  2934612 main.go:198] 		Platform: systrap
I1222 10:11:04.736382  2934612 main.go:199] 		FileAccess: exclusive
I1222 10:11:04.736391  2934612 main.go:200] 		Directfs: true
I1222 10:11:04.736399  2934612 main.go:201] 		Overlay: root:self
I1222 10:11:04.736409  2934612 main.go:202] 		Network: sandbox, logging: false
I1222 10:11:04.736426  2934612 main.go:203] 		Strace: false, max size: 1024, syscalls:
I1222 10:11:04.736436  2934612 main.go:204] 		IOURING: false
I1222 10:11:04.736444  2934612 main.go:205] 		Debug: true
I1222 10:11:04.736452  2934612 main.go:206] 		Systemd: false
I1222 10:11:04.736460  2934612 main.go:207] ***************************
D1222 10:11:04.736503  2934612 state_file.go:78] Load container, rootDir: "/run/containerd/runsc/k8s.io", id: {SandboxID: ContainerID:52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707}, opts: {Exact:false SkipCheck:false TryLock:false RootContainer:false}
D1222 10:11:04.738498  2934612 container.go:673] Signal container, cid: 52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707, signal: signal 0 (0)
D1222 10:11:04.738533  2934612 sandbox.go:1211] Signal sandbox "52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707"
D1222 10:11:04.738544  2934612 sandbox.go:613] Connecting to sandbox "52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707"
D1222 10:11:04.738647  2934612 urpc.go:568] urpc: successfully marshalled 144 bytes.
D1222 10:11:04.797023  2934612 urpc.go:611] urpc: unmarshal success.
D1222 10:11:04.797094  2934612 container.go:673] Signal container, cid: 52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707, signal: terminated (15)
D1222 10:11:04.797115  2934612 sandbox.go:1211] Signal sandbox "52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707"
D1222 10:11:04.797126  2934612 sandbox.go:613] Connecting to sandbox "52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707"
D1222 10:11:04.797191  2934612 urpc.go:568] urpc: successfully marshalled 145 bytes.
D1222 10:11:04.874234  2934612 urpc.go:611] urpc: unmarshal success.
I1222 10:11:04.874304  2934612 main.go:224] Exiting with status: 0

Dec 22 '23 10:12 markusthoemmes

Just seen another occurrence of containerd talking about an exit code but the respective process still be running.

Dec 13 18:58:56 fancy-machine containerd[602]: time="2023-12-13T18:58:56.486265600Z" level=info msg="CreateContainer within sandbox \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" for &ContainerMetadata{Name:blog-os,Attempt:0,} returns container id \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 13 18:58:56 fancy-machine containerd[602]: time="2023-12-13T18:58:56.487074018Z" level=info msg="StartContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 13 18:58:56 fancy-machine containerd[602]: time="2023-12-13T18:58:56.572780483Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a pid=29280
Dec 13 18:58:56 fancy-machine containerd[602]: time="2023-12-13T18:58:56.870377321Z" level=info msg="StartContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" returns successfully"
Dec 27 10:43:55 fancy-machine containerd[602]: time="2023-12-27T10:43:55.203486189Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 10:43:55 fancy-machine containerd[602]: time="2023-12-27T10:43:55.268364859Z" level=info msg="Stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with signal terminated"
Dec 27 10:43:58 fancy-machine containerd[602]: time="2023-12-27T10:43:58.132246202Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:43:58 fancy-machine containerd[602]: time="2023-12-27T10:43:58.132643821Z" level=error msg="failed to handle container TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}" error="failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:43:59 fancy-machine containerd[602]: time="2023-12-27T10:43:59.320747851Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:43:59 fancy-machine containerd[602]: time="2023-12-27T10:43:59.360060515Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:44:01 fancy-machine containerd[602]: time="2023-12-27T10:44:01.920681887Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:44:01 fancy-machine containerd[602]: time="2023-12-27T10:44:01.921129885Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:44:04 fancy-machine containerd[602]: time="2023-12-27T10:44:04.320689694Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:44:04 fancy-machine containerd[602]: time="2023-12-27T10:44:04.354919559Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:44:06 fancy-machine containerd[602]: time="2023-12-27T10:44:06.908520449Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:44:06 fancy-machine containerd[602]: time="2023-12-27T10:44:06.908862175Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:44:11 fancy-machine containerd[602]: time="2023-12-27T10:44:11.321023051Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:44:11 fancy-machine containerd[602]: time="2023-12-27T10:44:11.366364251Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:44:13 fancy-machine containerd[602]: time="2023-12-27T10:44:13.900816800Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:44:13 fancy-machine containerd[602]: time="2023-12-27T10:44:13.901147811Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:44:22 fancy-machine containerd[602]: time="2023-12-27T10:44:22.320618783Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:44:22 fancy-machine containerd[602]: time="2023-12-27T10:44:22.350471321Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:44:24 fancy-machine containerd[602]: time="2023-12-27T10:44:24.890361031Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:44:24 fancy-machine containerd[602]: time="2023-12-27T10:44:24.890671507Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:44:25 fancy-machine containerd[602]: time="2023-12-27T10:44:25.359731943Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 10:44:41 fancy-machine containerd[602]: time="2023-12-27T10:44:41.320578265Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:44:41 fancy-machine containerd[602]: time="2023-12-27T10:44:41.367188947Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:44:43 fancy-machine containerd[602]: time="2023-12-27T10:44:43.916552913Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:44:43 fancy-machine containerd[602]: time="2023-12-27T10:44:43.916868097Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:45:16 fancy-machine containerd[602]: time="2023-12-27T10:45:16.321313852Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:45:16 fancy-machine containerd[602]: time="2023-12-27T10:45:16.351156900Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:45:18 fancy-machine containerd[602]: time="2023-12-27T10:45:18.893620090Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:45:18 fancy-machine containerd[602]: time="2023-12-27T10:45:18.894006674Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:46:23 fancy-machine containerd[602]: time="2023-12-27T10:46:23.321499647Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:46:23 fancy-machine containerd[602]: time="2023-12-27T10:46:23.350618908Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:46:25 fancy-machine containerd[602]: time="2023-12-27T10:46:25.897051670Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:46:25 fancy-machine containerd[602]: time="2023-12-27T10:46:25.897450120Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:48:34 fancy-machine containerd[602]: time="2023-12-27T10:48:34.320550779Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:48:34 fancy-machine containerd[602]: time="2023-12-27T10:48:34.359004059Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:48:36 fancy-machine containerd[602]: time="2023-12-27T10:48:36.909530215Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:48:36 fancy-machine containerd[602]: time="2023-12-27T10:48:36.910409621Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:52:53 fancy-machine containerd[602]: time="2023-12-27T10:52:53.320863379Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:52:53 fancy-machine containerd[602]: time="2023-12-27T10:52:53.351557628Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:52:55 fancy-machine containerd[602]: time="2023-12-27T10:52:55.900714720Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:52:55 fancy-machine containerd[602]: time="2023-12-27T10:52:55.900986827Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:57:56 fancy-machine containerd[602]: time="2023-12-27T10:57:56.321356842Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 10:57:56 fancy-machine containerd[602]: time="2023-12-27T10:57:56.355600076Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 10:57:58 fancy-machine containerd[602]: time="2023-12-27T10:57:58.902560761Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 10:57:58 fancy-machine containerd[602]: time="2023-12-27T10:57:58.903009915Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 10:59:25 fancy-machine containerd[602]: time="2023-12-27T10:59:25.205816983Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 10:59:25 fancy-machine containerd[602]: time="2023-12-27T10:59:25.206394113Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 11:02:59 fancy-machine containerd[602]: time="2023-12-27T11:02:59.321210885Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:02:59 fancy-machine containerd[602]: time="2023-12-27T11:02:59.346403108Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:03:01 fancy-machine containerd[602]: time="2023-12-27T11:03:01.882268105Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:03:01 fancy-machine containerd[602]: time="2023-12-27T11:03:01.882607608Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:08:02 fancy-machine containerd[602]: time="2023-12-27T11:08:02.320580479Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:08:02 fancy-machine containerd[602]: time="2023-12-27T11:08:02.347364547Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:08:04 fancy-machine containerd[602]: time="2023-12-27T11:08:04.897192996Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:08:04 fancy-machine containerd[602]: time="2023-12-27T11:08:04.897559001Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:13:05 fancy-machine containerd[602]: time="2023-12-27T11:13:05.320773336Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:13:05 fancy-machine containerd[602]: time="2023-12-27T11:13:05.362421076Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:13:07 fancy-machine containerd[602]: time="2023-12-27T11:13:07.916675967Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:13:07 fancy-machine containerd[602]: time="2023-12-27T11:13:07.917434204Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:14:25 fancy-machine containerd[602]: time="2023-12-27T11:14:25.205525089Z" level=error msg="StopPodSandbox for \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" failed" error="rpc error: code = Canceled desc = failed to stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 11:14:25 fancy-machine containerd[602]: time="2023-12-27T11:14:25.259748851Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 11:14:25 fancy-machine containerd[602]: time="2023-12-27T11:14:25.260302697Z" level=info msg="Skipping the sending of signal terminated to container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" because a prior stop with timeout>0 request already sent the signal"
Dec 27 11:14:55 fancy-machine containerd[602]: time="2023-12-27T11:14:55.260464684Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 11:18:08 fancy-machine containerd[602]: time="2023-12-27T11:18:08.321553442Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:18:08 fancy-machine containerd[602]: time="2023-12-27T11:18:08.346909104Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:18:10 fancy-machine containerd[602]: time="2023-12-27T11:18:10.885180857Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:18:10 fancy-machine containerd[602]: time="2023-12-27T11:18:10.885661545Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:23:11 fancy-machine containerd[602]: time="2023-12-27T11:23:11.320754927Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:23:11 fancy-machine containerd[602]: time="2023-12-27T11:23:11.370576174Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:23:13 fancy-machine containerd[602]: time="2023-12-27T11:23:13.900365136Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:23:13 fancy-machine containerd[602]: time="2023-12-27T11:23:13.900645453Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:28:14 fancy-machine containerd[602]: time="2023-12-27T11:28:14.321353075Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:28:14 fancy-machine containerd[602]: time="2023-12-27T11:28:14.372025568Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:28:16 fancy-machine containerd[602]: time="2023-12-27T11:28:16.916040260Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:28:16 fancy-machine containerd[602]: time="2023-12-27T11:28:16.916423825Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:29:55 fancy-machine containerd[602]: time="2023-12-27T11:29:55.260051084Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 11:29:55 fancy-machine containerd[602]: time="2023-12-27T11:29:55.261764027Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 11:33:17 fancy-machine containerd[602]: time="2023-12-27T11:33:17.320751273Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:33:17 fancy-machine containerd[602]: time="2023-12-27T11:33:17.354596559Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:33:19 fancy-machine containerd[602]: time="2023-12-27T11:33:19.892244789Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:33:19 fancy-machine containerd[602]: time="2023-12-27T11:33:19.892550480Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:38:20 fancy-machine containerd[602]: time="2023-12-27T11:38:20.321402248Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:38:20 fancy-machine containerd[602]: time="2023-12-27T11:38:20.371695346Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:38:22 fancy-machine containerd[602]: time="2023-12-27T11:38:22.928460729Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:38:22 fancy-machine containerd[602]: time="2023-12-27T11:38:22.928855511Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:43:23 fancy-machine containerd[602]: time="2023-12-27T11:43:23.320694122Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:43:23 fancy-machine containerd[602]: time="2023-12-27T11:43:23.358964313Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:43:25 fancy-machine containerd[602]: time="2023-12-27T11:43:25.928893030Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:43:25 fancy-machine containerd[602]: time="2023-12-27T11:43:25.929203012Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:44:55 fancy-machine containerd[602]: time="2023-12-27T11:44:55.265958770Z" level=error msg="StopPodSandbox for \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" failed" error="rpc error: code = Canceled desc = failed to stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 11:44:56 fancy-machine containerd[602]: time="2023-12-27T11:44:56.072959725Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 11:44:56 fancy-machine containerd[602]: time="2023-12-27T11:44:56.073481240Z" level=info msg="Skipping the sending of signal terminated to container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" because a prior stop with timeout>0 request already sent the signal"
Dec 27 11:45:26 fancy-machine containerd[602]: time="2023-12-27T11:45:26.074257606Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 11:48:26 fancy-machine containerd[602]: time="2023-12-27T11:48:26.320886179Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:48:26 fancy-machine containerd[602]: time="2023-12-27T11:48:26.354526275Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:48:28 fancy-machine containerd[602]: time="2023-12-27T11:48:28.900804992Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:48:28 fancy-machine containerd[602]: time="2023-12-27T11:48:28.901232256Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:53:29 fancy-machine containerd[602]: time="2023-12-27T11:53:29.321066456Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:53:29 fancy-machine containerd[602]: time="2023-12-27T11:53:29.359382279Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:53:31 fancy-machine containerd[602]: time="2023-12-27T11:53:31.917832563Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:53:31 fancy-machine containerd[602]: time="2023-12-27T11:53:31.918244298Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 11:58:32 fancy-machine containerd[602]: time="2023-12-27T11:58:32.321443987Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 11:58:32 fancy-machine containerd[602]: time="2023-12-27T11:58:32.370647507Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 11:58:34 fancy-machine containerd[602]: time="2023-12-27T11:58:34.936038349Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 11:58:34 fancy-machine containerd[602]: time="2023-12-27T11:58:34.936321926Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:00:26 fancy-machine containerd[602]: time="2023-12-27T12:00:26.073355453Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = DeadlineExceeded desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context deadline exceeded"
Dec 27 12:00:26 fancy-machine containerd[602]: time="2023-12-27T12:00:26.074777747Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 12:03:35 fancy-machine containerd[602]: time="2023-12-27T12:03:35.321711497Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:03:35 fancy-machine containerd[602]: time="2023-12-27T12:03:35.367398394Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:03:37 fancy-machine containerd[602]: time="2023-12-27T12:03:37.908786189Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:03:37 fancy-machine containerd[602]: time="2023-12-27T12:03:37.909169990Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:08:38 fancy-machine containerd[602]: time="2023-12-27T12:08:38.321919815Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:08:38 fancy-machine containerd[602]: time="2023-12-27T12:08:38.375252620Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:08:40 fancy-machine containerd[602]: time="2023-12-27T12:08:40.920399864Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:08:40 fancy-machine containerd[602]: time="2023-12-27T12:08:40.921266937Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:13:41 fancy-machine containerd[602]: time="2023-12-27T12:13:41.321043471Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:13:41 fancy-machine containerd[602]: time="2023-12-27T12:13:41.346366882Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:13:43 fancy-machine containerd[602]: time="2023-12-27T12:13:43.881976188Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:13:43 fancy-machine containerd[602]: time="2023-12-27T12:13:43.882339531Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:15:26 fancy-machine containerd[602]: time="2023-12-27T12:15:26.074740369Z" level=error msg="StopPodSandbox for \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" failed" error="rpc error: code = Canceled desc = failed to stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 12:15:26 fancy-machine containerd[602]: time="2023-12-27T12:15:26.835437084Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 12:15:26 fancy-machine containerd[602]: time="2023-12-27T12:15:26.835963491Z" level=info msg="Skipping the sending of signal terminated to container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" because a prior stop with timeout>0 request already sent the signal"
Dec 27 12:15:56 fancy-machine containerd[602]: time="2023-12-27T12:15:56.836092777Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 12:18:44 fancy-machine containerd[602]: time="2023-12-27T12:18:44.320986197Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:18:44 fancy-machine containerd[602]: time="2023-12-27T12:18:44.362803161Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:18:46 fancy-machine containerd[602]: time="2023-12-27T12:18:46.904671420Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:18:46 fancy-machine containerd[602]: time="2023-12-27T12:18:46.904984553Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:23:47 fancy-machine containerd[602]: time="2023-12-27T12:23:47.320968708Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:23:47 fancy-machine containerd[602]: time="2023-12-27T12:23:47.374836110Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:23:49 fancy-machine containerd[602]: time="2023-12-27T12:23:49.923933928Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:23:49 fancy-machine containerd[602]: time="2023-12-27T12:23:49.924276692Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:28:50 fancy-machine containerd[602]: time="2023-12-27T12:28:50.321219302Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:28:50 fancy-machine containerd[602]: time="2023-12-27T12:28:50.350449374Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:28:52 fancy-machine containerd[602]: time="2023-12-27T12:28:52.900217479Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:28:52 fancy-machine containerd[602]: time="2023-12-27T12:28:52.900568956Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:30:56 fancy-machine containerd[602]: time="2023-12-27T12:30:56.836104297Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 12:30:56 fancy-machine containerd[602]: time="2023-12-27T12:30:56.836993135Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 12:33:53 fancy-machine containerd[602]: time="2023-12-27T12:33:53.321524453Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:33:53 fancy-machine containerd[602]: time="2023-12-27T12:33:53.371254331Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:33:55 fancy-machine containerd[602]: time="2023-12-27T12:33:55.912796444Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:33:55 fancy-machine containerd[602]: time="2023-12-27T12:33:55.913139484Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:38:56 fancy-machine containerd[602]: time="2023-12-27T12:38:56.321200103Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:38:56 fancy-machine containerd[602]: time="2023-12-27T12:38:56.362710078Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:38:58 fancy-machine containerd[602]: time="2023-12-27T12:38:58.913383132Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:38:58 fancy-machine containerd[602]: time="2023-12-27T12:38:58.913961441Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:43:59 fancy-machine containerd[602]: time="2023-12-27T12:43:59.321275197Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:43:59 fancy-machine containerd[602]: time="2023-12-27T12:43:59.346751004Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:44:01 fancy-machine containerd[602]: time="2023-12-27T12:44:01.901271153Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:44:01 fancy-machine containerd[602]: time="2023-12-27T12:44:01.901623502Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:45:56 fancy-machine containerd[602]: time="2023-12-27T12:45:56.836786437Z" level=error msg="StopPodSandbox for \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" failed" error="rpc error: code = Canceled desc = failed to stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 12:45:57 fancy-machine containerd[602]: time="2023-12-27T12:45:57.502235878Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 12:45:57 fancy-machine containerd[602]: time="2023-12-27T12:45:57.502711217Z" level=info msg="Skipping the sending of signal terminated to container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" because a prior stop with timeout>0 request already sent the signal"
Dec 27 12:46:27 fancy-machine containerd[602]: time="2023-12-27T12:46:27.503451959Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 12:49:02 fancy-machine containerd[602]: time="2023-12-27T12:49:02.320594055Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:49:02 fancy-machine containerd[602]: time="2023-12-27T12:49:02.351580645Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:49:04 fancy-machine containerd[602]: time="2023-12-27T12:49:04.904325183Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:49:04 fancy-machine containerd[602]: time="2023-12-27T12:49:04.905173634Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:54:05 fancy-machine containerd[602]: time="2023-12-27T12:54:05.320885430Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:54:05 fancy-machine containerd[602]: time="2023-12-27T12:54:05.371477781Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:54:07 fancy-machine containerd[602]: time="2023-12-27T12:54:07.908998846Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:54:07 fancy-machine containerd[602]: time="2023-12-27T12:54:07.909376655Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 12:59:08 fancy-machine containerd[602]: time="2023-12-27T12:59:08.321099573Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 12:59:08 fancy-machine containerd[602]: time="2023-12-27T12:59:08.347282535Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 12:59:10 fancy-machine containerd[602]: time="2023-12-27T12:59:10.894398706Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 12:59:10 fancy-machine containerd[602]: time="2023-12-27T12:59:10.894756351Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:01:27 fancy-machine containerd[602]: time="2023-12-27T13:01:27.502578344Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = DeadlineExceeded desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context deadline exceeded"
Dec 27 13:01:27 fancy-machine containerd[602]: time="2023-12-27T13:01:27.503706816Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 13:04:11 fancy-machine containerd[602]: time="2023-12-27T13:04:11.320678920Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:04:11 fancy-machine containerd[602]: time="2023-12-27T13:04:11.362571404Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:04:13 fancy-machine containerd[602]: time="2023-12-27T13:04:13.908003011Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:04:13 fancy-machine containerd[602]: time="2023-12-27T13:04:13.908318234Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:09:14 fancy-machine containerd[602]: time="2023-12-27T13:09:14.321338288Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:09:14 fancy-machine containerd[602]: time="2023-12-27T13:09:14.350737131Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:09:16 fancy-machine containerd[602]: time="2023-12-27T13:09:16.888174691Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:09:16 fancy-machine containerd[602]: time="2023-12-27T13:09:16.888492638Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:14:17 fancy-machine containerd[602]: time="2023-12-27T13:14:17.321216986Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:14:17 fancy-machine containerd[602]: time="2023-12-27T13:14:17.346786004Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:14:19 fancy-machine containerd[602]: time="2023-12-27T13:14:19.895798019Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:14:19 fancy-machine containerd[602]: time="2023-12-27T13:14:19.896169463Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:16:27 fancy-machine containerd[602]: time="2023-12-27T13:16:27.503653058Z" level=error msg="StopPodSandbox for \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" failed" error="rpc error: code = DeadlineExceeded desc = failed to stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context deadline exceeded"
Dec 27 13:16:28 fancy-machine containerd[602]: time="2023-12-27T13:16:28.219746794Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 13:16:28 fancy-machine containerd[602]: time="2023-12-27T13:16:28.220240937Z" level=info msg="Skipping the sending of signal terminated to container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" because a prior stop with timeout>0 request already sent the signal"
Dec 27 13:16:58 fancy-machine containerd[602]: time="2023-12-27T13:16:58.220750090Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 13:19:20 fancy-machine containerd[602]: time="2023-12-27T13:19:20.320950911Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:19:20 fancy-machine containerd[602]: time="2023-12-27T13:19:20.346618034Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:19:22 fancy-machine containerd[602]: time="2023-12-27T13:19:22.889375596Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:19:22 fancy-machine containerd[602]: time="2023-12-27T13:19:22.889668554Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:24:23 fancy-machine containerd[602]: time="2023-12-27T13:24:23.320983043Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:24:23 fancy-machine containerd[602]: time="2023-12-27T13:24:23.354973509Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:24:25 fancy-machine containerd[602]: time="2023-12-27T13:24:25.883506142Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:24:25 fancy-machine containerd[602]: time="2023-12-27T13:24:25.883792686Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:29:26 fancy-machine containerd[602]: time="2023-12-27T13:29:26.321281752Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:29:26 fancy-machine containerd[602]: time="2023-12-27T13:29:26.351097506Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:29:28 fancy-machine containerd[602]: time="2023-12-27T13:29:28.893908947Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:29:28 fancy-machine containerd[602]: time="2023-12-27T13:29:28.894340199Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:31:58 fancy-machine containerd[602]: time="2023-12-27T13:31:58.219876189Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 13:31:58 fancy-machine containerd[602]: time="2023-12-27T13:31:58.220803715Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 13:34:29 fancy-machine containerd[602]: time="2023-12-27T13:34:29.320782020Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:34:29 fancy-machine containerd[602]: time="2023-12-27T13:34:29.366745945Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:34:31 fancy-machine containerd[602]: time="2023-12-27T13:34:31.899591196Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:34:31 fancy-machine containerd[602]: time="2023-12-27T13:34:31.899929978Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:39:32 fancy-machine containerd[602]: time="2023-12-27T13:39:32.320611674Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:39:32 fancy-machine containerd[602]: time="2023-12-27T13:39:32.354708863Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:39:34 fancy-machine containerd[602]: time="2023-12-27T13:39:34.913233616Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:39:34 fancy-machine containerd[602]: time="2023-12-27T13:39:34.913591171Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:44:35 fancy-machine containerd[602]: time="2023-12-27T13:44:35.321578028Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:44:35 fancy-machine containerd[602]: time="2023-12-27T13:44:35.346286702Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:44:37 fancy-machine containerd[602]: time="2023-12-27T13:44:37.880072596Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:44:37 fancy-machine containerd[602]: time="2023-12-27T13:44:37.880363856Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:46:58 fancy-machine containerd[602]: time="2023-12-27T13:46:58.221049040Z" level=error msg="StopPodSandbox for \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" failed" error="rpc error: code = Canceled desc = failed to stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 13:46:58 fancy-machine containerd[602]: time="2023-12-27T13:46:58.892947368Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 13:46:58 fancy-machine containerd[602]: time="2023-12-27T13:46:58.894185933Z" level=info msg="Skipping the sending of signal terminated to container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" because a prior stop with timeout>0 request already sent the signal"
Dec 27 13:47:28 fancy-machine containerd[602]: time="2023-12-27T13:47:28.894779036Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 13:49:38 fancy-machine containerd[602]: time="2023-12-27T13:49:38.321099753Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:49:38 fancy-machine containerd[602]: time="2023-12-27T13:49:38.358754481Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:49:40 fancy-machine containerd[602]: time="2023-12-27T13:49:40.893235371Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:49:40 fancy-machine containerd[602]: time="2023-12-27T13:49:40.893549795Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:54:41 fancy-machine containerd[602]: time="2023-12-27T13:54:41.320563512Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:54:41 fancy-machine containerd[602]: time="2023-12-27T13:54:41.350864584Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:54:43 fancy-machine containerd[602]: time="2023-12-27T13:54:43.896043613Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:54:43 fancy-machine containerd[602]: time="2023-12-27T13:54:43.896324831Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 13:59:44 fancy-machine containerd[602]: time="2023-12-27T13:59:44.321013426Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 13:59:44 fancy-machine containerd[602]: time="2023-12-27T13:59:44.362207661Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 13:59:46 fancy-machine containerd[602]: time="2023-12-27T13:59:46.925443875Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 13:59:46 fancy-machine containerd[602]: time="2023-12-27T13:59:46.925802320Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:02:28 fancy-machine containerd[602]: time="2023-12-27T14:02:28.892672822Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 14:02:28 fancy-machine containerd[602]: time="2023-12-27T14:02:28.893465703Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 14:04:47 fancy-machine containerd[602]: time="2023-12-27T14:04:47.321093157Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:04:47 fancy-machine containerd[602]: time="2023-12-27T14:04:47.355111898Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:04:49 fancy-machine containerd[602]: time="2023-12-27T14:04:49.899840172Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:04:49 fancy-machine containerd[602]: time="2023-12-27T14:04:49.900190557Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:09:50 fancy-machine containerd[602]: time="2023-12-27T14:09:50.321447569Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:09:50 fancy-machine containerd[602]: time="2023-12-27T14:09:50.358521671Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:09:52 fancy-machine containerd[602]: time="2023-12-27T14:09:52.916518630Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:09:52 fancy-machine containerd[602]: time="2023-12-27T14:09:52.917310243Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:14:53 fancy-machine containerd[602]: time="2023-12-27T14:14:53.320678836Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:14:53 fancy-machine containerd[602]: time="2023-12-27T14:14:53.355578686Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:14:55 fancy-machine containerd[602]: time="2023-12-27T14:14:55.909235080Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:14:55 fancy-machine containerd[602]: time="2023-12-27T14:14:55.909630385Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:17:28 fancy-machine containerd[602]: time="2023-12-27T14:17:28.893473917Z" level=error msg="StopPodSandbox for \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" failed" error="rpc error: code = Canceled desc = failed to stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 14:17:29 fancy-machine containerd[602]: time="2023-12-27T14:17:29.587314883Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 14:17:29 fancy-machine containerd[602]: time="2023-12-27T14:17:29.587868474Z" level=info msg="Skipping the sending of signal terminated to container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" because a prior stop with timeout>0 request already sent the signal"
Dec 27 14:17:59 fancy-machine containerd[602]: time="2023-12-27T14:17:59.588892538Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 14:19:56 fancy-machine containerd[602]: time="2023-12-27T14:19:56.321069935Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:19:56 fancy-machine containerd[602]: time="2023-12-27T14:19:56.362345208Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:19:58 fancy-machine containerd[602]: time="2023-12-27T14:19:58.900210494Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:19:58 fancy-machine containerd[602]: time="2023-12-27T14:19:58.900539062Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:24:59 fancy-machine containerd[602]: time="2023-12-27T14:24:59.321439176Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:24:59 fancy-machine containerd[602]: time="2023-12-27T14:24:59.355937790Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:25:01 fancy-machine containerd[602]: time="2023-12-27T14:25:01.893093320Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:25:01 fancy-machine containerd[602]: time="2023-12-27T14:25:01.893396616Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:30:02 fancy-machine containerd[602]: time="2023-12-27T14:30:02.321217709Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:30:02 fancy-machine containerd[602]: time="2023-12-27T14:30:02.366723864Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:30:04 fancy-machine containerd[602]: time="2023-12-27T14:30:04.912769844Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:30:04 fancy-machine containerd[602]: time="2023-12-27T14:30:04.912460706Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:32:59 fancy-machine containerd[602]: time="2023-12-27T14:32:59.588014220Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = DeadlineExceeded desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context deadline exceeded"
Dec 27 14:32:59 fancy-machine containerd[602]: time="2023-12-27T14:32:59.589500817Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 14:35:05 fancy-machine containerd[602]: time="2023-12-27T14:35:05.320959295Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:35:05 fancy-machine containerd[602]: time="2023-12-27T14:35:05.363030933Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:35:07 fancy-machine containerd[602]: time="2023-12-27T14:35:07.899926424Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:35:07 fancy-machine containerd[602]: time="2023-12-27T14:35:07.900197100Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:40:08 fancy-machine containerd[602]: time="2023-12-27T14:40:08.321003596Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:40:08 fancy-machine containerd[602]: time="2023-12-27T14:40:08.351179281Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:40:10 fancy-machine containerd[602]: time="2023-12-27T14:40:10.891890379Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:40:10 fancy-machine containerd[602]: time="2023-12-27T14:40:10.892196676Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:45:11 fancy-machine containerd[602]: time="2023-12-27T14:45:11.320979119Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:45:11 fancy-machine containerd[602]: time="2023-12-27T14:45:11.346236823Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:45:13 fancy-machine containerd[602]: time="2023-12-27T14:45:13.884235147Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:45:13 fancy-machine containerd[602]: time="2023-12-27T14:45:13.884568252Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:47:59 fancy-machine containerd[602]: time="2023-12-27T14:47:59.589149269Z" level=error msg="StopPodSandbox for \"18517e9ae953629c515c5381bccb63cf7d66536fc97dfeabd5d1d4a792340b21\" failed" error="rpc error: code = DeadlineExceeded desc = failed to stop container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context deadline exceeded"
Dec 27 14:48:00 fancy-machine containerd[602]: time="2023-12-27T14:48:00.312583308Z" level=info msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" with timeout 30 (s)"
Dec 27 14:48:00 fancy-machine containerd[602]: time="2023-12-27T14:48:00.313085639Z" level=info msg="Skipping the sending of signal terminated to container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" because a prior stop with timeout>0 request already sent the signal"
Dec 27 14:48:30 fancy-machine containerd[602]: time="2023-12-27T14:48:30.313462730Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 14:50:14 fancy-machine containerd[602]: time="2023-12-27T14:50:14.321509361Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:50:14 fancy-machine containerd[602]: time="2023-12-27T14:50:14.362522754Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:50:16 fancy-machine containerd[602]: time="2023-12-27T14:50:16.912842802Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:50:16 fancy-machine containerd[602]: time="2023-12-27T14:50:16.913269262Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 14:55:17 fancy-machine containerd[602]: time="2023-12-27T14:55:17.321254400Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 14:55:17 fancy-machine containerd[602]: time="2023-12-27T14:55:17.355692313Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 14:55:19 fancy-machine containerd[602]: time="2023-12-27T14:55:19.909404357Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 14:55:19 fancy-machine containerd[602]: time="2023-12-27T14:55:19.909761889Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 15:00:20 fancy-machine containerd[602]: time="2023-12-27T15:00:20.321454327Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 15:00:20 fancy-machine containerd[602]: time="2023-12-27T15:00:20.362744854Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 15:00:22 fancy-machine containerd[602]: time="2023-12-27T15:00:22.904767760Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 15:00:22 fancy-machine containerd[602]: time="2023-12-27T15:00:22.905090613Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 15:03:30 fancy-machine containerd[602]: time="2023-12-27T15:03:30.312782700Z" level=error msg="StopContainer for \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" to be killed: wait container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": context canceled"
Dec 27 15:03:30 fancy-machine containerd[602]: time="2023-12-27T15:03:30.315478522Z" level=info msg="Kill container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\""
Dec 27 15:05:23 fancy-machine containerd[602]: time="2023-12-27T15:05:23.321157730Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 15:05:23 fancy-machine containerd[602]: time="2023-12-27T15:05:23.350507841Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 15:05:25 fancy-machine containerd[602]: time="2023-12-27T15:05:25.891997844Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 15:05:25 fancy-machine containerd[602]: time="2023-12-27T15:05:25.892299576Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"
Dec 27 15:10:26 fancy-machine containerd[602]: time="2023-12-27T15:10:26.320983632Z" level=info msg="TaskExit event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856}"
Dec 27 15:10:26 fancy-machine containerd[602]: time="2023-12-27T15:10:26.352448171Z" level=warning msg="Ignoring error killing container \"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Dec 27 15:10:28 fancy-machine containerd[602]: time="2023-12-27T15:10:28.897571677Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Dec 27 15:10:28 fancy-machine containerd[602]: time="2023-12-27T15:10:28.897920677Z" level=error msg="Failed to handle backOff event container_id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" id:\"dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a\" pid:29098 exit_status:143 exited_at:{seconds:1703673835 nanos:457869856} for dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/dd57742f947c373043ac39c78d0ee7a1d01f02a92c1df4b22bff704dc9cc754a/rootfs: device or resource busy: unknown"

Dec 27 '23 15:12 markusthoemmes

Here's the stacktrace of the sandbox in such a case. The 68min goroutines are when the kill signal was sent. Superficially, this looks to me as if the termination signal was indeed received but the sandbox is hung while shutting down.

Found sandbox "7aa7bce92e1e8639b62029ac4ce95b835cbb00b62cd99477ad7121e40c767fa3", PID: 14257
Retrieving sandbox stacks
     *** Stack dump ***
goroutine 2060216 [running]:
gvisor.dev/gvisor/pkg/log.Stacks(0x0?)
	pkg/log/log.go:319 +0x67
gvisor.dev/gvisor/runsc/boot.(*debug).Stacks(0x2, 0xc0006e4530?, 0xc00052a070)
	runsc/boot/debug.go:26 +0x1d
reflect.Value.call({0xc000615380?, 0xc0000e4538?, 0xc000163c40?}, {0x1225b9b, 0x4}, {0xc000163e90, 0x3, 0xc000163c70?})
	GOROOT/src/reflect/value.go:596 +0xce7
reflect.Value.Call({0xc000615380?, 0xc0000e4538?, 0x1eb8b40?}, {0xc000163e90?, 0x1eb8b40?, 0x16?})
	GOROOT/src/reflect/value.go:380 +0xb9
gvisor.dev/gvisor/pkg/urpc.(*Server).handleOne(0xc0000e21e0, 0xc000e18180)
	pkg/urpc/urpc.go:338 +0x4b9
gvisor.dev/gvisor/pkg/urpc.(*Server).handleRegistered(...)
	pkg/urpc/urpc.go:433
gvisor.dev/gvisor/pkg/urpc.(*Server).StartHandling.func1()
	pkg/urpc/urpc.go:453 +0x76
created by gvisor.dev/gvisor/pkg/urpc.(*Server).StartHandling in goroutine 39
	pkg/urpc/urpc.go:451 +0x6b

goroutine 1 [semacquire, 21680 minutes]:
sync.runtime_Semacquire(0xc000107608?)
	GOROOT/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x0?)
	GOROOT/src/sync/waitgroup.go:116 +0x48
gvisor.dev/gvisor/pkg/sentry/kernel.(*Kernel).WaitExited(...)
	pkg/sentry/kernel/kernel.go:1178
gvisor.dev/gvisor/runsc/boot.(*Loader).WaitExit(0xc0001ca400)
	runsc/boot/loader.go:1276 +0x28
gvisor.dev/gvisor/runsc/cmd.(*Boot).Execute(0xc0002e2000, {0xc0001220b0?, 0xc00015bac0?}, 0xc0001f1b90, {0xc00015bac0, 0x2, 0x1b?})
	runsc/cmd/boot.go:497 +0x18e5
github.com/google/subcommands.(*Commander).Execute(0xc00014c000, {0x144e880, 0x1eb8b40}, {0xc00015bac0, 0x2, 0x2})
	external/com_github_google_subcommands/subcommands.go:200 +0x38c
github.com/google/subcommands.Execute(...)
	external/com_github_google_subcommands/subcommands.go:481
gvisor.dev/gvisor/runsc/cli.Main()
	runsc/cli/main.go:219 +0x14b0
main.main()
	runsc/main.go:31 +0xf

goroutine 9 [sync.Cond.Wait, 21680 minutes]:
sync.runtime_notifyListWait(0xc0000aa6c8, 0x0)
	GOROOT/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc0000aa000?)
	GOROOT/src/sync/cond.go:70 +0x85
gvisor.dev/gvisor/pkg/sentry/pgalloc.(*MemoryFile).findReclaimable(0xc0000aa000)
	pkg/sentry/pgalloc/pgalloc.go:1436 +0xbd
gvisor.dev/gvisor/pkg/sentry/pgalloc.(*MemoryFile).runReclaim(0xc0000aa000)
	pkg/sentry/pgalloc/pgalloc.go:1345 +0x78
created by gvisor.dev/gvisor/pkg/sentry/pgalloc.NewMemoryFile in goroutine 1
	pkg/sentry/pgalloc/pgalloc.go:368 +0x27b

goroutine 10 [chan receive, 21441 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 1
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 11 [sleep]:
time.Sleep(0x61a80)
	GOROOT/src/runtime/time.go:195 +0x125
gvisor.dev/gvisor/pkg/sentry/platform/systrap.controlFastPath()
	pkg/sentry/platform/systrap/metrics.go:263 +0x18
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.New.func2 in goroutine 1
	pkg/sentry/platform/systrap/systrap.go:345 +0x1a

goroutine 12 [sync.Cond.Wait, 68 minutes]:
sync.runtime_notifyListWait(0xc0000aaec8, 0x3f1eb4)
	GOROOT/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc0000aa800?)
	GOROOT/src/sync/cond.go:70 +0x85
gvisor.dev/gvisor/pkg/sentry/pgalloc.(*MemoryFile).findReclaimable(0xc0000aa800)
	pkg/sentry/pgalloc/pgalloc.go:1436 +0xbd
gvisor.dev/gvisor/pkg/sentry/pgalloc.(*MemoryFile).runReclaim(0xc0000aa800)
	pkg/sentry/pgalloc/pgalloc.go:1345 +0x78
created by gvisor.dev/gvisor/pkg/sentry/pgalloc.NewMemoryFile in goroutine 1
	pkg/sentry/pgalloc/pgalloc.go:368 +0x27b

goroutine 13 [select]:
gvisor.dev/gvisor/pkg/sentry/kernel.(*Timekeeper).startUpdater.func1()
	pkg/sentry/kernel/timekeeper.go:254 +0x159
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Timekeeper).startUpdater in goroutine 1
	pkg/sentry/kernel/timekeeper.go:224 +0xd3

goroutine 14 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc00013ad98, 0x1, 0x1?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00013ad80?, 0x0?, 0x87?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start(0xc00013ad80, 0x0?)
	pkg/tcpip/transport/tcp/dispatcher.go:287 +0xb5
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*dispatcher).init in goroutine 1
	pkg/tcpip/transport/tcp/dispatcher.go:391 +0x13d

goroutine 15 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc00013ae28, 0x1, 0x1?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00013ae10?, 0x0?, 0xce?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start(0xc00013ae10, 0x0?)
	pkg/tcpip/transport/tcp/dispatcher.go:287 +0xb5
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*dispatcher).init in goroutine 1
	pkg/tcpip/transport/tcp/dispatcher.go:391 +0x13d

goroutine 16 [select, 70 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc00013aeb8, 0x1, 0x1?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00013aea0?, 0x0?, 0xbc?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start(0xc00013aea0, 0x0?)
	pkg/tcpip/transport/tcp/dispatcher.go:287 +0xb5
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*dispatcher).init in goroutine 1
	pkg/tcpip/transport/tcp/dispatcher.go:391 +0x13d

goroutine 33 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc00013af48, 0x1, 0x1?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00013af30?, 0x0?, 0xa0?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start(0xc00013af30, 0x0?)
	pkg/tcpip/transport/tcp/dispatcher.go:287 +0xb5
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*dispatcher).init in goroutine 1
	pkg/tcpip/transport/tcp/dispatcher.go:391 +0x13d

goroutine 34 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc00013afd8, 0x1, 0x1?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00013afc0?, 0x0?, 0x9c?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start(0xc00013afc0, 0x0?)
	pkg/tcpip/transport/tcp/dispatcher.go:287 +0xb5
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*dispatcher).init in goroutine 1
	pkg/tcpip/transport/tcp/dispatcher.go:391 +0x13d

goroutine 35 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc00013b068, 0x1, 0x1?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00013b050?, 0x0?, 0xc7?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start(0xc00013b050, 0x0?)
	pkg/tcpip/transport/tcp/dispatcher.go:287 +0xb5
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*dispatcher).init in goroutine 1
	pkg/tcpip/transport/tcp/dispatcher.go:391 +0x13d

goroutine 36 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc00013b0f8, 0x1, 0x1?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00013b0e0?, 0x0?, 0x5c?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start(0xc00013b0e0, 0x0?)
	pkg/tcpip/transport/tcp/dispatcher.go:287 +0xb5
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*dispatcher).init in goroutine 1
	pkg/tcpip/transport/tcp/dispatcher.go:391 +0x13d

goroutine 37 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc00013b188, 0x1, 0x1?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00013b170?, 0x0?, 0x63?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*processor).start(0xc00013b170, 0x0?)
	pkg/tcpip/transport/tcp/dispatcher.go:287 +0xb5
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*dispatcher).init in goroutine 1
	pkg/tcpip/transport/tcp/dispatcher.go:391 +0x13d

goroutine 39 [syscall]:
syscall.Syscall6(0x0?, 0x0?, 0xffffffffffffffff?, 0x0?, 0xb?, 0xffffffffffffffff?, 0xc00071a6cc?)
	GOROOT/src/syscall/syscall_linux.go:91 +0x30
gvisor.dev/gvisor/pkg/unet.(*Socket).wait(0xc0000007b0, 0x0)
	pkg/unet/unet_unsafe.go:53 +0x9b
gvisor.dev/gvisor/pkg/unet.(*ServerSocket).Accept(0xc0000e4050)
	pkg/unet/unet.go:517 +0x125
gvisor.dev/gvisor/pkg/control/server.(*Server).serve(0xc0000c9620)
	pkg/control/server/server.go:104 +0x39
gvisor.dev/gvisor/pkg/control/server.(*Server).StartServing.func1()
	pkg/control/server/server.go:92 +0x1c
created by gvisor.dev/gvisor/pkg/control/server.(*Server).StartServing in goroutine 1
	pkg/control/server/server.go:91 +0x85

goroutine 49 [select]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc0000c96e0, 0x1, 0x0?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc000816f78?, 0x0?, 0xe3?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.(*queueDispatcher).dispatchLoop(0xc000268000)
	pkg/tcpip/link/qdisc/fifo/fifo.go:96 +0xcf
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New.func1()
	pkg/tcpip/link/qdisc/fifo/fifo.go:82 +0x4f
created by gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New in goroutine 40
	pkg/tcpip/link/qdisc/fifo/fifo.go:80 +0x9f

goroutine 50 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc0000c9700, 0x1, 0x0?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc0000dbf78?, 0xe0?, 0x24?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.(*queueDispatcher).dispatchLoop(0xc000268070)
	pkg/tcpip/link/qdisc/fifo/fifo.go:96 +0xcf
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New.func1()
	pkg/tcpip/link/qdisc/fifo/fifo.go:82 +0x4f
created by gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New in goroutine 40
	pkg/tcpip/link/qdisc/fifo/fifo.go:80 +0x9f

goroutine 51 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc0000c9720, 0x1, 0x0?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc000c06f78?, 0x20?, 0xa3?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.(*queueDispatcher).dispatchLoop(0xc0002680e0)
	pkg/tcpip/link/qdisc/fifo/fifo.go:96 +0xcf
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New.func1()
	pkg/tcpip/link/qdisc/fifo/fifo.go:82 +0x4f
created by gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New in goroutine 40
	pkg/tcpip/link/qdisc/fifo/fifo.go:80 +0x9f

goroutine 52 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc0000c96c0, 0x1, 0x0?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc000810f78?, 0x10?, 0xe3?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.(*queueDispatcher).dispatchLoop(0xc000268150)
	pkg/tcpip/link/qdisc/fifo/fifo.go:96 +0xcf
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New.func1()
	pkg/tcpip/link/qdisc/fifo/fifo.go:82 +0x4f
created by gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New in goroutine 40
	pkg/tcpip/link/qdisc/fifo/fifo.go:80 +0x9f

goroutine 53 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc0000c9760, 0x1, 0x0?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc000c03f78?, 0x60?, 0x47?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.(*queueDispatcher).dispatchLoop(0xc0002681c0)
	pkg/tcpip/link/qdisc/fifo/fifo.go:96 +0xcf
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New.func1()
	pkg/tcpip/link/qdisc/fifo/fifo.go:82 +0x4f
created by gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New in goroutine 40
	pkg/tcpip/link/qdisc/fifo/fifo.go:80 +0x9f

goroutine 54 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc0000c9740, 0x1, 0x0?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc000b2cf78?, 0x70?, 0x7?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.(*queueDispatcher).dispatchLoop(0xc000268230)
	pkg/tcpip/link/qdisc/fifo/fifo.go:96 +0xcf
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New.func1()
	pkg/tcpip/link/qdisc/fifo/fifo.go:82 +0x4f
created by gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New in goroutine 40
	pkg/tcpip/link/qdisc/fifo/fifo.go:80 +0x9f

goroutine 55 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc0000c9780, 0x1, 0x0?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc000b29f78?, 0xc0?, 0xa7?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.(*queueDispatcher).dispatchLoop(0xc0002682a0)
	pkg/tcpip/link/qdisc/fifo/fifo.go:96 +0xcf
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New.func1()
	pkg/tcpip/link/qdisc/fifo/fifo.go:82 +0x4f
created by gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New in goroutine 40
	pkg/tcpip/link/qdisc/fifo/fifo.go:80 +0x9f

goroutine 56 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sync.Gopark(...)
	pkg/sync/runtime_unsafe.go:33
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).nextWaker(0xc0000c97a0, 0x1, 0x0?)
	pkg/sleep/sleep_unsafe.go:209 +0x7a
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).fetch(0xc00040df78?, 0xb0?, 0xa5?)
	pkg/sleep/sleep_unsafe.go:256 +0x2b
gvisor.dev/gvisor/pkg/sleep.(*Sleeper).Fetch(...)
	pkg/sleep/sleep_unsafe.go:279
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.(*queueDispatcher).dispatchLoop(0xc000268310)
	pkg/tcpip/link/qdisc/fifo/fifo.go:96 +0xcf
gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New.func1()
	pkg/tcpip/link/qdisc/fifo/fifo.go:82 +0x4f
created by gvisor.dev/gvisor/pkg/tcpip/link/qdisc/fifo.New in goroutine 40
	pkg/tcpip/link/qdisc/fifo/fifo.go:80 +0x9f

goroutine 65 [syscall]:
syscall.Syscall6(0xc000310218?, 0xc00056bdb0?, 0x47c94a?, 0x1e619c0?, 0xc000000800?, 0xc0012d8500?, 0xc00056bde0?)
	GOROOT/src/syscall/syscall_linux.go:91 +0x30
gvisor.dev/gvisor/pkg/tcpip/link/rawfile.BlockingPollUntilStopped(0x14453a0?, 0xc00056be78?, 0x7af1?)
	pkg/tcpip/link/rawfile/rawfile_unsafe.go:248 +0x65
gvisor.dev/gvisor/pkg/tcpip/link/fdbased.(*packetMMapDispatcher).readMMappedPacket(0xc0000a03c0)
	pkg/tcpip/link/fdbased/mmap.go:140 +0x8f
gvisor.dev/gvisor/pkg/tcpip/link/fdbased.(*packetMMapDispatcher).dispatch(0xc0000a03c0)
	pkg/tcpip/link/fdbased/mmap.go:172 +0x45
gvisor.dev/gvisor/pkg/tcpip/link/fdbased.(*endpoint).dispatchLoop(0xc0001cc000, {0x144dcb0, 0xc0000a03c0})
	pkg/tcpip/link/fdbased/endpoint.go:755 +0x32
gvisor.dev/gvisor/pkg/tcpip/link/fdbased.(*endpoint).Attach.func1(0xc0000b3f88?)
	pkg/tcpip/link/fdbased/endpoint.go:430 +0x39
created by gvisor.dev/gvisor/pkg/tcpip/link/fdbased.(*endpoint).Attach in goroutine 40
	pkg/tcpip/link/fdbased/endpoint.go:429 +0x11e

goroutine 41 [syscall, 21680 minutes]:
syscall.Syscall6(0xc000271660?, 0x424c9c?, 0x1eba300?, 0x40fc7e?, 0x7fcea3d9e420?, 0x101010101010101?, 0x0?)
	GOROOT/src/syscall/syscall_linux.go:91 +0x30
golang.org/x/sys/unix.ppoll(0x40e125?, 0x10?, 0x1169f20?, 0xc0002e83f0?)
	external/org_golang_x_sys/unix/zsyscall_linux.go:124 +0x57
golang.org/x/sys/unix.Ppoll({0xc000271760?, 0x40?, 0xc0002ee000?}, 0x0?, 0x0?)
	external/org_golang_x_sys/unix/syscall_linux.go:149 +0x38
gvisor.dev/gvisor/runsc/boot.(*Loader).startGoferMonitor.func1.1(...)
	runsc/boot/loader.go:1076
gvisor.dev/gvisor/runsc/specutils.RetryEintr(...)
	runsc/specutils/specutils.go:698
gvisor.dev/gvisor/runsc/boot.(*Loader).startGoferMonitor.func1()
	runsc/boot/loader.go:1074 +0xf3
created by gvisor.dev/gvisor/runsc/boot.(*Loader).startGoferMonitor in goroutine 1
	runsc/boot/loader.go:1066 +0x105

goroutine 42 [syscall, 21680 minutes]:
syscall.Syscall6(0x3?, 0xc00025c504?, 0x4?, 0xc00025c594?, 0x4?, 0xc00025c608?, 0x4?)
	GOROOT/src/syscall/syscall_linux.go:91 +0x30
golang.org/x/sys/unix.ppoll(0x0?, 0x0?, 0x0?, 0x0?)
	external/org_golang_x_sys/unix/zsyscall_linux.go:124 +0x57
golang.org/x/sys/unix.Ppoll({0xc000271f70?, 0x1?, 0x0?}, 0x824?, 0xc00025ce9c?)
	external/org_golang_x_sys/unix/syscall_linux.go:149 +0x38
gvisor.dev/gvisor/pkg/lisafs.(*Client).watchdog(0xc00038c2c0)
	pkg/lisafs/client.go:172 +0x9f
created by gvisor.dev/gvisor/pkg/lisafs.NewClient in goroutine 1
	pkg/lisafs/client.go:84 +0x1c5

goroutine 113 [chan receive, 21680 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 100
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 66 [syscall, 21680 minutes]:
syscall.Syscall6(0xc00025c1b3?, 0x3?, 0xc00025c5d8?, 0x4?, 0xc00025cba4?, 0x4?, 0x0?)
	GOROOT/src/syscall/syscall_linux.go:91 +0x30
golang.org/x/sys/unix.ppoll(0xc000272e78?, 0x88d160?, 0xc000272e2f?, 0x0?)
	external/org_golang_x_sys/unix/zsyscall_linux.go:124 +0x57
golang.org/x/sys/unix.Ppoll({0xc000272f70?, 0x0?, 0xc000272fd0?}, 0x865b45?, 0x24b7e1?)
	external/org_golang_x_sys/unix/syscall_linux.go:149 +0x38
gvisor.dev/gvisor/pkg/lisafs.(*Client).watchdog(0xc0001bca50)
	pkg/lisafs/client.go:172 +0x9f
created by gvisor.dev/gvisor/pkg/lisafs.NewClient in goroutine 1
	pkg/lisafs/client.go:84 +0x1c5

goroutine 47 [select, 21680 minutes]:
gvisor.dev/gvisor/pkg/sentry/kernel/time.(*Timer).runGoroutine(0xc00039e200)
	pkg/sentry/kernel/time/time.go:507 +0x6d
created by gvisor.dev/gvisor/pkg/sentry/kernel/time.(*Timer).init in goroutine 1
	pkg/sentry/kernel/time/time.go:487 +0x156

goroutine 97 [select, 21680 minutes]:
reflect.rselect({0xc00013a900, 0x22, 0x2?})
	GOROOT/src/runtime/select.go:589 +0x2c5
reflect.Select({0xc0000fc000?, 0x22, 0x0?})
	GOROOT/src/reflect/value.go:3104 +0x5ea
gvisor.dev/gvisor/pkg/sighandling.handleSignals({0xc0002ec200, 0x21, 0xc0001bca50?}, 0xc0004ea450, 0xc0003f31f0?, 0xc0003fdfb8?)
	pkg/sighandling/sighandling.go:44 +0x306
created by gvisor.dev/gvisor/pkg/sighandling.StartSignalForwarding in goroutine 1
	pkg/sighandling/sighandling.go:107 +0x229

goroutine 98 [select]:
gvisor.dev/gvisor/pkg/sentry/watchdog.(*Watchdog).loop(0xc00009e100)
	pkg/sentry/watchdog/watchdog.go:250 +0x7b
created by gvisor.dev/gvisor/pkg/sentry/watchdog.(*Watchdog).Start in goroutine 1
	pkg/sentry/watchdog/watchdog.go:206 +0x1cc

goroutine 81 [syscall, 21680 minutes]:
os/signal.signal_recv()
	GOROOT/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
	GOROOT/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
	GOROOT/src/os/signal/signal.go:151 +0x1f

goroutine 99 [sync.Cond.Wait, 68 minutes]:
sync.runtime_notifyListWait(0xc00033dcf8, 0x761d7b)
	GOROOT/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000308000?)
	GOROOT/src/sync/cond.go:70 +0x85
gvisor.dev/gvisor/pkg/sentry/kernel.(*Kernel).runCPUClockTicker(0xc00033dc00)
	pkg/sentry/kernel/task_sched.go:349 +0x170
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Kernel).Start in goroutine 1
	pkg/sentry/kernel/kernel.go:1009 +0x179

goroutine 100 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).block(0xc000352a80, 0x0, 0x0)
	pkg/sentry/kernel/task_block.go:164 +0x14b
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Block(...)
	pkg/sentry/kernel/task_block.go:119
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.Pause(0xc00051b990?, 0x6d74bd?, {{0x11}, {0x7eea20872b20}, {0x0}, {0x8}, {0x0}, {0x0}})
	pkg/sentry/syscalls/linux/sys_signal.go:333 +0x18
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0xc000352a80, 0x22, {{0x11}, {0x7eea20872b20}, {0x0}, {0x8}, {0x0}, {0x0}})
	pkg/sentry/kernel/task_syscall.go:142 +0x673
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallInvoke(0xc000352a80, 0xc0001edc80?, {{0x11}, {0x7eea20872b20}, {0x0}, {0x8}, {0x0}, {0x0}})
	pkg/sentry/kernel/task_syscall.go:322 +0x45
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallEnter(0xc00051be00?, 0xcf4907?, {{0x11}, {0x7eea20872b20}, {0x0}, {0x8}, {0x0}, {0x0}})
	pkg/sentry/kernel/task_syscall.go:282 +0x59
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscall(0xc0001c6340?)
	pkg/sentry/kernel/task_syscall.go:257 +0x2d5
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0xc000352a80?, 0xc000352a80)
	pkg/sentry/kernel/task_run.go:269 +0xfb7
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0xc000352a80, 0x1)
	pkg/sentry/kernel/task_run.go:98 +0x1ef
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start in goroutine 1
	pkg/sentry/kernel/task_start.go:391 +0xe5

goroutine 82 [select, 21680 minutes]:
gvisor.dev/gvisor/pkg/sentry/kernel/time.(*Timer).runGoroutine(0xc00014c100)
	pkg/sentry/kernel/time/time.go:507 +0x6d
created by gvisor.dev/gvisor/pkg/sentry/kernel/time.(*Timer).init in goroutine 100
	pkg/sentry/kernel/time/time.go:487 +0x156

goroutine 59 [semacquire, 21680 minutes]:
sync.runtime_Semacquire(0x0?)
	GOROOT/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc000619490?)
	GOROOT/src/sync/waitgroup.go:116 +0x48
gvisor.dev/gvisor/pkg/sentry/kernel.(*ThreadGroup).WaitExited(...)
	pkg/sentry/kernel/task_run.go:388
gvisor.dev/gvisor/runsc/boot.(*Loader).wait(0xc000619520?, 0xc0000f4000)
	runsc/boot/loader.go:1264 +0x25
gvisor.dev/gvisor/runsc/boot.(*Loader).waitContainer(0xc0001ca400, {0xc000154400, 0x40}, 0xc0003f3480)
	runsc/boot/loader.go:1210 +0xbe
gvisor.dev/gvisor/runsc/boot.(*containerManager).Wait(0xc000011a58, 0xc00037a130, 0xc0003f3480)
	runsc/boot/controller.go:585 +0xaf
reflect.Value.call({0xc000614b40?, 0xc0000e4290?, 0xc000619c40?}, {0x1225b9b, 0x4}, {0xc000619e90, 0x3, 0xc000619c70?})
	GOROOT/src/reflect/value.go:596 +0xce7
reflect.Value.Call({0xc000614b40?, 0xc0000e4290?, 0xc00037a130?}, {0xc000619e90?, 0xc00037a130?, 0x16?})
	GOROOT/src/reflect/value.go:380 +0xb9
gvisor.dev/gvisor/pkg/urpc.(*Server).handleOne(0xc0000e21e0, 0xc0001e6ba0)
	pkg/urpc/urpc.go:338 +0x4b9
gvisor.dev/gvisor/pkg/urpc.(*Server).handleRegistered(...)
	pkg/urpc/urpc.go:433
gvisor.dev/gvisor/pkg/urpc.(*Server).StartHandling.func1()
	pkg/urpc/urpc.go:453 +0x76
created by gvisor.dev/gvisor/pkg/urpc.(*Server).StartHandling in goroutine 39
	pkg/urpc/urpc.go:451 +0x6b

goroutine 2060231 [select]:
gvisor.dev/gvisor/pkg/sentry/kernel/time.(*Timer).runGoroutine(0xc0014a0500)
	pkg/sentry/kernel/time/time.go:507 +0x6d
created by gvisor.dev/gvisor/pkg/sentry/kernel/time.(*Timer).init in goroutine 65
	pkg/sentry/kernel/time/time.go:487 +0x156

goroutine 60 [syscall, 68 minutes]:
syscall.Syscall6(0xc000731aa0?, 0xc000a31f78?, 0x1e8f7d0?, 0x408e80?, 0xc000a31f20?, 0xc000204a50?, 0x409f1a?)
	GOROOT/src/syscall/syscall_linux.go:91 +0x30
gvisor.dev/gvisor/pkg/fdnotifier.epollWait(0xc000bdc520?, {0xc000204af0?, 0x275?, 0x1e875e0?}, 0xc0001e6cc0?)
	pkg/fdnotifier/poll_unsafe.go:77 +0x4d
gvisor.dev/gvisor/pkg/fdnotifier.(*notifier).waitAndNotify(0xc00038a3f0)
	pkg/fdnotifier/fdnotifier.go:149 +0x58
created by gvisor.dev/gvisor/pkg/fdnotifier.newNotifier in goroutine 115
	pkg/fdnotifier/fdnotifier.go:64 +0xb6

goroutine 75 [sync.Cond.Wait, 68 minutes]:
sync.runtime_notifyListWait(0xc00001f6c8, 0x1)
	GOROOT/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc00001f000?)
	GOROOT/src/sync/cond.go:70 +0x85
gvisor.dev/gvisor/pkg/sentry/pgalloc.(*MemoryFile).findReclaimable(0xc00001f000)
	pkg/sentry/pgalloc/pgalloc.go:1436 +0xbd
gvisor.dev/gvisor/pkg/sentry/pgalloc.(*MemoryFile).runReclaim(0xc00001f000)
	pkg/sentry/pgalloc/pgalloc.go:1345 +0x78
created by gvisor.dev/gvisor/pkg/sentry/pgalloc.NewMemoryFile in goroutine 115
	pkg/sentry/pgalloc/pgalloc.go:368 +0x27b

goroutine 145 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 122
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 31 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 210
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 147 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 181
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 126 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 124
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 30 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 28
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 162 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 128
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 146 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 124
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 195 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 193
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 138 [chan receive, 21679 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 137
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 13731 [chan receive, 21445 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 13664
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 14033 [chan receive, 21441 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 13966
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 13730 [chan receive, 21441 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 13664
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 13921 [chan receive, 17118 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 13882
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 14003 [chan receive, 21441 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 13983
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

goroutine 13646 [chan receive, 21445 minutes]:
gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess.func1()
	pkg/sentry/platform/systrap/subprocess.go:316 +0x54
created by gvisor.dev/gvisor/pkg/sentry/platform/systrap.newSubprocess in goroutine 13713
	pkg/sentry/platform/systrap/subprocess.go:313 +0x265

Dec 27 '23 15:12 markusthoemmes

The sandbox has one alive process:

goroutine 100 [select, 68 minutes]:
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).block(0xc000352a80, 0x0, 0x0)
        pkg/sentry/kernel/task_block.go:164 +0x14b
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Block(...)
        pkg/sentry/kernel/task_block.go:119
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.Pause(0xc00051b990?, 0x6d74bd?, {{0x11}, {0x7eea20872b20}, {0x0}, {0x8}, {0x0}, {0x0}})
        pkg/sentry/syscalls/linux/sys_signal.go:333 +0x18
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0xc000352a80, 0x22, {{0x11}, {0x7eea20872b20}, {0x0}, {0x8}, {0x0}, {0x0}})
        pkg/sentry/kernel/task_syscall.go:142 +0x673
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallInvoke(0xc000352a80, 0xc0001edc80?, {{0x11}, {0x7eea20872b20}, {0x0}, {0x8}, {0x0}, {0x0}})
        pkg/sentry/kernel/task_syscall.go:322 +0x45
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallEnter(0xc00051be00?, 0xcf4907?, {{0x11}, {0x7eea20872b20}, {0x0}, {0x8}, {0x0}, {0x0}})
        pkg/sentry/kernel/task_syscall.go:282 +0x59
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscall(0xc0001c6340?)
        pkg/sentry/kernel/task_syscall.go:257 +0x2d5
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0xc000352a80?, 0xc000352a80)
        pkg/sentry/kernel/task_run.go:269 +0xfb7
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0xc000352a80, 0x1)
        pkg/sentry/kernel/task_run.go:98 +0x1ef
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start in goroutine 1
        pkg/sentry/kernel/task_start.go:391 +0xe5

Jan 02 '24 19:01 avagin

Both runsc kill (first to the container, then to the sandbox directly) did nothing to actually stop either container or sandbox.

D1222 10:11:04.797094  2934612 container.go:673] Signal container, cid: 52c47b6ff1867b52eca5babb8920c390577d96bb287d3c2170fc37a37415c707, signal: terminated (15)

runsc kill sends SETERM by default. This signal can be handled by a process. You can try to send SIGKILL.

Jan 02 '24 19:01 avagin

@avagin thanks for chiming in.

After your comments, I made a few more tests around SIGTERM handling to see if there's any regression here, but no: Even if I handle SIGTERM and then ignore it in the respective container, eventually (after the terminationGracePeriod) SIGKILL will be sent via Kubernetes/containerd. However, that seems to never reach the process in question and thus it's stuck.

Jan 03 '24 08:01 markusthoemmes

With a bit more digging, I think what I'm looking at is that the container has been successfully removed since its processes are gone (including its state file as mentioned by the logs above) but the actual processes of the container are running via the sandbox. They are holding onto the containers rootfs though and since nothing, seemingly, ever tried to remove the sandbox because final removal of the container already failed, we're locked.

The delete is then retried over and over by containerd, we keep alerting on a missing state file because we've already successfully deleted the container and the only contention left is that the processes running under the sandbox keep a hold of the rootfs.

Sorry for the probably very imprecise explanation. I'm building a mental model as I go here 😅

Jan 04 '24 11:01 markusthoemmes

@manninglucas any chance that the two rather recent commits https://github.com/google/gvisor/commit/3ab01aedb8741e09f90ec2858fbd80077757347b and https://github.com/google/gvisor/commit/6a112c60a257dadac59962e0bc9e9b5aee70b5b6 have anything to do with this?

Jan 04 '24 15:01 markusthoemmes

@markusthoemmes could you reproduce the issue and show outputs of 'ls -l /proc/PID/fd' for runsc processes?

Jan 04 '24 21:01 avagin

@avagin here we go. Sadly, as noted above, the "inner" container correctly stops and so there's no processes left that I could check there. All the python fds on the sandbox are part of the rootfs of the container image.

FWIW, both /run/containerd/io.containerd.runtime.v2.task/k8s.io/5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f/rootfs and /run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/rootfs are empty on ls -l

sandbox: 90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8 container: 5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f

$ ps aux | grep 5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f
# Already gone as noted above.

$ ps aux | grep 90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8
root     2392488  0.0  0.1 1261928 21736 ?       Ssl  Jan02   0:00 runsc-gofer --systemd-cgroup=true --directfs=false --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json --log-format=json --panic-log=/var/log/pods/<redacted>/gvisor_panic.log --platform=systrap --log-fd=3 gofer --bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8 --gofer-mount-confs=lisafs:none,lisafs:none --io-fds=6,7 --mounts-fd=5 --spec-fd=4 --sync-nvproxy-fd=-1 --sync-userns-fd=-1 --proc-mount-sync-fd=14 --apply-caps=false --setup-root=false
nobody   2392492  2.1  0.4 3547332 77984 ?       Ssl  Jan02  67:32 runsc-sandbox --platform=systrap --systemd-cgroup=true --directfs=false --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json --log-format=json --panic-log=/var/log/pods/<redacted>/gvisor_panic.log --log-fd=3 --panic-log-fd=4 boot --bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8 --controller-fd=10 --cpu-num=8 --dev-io-fd=-1 --gofer-mount-confs=lisafs:none,lisafs:none --io-fds=5,6 --mounts-fd=7 --setup-root=false --spec-fd=11 --start-sync-fd=8 --stdio-fds=12,13,14 --total-host-memory=16768503808 --total-memory=1073741824 --user-log-fd=9 --product-name=<redacted> --proc-mount-sync-fd=21 90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8
root     2392576  0.0  0.1 1253480 20360 ?       Sl   Jan02   0:00 runsc --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json --log-format=json --panic-log=/var/log/pods/<redacted>/gvisor_panic.log --systemd-cgroup=true --platform=systrap --directfs=false wait 90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8

$ ls -l /proc/2392488/fd 
lr-x------ 1 root root 64 Jan  3 18:23 0 -> /dev/null
l-wx------ 1 root root 64 Jan  3 18:23 1 -> /dev/null
lrwx------ 1 root root 64 Jan  3 18:23 10 -> 'anon_inode:[eventpoll]'
lr-x------ 1 root root 64 Jan  3 18:23 11 -> 'pipe:[1725863547]'
l-wx------ 1 root root 64 Jan  3 18:23 12 -> 'pipe:[1725864402]'
l-wx------ 1 root root 64 Jan  2 17:19 13 -> 'pipe:[1725863547]'
lrwx------ 1 root root 64 Jan  3 18:23 14 -> 'anon_inode:[eventfd]'
l-wx------ 1 root root 64 Jan  3 18:23 15 -> 'pipe:[1725863550]'
lr-x------ 1 root root 64 Jan  3 18:23 16 -> /
lrwx------ 1 root root 64 Jan  3 18:23 17 -> '/memfd:flipcall_packet_windows (deleted)'
lrwx------ 1 root root 64 Jan  3 18:23 18 -> '/memfd:flipcall_packet_windows (deleted)'
lr-x------ 1 root root 64 Jan  3 18:23 19 -> /root
l-wx------ 1 root root 64 Jan  3 18:23 2 -> /dev/null
lrwx------ 1 root root 64 Jan  3 18:23 20 -> 'socket:[1725866577]'
lrwx------ 1 root root 64 Jan  3 18:23 21 -> 'socket:[1725866579]'
lr-x------ 1 root root 64 Jan  3 18:23 22 -> /root/etc
lr-x------ 1 root root 64 Jan  3 18:23 23 -> /root/etc/resolv.conf
lr-x------ 1 root root 64 Jan  3 18:23 24 -> /root/etc/resolv.conf
lrwx------ 1 root root 64 Jan  3 18:23 25 -> 'socket:[1725863574]'
lrwx------ 1 root root 64 Jan  3 18:23 26 -> 'socket:[1725863579]'
lr-x------ 1 root root 64 Jan  3 18:23 27 -> /root/pause
lr-x------ 1 root root 64 Jan  3 18:23 28 -> /root/pause
l-wx------ 1 root root 64 Jan  3 18:23 3 -> /run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json
lr-x------ 1 root root 64 Jan  3 18:23 4 -> /run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/config.json
lrwx------ 1 root root 64 Jan  3 18:23 5 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Jan  3 18:23 6 -> 'socket:[1725867294]'
lrwx------ 1 root root 64 Jan  3 18:23 7 -> 'socket:[1725867296]'
lr-x------ 1 root root 64 Jan  3 18:23 8 -> 'pipe:[1725864402]'
lr-x------ 1 root root 64 Jan  3 18:23 9 -> 'pipe:[1725863550]'

$ ls -l /proc/2392492/fd
total 0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 0 -> /dev/null
l-wx------ 1 nobody nogroup 64 Jan  3 18:23 1 -> /dev/null
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 10 -> 'socket:[1725867313]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 100 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/dumper.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 101 -> /usr/local/lib/python3.8/site-packages/drf_yasg/__pycache__/openapi.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 102 -> /usr/local/lib/python3.8/site-packages/inflection/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 103 -> /usr/local/lib/python3.8/site-packages/_cffi_backend.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 104 -> /usr/local/lib/python3.8/site-packages/rest_framework/templatetags/__pycache__/rest_framework.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 105 -> /usr/local/lib/python3.8/site-packages/drf_yasg/inspectors/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 106 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/versioning.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 107 -> /usr/local/lib/python3.8/lib-dynload/_posixsubprocess.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 108 -> /usr/local/lib/python3.8/lib-dynload/select.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 109 -> /usr/local/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 11 -> '/memfd:systrap-memory (deleted)'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 110 -> /usr/local/lib/python3.8/site-packages/drf_yasg/inspectors/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 111 -> /usr/local/lib/python3.8/site-packages/drf_yasg/__pycache__/errors.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 112 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/exceptions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 113 -> /usr/local/lib/python3.8/site-packages/django/contrib/staticfiles/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 114 -> /usr/local/lib/python3.8/site-packages/drf_yasg/__pycache__/generators.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 115 -> /usr/local/lib/python3.8/lib-dynload/pyexpat.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 116 -> /usr/lib/libexpat.so.1.6.11
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 117 -> /usr/local/lib/python3.8/site-packages/packaging/__pycache__/version.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 118 -> /usr/local/lib/python3.8/site-packages/packaging/__pycache__/_structures.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 119 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/__init__.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 12 -> '/memfd:runsc-memory (deleted)'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 120 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/fields.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 121 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/connection.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 122 -> /usr/local/lib/python3.8/site-packages/dal_select2/__pycache__/fields.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 124 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/_collections.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 125 -> /usr/local/lib/python3.8/site-packages/drf_yasg/inspectors/__pycache__/query.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 126 -> /usr/local/lib/python3.8/site-packages/drf_yasg/inspectors/__pycache__/view.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 127 -> /usr/local/lib/python3.8/site-packages/django/contrib/contenttypes/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 128 -> /usr/local/lib/python3.8/site-packages/jet/__pycache__/urls.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 129 -> /usr/local/lib/python3.8/lib-dynload/zlib.cpython-38-x86_64-linux-gnu.so
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 13 -> 'anon_inode:[eventfd]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 130 -> /lib/libz.so.1.2.11
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 131 -> /usr/local/lib/python3.8/site-packages/coreapi/transports/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 132 -> /usr/local/lib/python3.8/site-packages/jet/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 133 -> /usr/local/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 134 -> /usr/lib/libbz2.so.1.0.8
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 135 -> /usr/local/lib/python3.8/site-packages/django/middleware/__pycache__/http.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 136 -> /usr/local/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 137 -> /usr/lib/liblzma.so.5.2.4
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 138 -> /usr/local/lib/python3.8/lib-dynload/grp.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 139 -> /usr/local/lib/python3.8/site-packages/django_extensions/templatetags/__pycache__/syntax_color.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  2 17:19 14 -> 'anon_inode:[eventfd]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 140 -> /usr/lib/libpoppler.so.92.0.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 141 -> /usr/lib/libjson-c.so.4.0.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 142 -> /usr/lib/libgeos_c.so.1.13.1
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 143 -> /usr/local/lib/python3.8/site-packages/coreapi/transports/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 144 -> /usr/lib/libodbcinst.so.2.0.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 145 -> /usr/lib/libtiff.so.5.5.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 147 -> /usr/local/lib/python3.8/lib-dynload/_opcode.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 148 -> /usr/lib/libxml2.so.2.9.12
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 149 -> /usr/lib/libstdc++.so.6.0.28
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 15 -> 'pipe:[1725867330]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 150 -> /usr/lib/libfontconfig.so.1.12.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 151 -> /usr/lib/libicuuc.so.64.2
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 152 -> /usr/local/lib/python3.8/site-packages/django/templatetags/__pycache__/cache.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 153 -> /usr/local/lib/python3.8/site-packages/django/templatetags/__pycache__/l10n.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 154 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/templatetags/__pycache__/jet_dashboard_tags.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 155 -> /usr/local/lib/python3.8/site-packages/pdfkit/__pycache__/api.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 156 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/urls.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 157 -> /usr/local/lib/python3.8/site-packages/uritemplate/__pycache__/api.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 158 -> /usr/local/lib/python3.8/site-packages/uritemplate/__pycache__/orderedset.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 159 -> /usr/local/lib/python3.8/site-packages/jet/templatetags/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 16 -> 'pipe:[1725867342]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 160 -> /usr/local/lib/python3.8/site-packages/jet/templatetags/__pycache__/jet_tags.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 161 -> /usr/local/lib/python3.8/site-packages/braces/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 162 -> /usr/local/lib/python3.8/site-packages/braces/views/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 163 -> /usr/local/lib/python3.8/site-packages/django/contrib/admin/templatetags/__pycache__/admin_list.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 164 -> /usr/local/lib/python3.8/lib-dynload/_socket.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 165 -> /usr/local/lib/python3.8/site-packages/uritemplate/__pycache__/template.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 166 -> /usr/local/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 167 -> /usr/local/lib/python3.8/site-packages/braces/views/__pycache__/_ajax.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 168 -> /usr/local/lib/python3.8/site-packages/django/contrib/admin/views/__pycache__/main.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 169 -> /usr/local/lib/python3.8/lib-dynload/_struct.cpython-38-x86_64-linux-gnu.so
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 17 -> 'anon_inode:[eventpoll]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 170 -> /usr/local/lib/python3.8/lib-dynload/binascii.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 171 -> /usr/local/lib/python3.8/site-packages/coreapi/__pycache__/client.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 172 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 173 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/core.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 174 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/util.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 175 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/preprocessors.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 176 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/inlinepatterns.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 177 -> /usr/local/lib/python3.8/site-packages/braces/views/__pycache__/_other.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 178 -> /usr/local/lib/python3.8/lib-dynload/_contextvars.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 179 -> /usr/local/lib/python3.8/site-packages/braces/views/__pycache__/_queries.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 18 -> 'pipe:[1725867337]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 180 -> /usr/local/lib/python3.8/lib-dynload/_asyncio.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 181 -> /usr/local/lib/python3.8/site-packages/django/contrib/admin/templatetags/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 182 -> /usr/local/lib/python3.8/site-packages/django/contrib/admin/templatetags/__pycache__/admin_modify.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 183 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/_internal_utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 184 -> /usr/local/lib/python3.8/site-packages/django/contrib/admin/templatetags/__pycache__/log.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 185 -> /usr/local/lib/python3.8/site-packages/django/core/cache/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 186 -> /usr/local/lib/python3.8/site-packages/django/templatetags/__pycache__/tz.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 187 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 188 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/templatetags/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 189 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/serializer_helpers.cpython-38.pyc
l-wx------ 1 nobody nogroup 64 Jan  3 18:23 19 -> 'pipe:[1725867330]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 190 -> /usr/local/lib/python3.8/site-packages/rangefilter/templatetags/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 191 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/status.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 192 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/json.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 193 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/fields.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 194 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/exceptions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 195 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/treeprocessors.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 196 -> /usr/lib/libgobject-2.0.so.0.6200.6
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 197 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/html.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 198 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/humanize_datetime.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 199 -> /usr/lib/libglib-2.0.so.0.6200.6
l-wx------ 1 nobody nogroup 64 Jan  3 18:23 2 -> /var/log/pods/<redacted>/gvisor_panic.log
l-wx------ 1 nobody nogroup 64 Jan  2 17:19 20 -> 'pipe:[1725867337]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 200 -> /usr/lib/libpango-1.0.so.0.4400.7
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 201 -> /usr/lib/libfribidi.so.0.4.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 202 -> /usr/local/lib/python3.8/lib-dynload/_queue.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 203 -> /usr/lib/libharfbuzz.so.0.20600.4
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 204 -> /usr/lib/libgraphite2.so.3.2.1
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 205 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/poolmanager.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 206 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/formatting.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 207 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/validators.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 208 -> /usr/local/lib/python3.8/lib-dynload/_bisect.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 209 -> /usr/local/lib/python3.8/lib-dynload/_sha512.cpython-38-x86_64-linux-gnu.so
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 21 -> 'anon_inode:[eventfd]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 210 -> /usr/local/lib/python3.8/lib-dynload/_random.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 211 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/field_mapping.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 212 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/relations.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 213 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/reverse.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 214 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/pagination.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 215 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/response.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 216 -> /usr/local/lib/python3.8/site-packages/jet/__pycache__/admin.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 217 -> /usr/local/lib/python3.8/site-packages/django/contrib/gis/admin/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 218 -> /usr/local/lib/python3.8/site-packages/django/contrib/gis/admin/__pycache__/options.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 219 -> /usr/local/lib/python3.8/site-packages/django/contrib/gis/admin/__pycache__/widgets.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 22 -> 'socket:[1725867313]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 220 -> /usr/local/lib/python3.8/site-packages/django/contrib/auth/__pycache__/admin.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 221 -> /usr/local/lib/python3.8/site-packages/django/contrib/auth/__pycache__/forms.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 222 -> /usr/local/lib/python3.8/site-packages/rangefilter/templatetags/__pycache__/rangefilter_compat.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 223 -> /usr/local/lib/python3.8/site-packages/django_extensions/templatetags/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 224 -> /usr/local/lib/python3.8/site-packages/django/contrib/contenttypes/__pycache__/admin.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 225 -> /usr/local/lib/python3.8/site-packages/django_extensions/templatetags/__pycache__/debugger_tags.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 226 -> /usr/local/lib/python3.8/site-packages/django_extensions/templatetags/__pycache__/highlighting.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 227 -> /usr/local/lib/python3.8/site-packages/django_extensions/templatetags/__pycache__/indent_text.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 228 -> /usr/local/lib/python3.8/site-packages/django_extensions/templatetags/__pycache__/widont.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 229 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/templatetags/__pycache__/__init__.cpython-38.pyc
l-wx------ 1 nobody nogroup 64 Jan  3 18:23 23 -> 'pipe:[1725867342]'
lr-x------ 1 nobody nogroup 64 Jan  3 21:43 232 -> /etc/passwd
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 237 -> /usr/local/lib/python3.8/site-packages/idna/__pycache__/intranges.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 238 -> /usr/local/lib/python3.8/lib-dynload/_hashlib.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 239 -> /usr/local/lib/python3.8/lib-dynload/_blake2.cpython-38-x86_64-linux-gnu.so
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 24 -> 'anon_inode:[eventfd]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 240 -> /usr/local/lib/python3.8/lib-dynload/_sha3.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 241 -> /usr/local/lib/python3.8/site-packages/idna/__pycache__/idnadata.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 242 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 243 -> /usr/local/lib/python3.8/lib-dynload/_pickle.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 244 -> /usr/local/lib/python3.8/site-packages/certifi/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 245 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/certs.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 249 -> /usr/local/lib/python3.8/lib-dynload/fcntl.cpython-38-x86_64-linux-gnu.so
lrwx------ 1 nobody nogroup 64 Jan  2 17:19 25 -> 'socket:[1725869537]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 256 -> /dev/null
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 257 -> /dev/null
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 258 -> /dev/null
lrwx------ 1 nobody nogroup 64 Jan  2 17:19 26 -> 'anon_inode:[eventfd]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 261 -> /usr/local/lib/python3.8/site-packages/django/contrib/contenttypes/__pycache__/fields.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 262 -> /usr/local/lib/python3.8/site-packages/django/contrib/contenttypes/__pycache__/forms.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 266 -> /usr/local/lib/python3.8/lib-dynload/_decimal.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 268 -> /usr/local/lib/python3.8/site-packages/django_extensions/admin/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 269 -> /usr/local/lib/python3.8/lib-dynload/termios.cpython-38-x86_64-linux-gnu.so
lrwx------ 1 nobody nogroup 64 Jan  2 17:19 27 -> 'anon_inode:[eventfd]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 270 -> /usr/local/lib/python3.8/site-packages/django_extensions/admin/__pycache__/widgets.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 271 -> /usr/local/lib/python3.8/site-packages/constance/__pycache__/admin.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 273 -> /usr/local/lib/python3.8/site-packages/django_celery_beat/__pycache__/admin.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 274 -> /usr/local/lib/python3.8/site-packages/django_celery_beat/__pycache__/signals.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 28 -> 'socket:[1725866578]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 280 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/__pycache__/toolbar.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 281 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 282 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 283 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/history/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 284 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/history/__pycache__/panel.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 285 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/history/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 286 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/__pycache__/decorators.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 287 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/history/__pycache__/forms.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 288 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/versions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 289 -> /usr/local/lib/python3.8/lib-dynload/_csv.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 29 -> /root/pause
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 293 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/timer.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 297 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/settings.cpython-38.pyc
l-wx------ 1 nobody nogroup 64 Jan  3 18:23 3 -> /run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 30 -> 'socket:[1725866580]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 304 -> /usr/local/lib/python3.8/site-packages/certifi/__pycache__/core.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 305 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/cookies.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 308 -> /usr/local/lib/python3.8/site-packages/backports/zoneinfo/_czoneinfo.cpython-38-x86_64-linux-gnu.so
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 31 -> 'socket:[1725863575]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 311 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/headers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 313 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/request.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 314 -> /usr/local/lib/python3.8/lib-dynload/_uuid.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 315 -> /lib/libuuid.so.1.3.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 316 -> /usr/local/lib/python3.8/lib-dynload/array.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 318 -> /usr/local/lib/python3.8/site-packages/fontTools/misc/bezierTools.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 319 -> /usr/local/lib/python3.8/site-packages/qrcode/__pycache__/main.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 32 -> 'socket:[1725863580]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 320 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/structures.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 321 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/api.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 322 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/sessions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 323 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/adapters.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 324 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/auth.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 325 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/models.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 326 -> /usr/local/lib/python3.8/encodings/idna.py
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 327 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/hooks.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 328 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/filepost.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 329 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/status_codes.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 33 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 330 -> /usr/local/lib/python3.8/site-packages/urllib3/contrib/__pycache__/socks.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 331 -> /usr/local/lib/python3.8/site-packages/qrcode/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 332 -> /usr/local/lib/python3.8/site-packages/idna/__pycache__/package_data.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 333 -> /usr/local/lib/python3.8/site-packages/qrcode/__pycache__/constants.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 334 -> /usr/local/lib/python3.8/site-packages/qrcode/__pycache__/exceptions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 335 -> /usr/local/lib/python3.8/site-packages/qrcode/__pycache__/util.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 336 -> /usr/local/lib/python3.8/site-packages/qrcode/__pycache__/LUT.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 337 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/url.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 338 -> /usr/local/lib/python3.8/lib-dynload/resource.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 339 -> /usr/local/lib/python3.8/site-packages/qrcode/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 34 -> /lib/libssl.so.1.1
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 340 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/__pycache__/api.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 341 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/connection.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 342 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/response.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 343 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/_request_methods.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 344 -> /usr/local/lib/python3.8/site-packages/qrcode/image/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 345 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/ssltransport.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 346 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/connectionpool.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 347 -> /usr/local/lib/python3.8/site-packages/qrcode/image/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 348 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/ssl_match_hostname.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 349 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/exceptions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 35 -> /lib/libcrypto.so.1.1
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 350 -> /usr/local/lib/python3.8/site-packages/qrcode/image/styles/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 351 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/retry.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 352 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/ssl_.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 353 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/compat.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 354 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/wait.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 355 -> /usr/local/lib/python3.8/site-packages/qrcode/image/styles/moduledrawers/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 356 -> /usr/local/lib/python3.8/site-packages/qrcode/image/styles/moduledrawers/__pycache__/pil.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 357 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/_version.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 358 -> /usr/local/lib/python3.8/site-packages/qrcode/compat/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 359 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/util.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 360 -> /usr/local/lib/python3.8/site-packages/qrcode/compat/__pycache__/pil.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 361 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/request.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 362 -> /usr/local/lib/python3.8/site-packages/PIL/__pycache__/ImageDraw.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 363 -> /usr/local/lib/python3.8/site-packages/qrcode/image/styles/moduledrawers/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 364 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/sql/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 365 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/sql/__pycache__/panel.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 366 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/__pycache__/forms.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 37 -> /usr/local/lib/python3.8/site-packages/rest_framework/schemas/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 370 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/sql/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 372 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/sql/__pycache__/forms.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 374 -> /usr/local/lib/python3.8/site-packages/django/contrib/sessions/__pycache__/serializers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 375 -> /usr/local/lib/python3.8/site-packages/django/contrib/messages/storage/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 376 -> /usr/local/lib/python3.8/lib-dynload/_json.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 377 -> /usr/local/lib/python3.8/site-packages/django/contrib/messages/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 387 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 388 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 389 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/timeout.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 39 -> /usr/local/lib/python3.8/site-packages/rest_framework/schemas/__pycache__/coreapi.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 390 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/response.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 391 -> /usr/local/lib/python3.8/site-packages/urllib3/util/__pycache__/proxy.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 392 -> /usr/local/lib/python3.8/lib-dynload/_elementtree.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 393 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/__pycache__/cd.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 394 -> /usr/local/lib/python3.8/site-packages/qrcode/image/__pycache__/pure.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 395 -> /usr/local/lib/python3.8/site-packages/PIL/_imaging.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 396 -> /usr/local/lib/python3.8/site-packages/Pillow.libs/libjpeg-6aa56261.so.62.3.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 397 -> /usr/local/lib/python3.8/site-packages/Pillow.libs/libopenjp2-91bd27bd.so.2.5.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 398 -> /usr/local/lib/python3.8/site-packages/Pillow.libs/libtiff-9c086da0.so.6.0.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 399 -> /usr/local/lib/python3.8/site-packages/Pillow.libs/libxcb-7e4cbcc5.so.1.1.0
l-wx------ 1 nobody nogroup 64 Jan  3 18:23 4 -> /var/log/pods/<redacted>/gvisor_panic.log
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 40 -> /usr/local/lib/python3.8/site-packages/corsheaders/__pycache__/signals.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 400 -> /usr/local/lib/python3.8/site-packages/__pycache__/png.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 401 -> /
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 402 -> /
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 403 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/serializers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 404 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/compat.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 405 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 406 -> /usr/local/lib/python3.8/site-packages/_brotli.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 407 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/fields/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 408 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/fields/__pycache__/array.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 409 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/__pycache__/lookups.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 41 -> /usr/local/lib/python3.8/site-packages/django/contrib/sessions/__pycache__/middleware.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 410 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/__pycache__/search.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 411 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/forms/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 412 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/forms/__pycache__/array.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 413 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/__pycache__/validators.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 414 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 415 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/forms/__pycache__/hstore.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 416 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/forms/__pycache__/jsonb.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 417 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/forms/__pycache__/ranges.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 418 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/fields/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 419 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/fields/__pycache__/citext.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 42 -> /usr/local/bin/python3.8
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 420 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/__pycache__/constant.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 421 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/fields/__pycache__/hstore.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 422 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/fields/__pycache__/jsonb.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 423 -> /usr/local/lib/python3.8/site-packages/django/contrib/postgres/fields/__pycache__/ranges.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 424 -> /usr/local/lib/python3.8/site-packages/coreapi/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 425 -> /usr/local/lib/python3.8/site-packages/coreapi/__pycache__/auth.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 426 -> /usr/local/lib/python3.8/site-packages/coreapi/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 427 -> /usr/local/lib/python3.8/site-packages/coreapi/__pycache__/exceptions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 428 -> /usr/share/zoneinfo/America/Argentina/Buenos_Aires
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 429 -> /usr/local/lib/python3.8/site-packages/coreapi/__pycache__/compat.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 43 -> /usr/local/lib/libpython3.8.so.1.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 430 -> /usr/local/lib/python3.8/site-packages/coreapi/codecs/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 431 -> /usr/local/lib/python3.8/site-packages/coreapi/codecs/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 432 -> /usr/local/lib/python3.8/site-packages/__pycache__/itypes.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 433 -> /usr/local/lib/python3.8/site-packages/coreapi/codecs/__pycache__/corejson.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 434 -> /usr/local/lib/python3.8/site-packages/coreapi/__pycache__/document.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 435 -> /usr/local/lib/python3.8/site-packages/coreschema/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 436 -> /usr/local/lib/python3.8/site-packages/coreschema/__pycache__/schemas.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 437 -> /usr/local/lib/python3.8/site-packages/coreschema/__pycache__/compat.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 438 -> /usr/local/lib/python3.8/site-packages/coreschema/__pycache__/formats.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 439 -> /usr/local/lib/python3.8/site-packages/coreschema/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 440 -> /usr/local/lib/python3.8/site-packages/coreschema/encodings/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 441 -> /usr/local/lib/python3.8/site-packages/coreschema/encodings/__pycache__/html.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 442 -> /usr/local/lib/python3.8/site-packages/coreapi/codecs/__pycache__/display.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 443 -> /usr/local/lib/python3.8/site-packages/coreapi/codecs/__pycache__/download.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 444 -> /usr/local/lib/python3.8/site-packages/coreapi/codecs/__pycache__/python.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 445 -> /usr/local/lib/python3.8/site-packages/coreapi/codecs/__pycache__/text.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 446 -> /usr/local/lib/python3.8/site-packages/coreapi/transports/__pycache__/http.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 447 -> /usr/local/lib/python3.8/site-packages/uritemplate/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 448 -> /usr/local/lib/python3.8/site-packages/uritemplate/__pycache__/variable.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 449 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/htmlparser.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 450 -> /usr/local/lib/python3.8/html/parser.py
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 451 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/blockprocessors.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 452 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/postprocessors.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 453 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/__meta__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 454 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/sql/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 455 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/sql/__pycache__/tracking.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 456 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/staticfiles.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 457 -> /usr/local/lib/python3.8/site-packages/django/contrib/staticfiles/__pycache__/storage.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 458 -> /usr/local/lib/python3.8/site-packages/django/contrib/messages/storage/__pycache__/cookie.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 459 -> /usr/local/lib/python3.8/site-packages/django/contrib/messages/storage/__pycache__/session.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 460 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/templates/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 461 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/cache.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 462 -> /usr/local/lib/python3.8/site-packages/rest_framework_simplejwt/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 463 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/generics.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 464 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/profiling.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 465 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/mixins.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 466 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/request.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 467 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/encoders.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 468 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/breadcrumbs.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 469 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/authentication.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 47 -> '/run/containerd/io.containerd.runtime.v2.task/k8s.io/5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f/rootfs/.gvisor.filestore.90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8 (deleted)'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 470 -> /usr/local/lib/python3.8/site-packages/rest_framework_simplejwt/__pycache__/authentication.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 471 -> /usr/local/lib/python3.8/site-packages/rest_framework_simplejwt/__pycache__/exceptions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 472 -> /usr/local/lib/python3.8/site-packages/django/contrib/messages/__pycache__/middleware.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 473 -> /usr/local/lib/python3.8/site-packages/django/contrib/auth/__pycache__/backends.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 474 -> /usr/local/lib/python3.8/lib-dynload/unicodedata.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 475 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/permissions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 476 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/parsers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 477 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/negotiation.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 478 -> /usr/local/lib/python3.8/site-packages/markupsafe/_speedups.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 479 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/mediatypes.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 480 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/metadata.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 481 -> /usr/local/lib/python3.8/site-packages/rest_framework_simplejwt/__pycache__/serializers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 482 -> /usr/local/lib/python3.8/site-packages/rest_framework_simplejwt/__pycache__/tokens.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 483 -> /usr/local/lib/python3.8/site-packages/rest_framework_simplejwt/token_blacklist/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 484 -> /usr/local/lib/python3.8/site-packages/rest_framework_simplejwt/token_blacklist/__pycache__/models.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 485 -> /usr/local/lib/python3.8/site-packages/drf_yasg/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 486 -> /usr/local/lib/python3.8/site-packages/django/views/decorators/__pycache__/vary.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 487 -> /usr/local/lib/python3.8/site-packages/drf_yasg/__pycache__/app_settings.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 488 -> /usr/local/lib/python3.8/site-packages/drf_yasg/__pycache__/renderers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 489 -> /usr/local/lib/python3.8/site-packages/drf_yasg/__pycache__/codecs.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 490 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 491 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/cyaml.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 492 -> /usr/local/lib/python3.8/site-packages/_ruamel_yaml.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 493 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/error.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 494 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/reader.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 495 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/util.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 496 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/compat.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 497 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/docinfo.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 498 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/scanner.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 499 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/tokens.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 5 -> 'socket:[1725867293]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 500 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/parser.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 501 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/events.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 502 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/tag.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 503 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/comments.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 504 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/scalarstring.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 505 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/anchor.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 506 -> /usr/local/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 507 -> /usr/lib/libpq.so.5.12
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 508 -> /usr/lib/libldap_r-2.4.so.2.10.11
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 509 -> /usr/lib/liblber-2.4.so.2.10.11
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 510 -> /usr/lib/libsasl2.so.3.0.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 511 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/composer.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 512 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/nodes.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 513 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/constructor.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 514 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/scalarint.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 515 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/main.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 516 -> /usr/local/lib/python3.8/site-packages/drf_yasg/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 517 -> /usr/local/lib/python3.8/site-packages/django/contrib/staticfiles/__pycache__/urls.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 518 -> /usr/local/lib/python3.8/site-packages/rest_framework/templatetags/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 519 -> /usr/local/lib/python3.8/site-packages/drf_yasg/inspectors/__pycache__/field.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 520 -> /usr/local/lib/python3.8/site-packages/packaging/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 521 -> /usr/local/lib/python3.8/site-packages/django/views/decorators/__pycache__/http.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 522 -> /usr/local/lib/python3.8/site-packages/jet/__pycache__/forms.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 523 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/__pycache__/urls.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 524 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/__pycache__/dashboard.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 525 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/__pycache__/modules.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 526 -> /usr/local/lib/python3.8/site-packages/jet/__pycache__/ordered_set.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 527 -> /usr/local/lib/python3.8/site-packages/django/template/__pycache__/context_processors.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 528 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/md.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 529 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/__pycache__/views.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 53 -> 'anon_inode:[eventpoll]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 530 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/__pycache__/forms.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 531 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 532 -> /usr/local/lib/python3.8/site-packages/jet/dashboard/__pycache__/settings.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 533 -> /usr/local/lib/python3.8/site-packages/pdfkit/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 534 -> /usr/local/lib/python3.8/site-packages/pdfkit/__pycache__/pdfkit.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 535 -> /usr/local/lib/python3.8/site-packages/pdfkit/__pycache__/source.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 536 -> /usr/local/lib/python3.8/site-packages/pdfkit/__pycache__/configuration.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 537 -> /usr/local/lib/python3.8/site-packages/django/contrib/auth/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 538 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/routers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 539 -> /usr/local/lib/python3.8/site-packages/rest_framework/schemas/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 540 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/urlpatterns.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 541 -> /usr/local/lib/python3.8/site-packages/braces/views/__pycache__/_access.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 542 -> /usr/lib/libffi.so.6.0.4
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 543 -> /usr/local/lib/python3.8/site-packages/braces/views/__pycache__/_forms.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 544 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/viewsets.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 545 -> /usr/local/lib/python3.8/site-packages/django/template/loaders/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 546 -> /usr/local/lib/python3.8/site-packages/django/template/loaders/__pycache__/filesystem.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 547 -> /usr/local/lib/python3.8/site-packages/django/template/loaders/__pycache__/app_directories.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 548 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/decorators.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 549 -> /usr/local/lib/python3.8/site-packages/django/template/loaders/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 552 -> /usr/local/lib/python3.8/site-packages/django/template/loaders/__pycache__/cached.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 554 -> /usr/local/lib/python3.8/site-packages/fontTools/varLib/iup.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 21:43 555 -> /bin/busybox
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 556 -> /usr/lib/libgdal.so.26.0.3
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 557 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/md__mypyc.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 558 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 559 -> /usr/local/lib/python3.8/lib-dynload/_multibytecodec.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 560 -> /usr/lib/libgcc_s.so.1
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 564 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/__pycache__/models.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 565 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/__pycache__/legacy.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 566 -> /usr/lib/libwebp.so.7.0.5
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 567 -> /usr/lib/libsqlite3.so.0.8.6
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 568 -> /usr/lib/libodbc.so.2.0.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 569 -> /usr/lib/libxerces-c-3.2.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 570 -> /usr/lib/libopenjp2.so.2.4.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 571 -> /usr/lib/libgif.so.7.2.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 572 -> /usr/lib/libjpeg.so.8.2.2
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 573 -> /usr/lib/libpng16.so.16.37.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 574 -> /usr/lib/libproj.so.15.2.1
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 575 -> /usr/lib/libpcre.so.1.2.11
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 576 -> /usr/lib/libcurl.so.4.7.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 577 -> /usr/lib/libmariadb.so.3
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 578 -> /usr/lib/libfreetype.so.6.17.1
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 579 -> /usr/lib/liblcms2.so.2.0.8
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 580 -> /usr/lib/libgeos-3.8.0.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 581 -> /usr/lib/libnghttp2.so.14.19.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 582 -> /usr/lib/libicudata.so.64.2
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 591 -> /usr/local/lib/python3.8/site-packages/charset_normalizer/__pycache__/version.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 592 -> /usr/local/lib/python3.8/http/cookiejar.py
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 593 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/packages.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 594 -> /usr/local/lib/python3.8/site-packages/idna/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 595 -> /usr/local/lib/python3.8/site-packages/idna/__pycache__/core.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 596 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 597 -> /usr/local/lib/python3.8/stringprep.py
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 598 -> /usr/local/lib/python3.8/site-packages/urllib3/contrib/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 599 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/blockparser.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 6 -> 'socket:[1725867295]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 600 -> /usr/local/lib/python3.8/site-packages/markdown/extensions/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 601 -> /usr/local/lib/python3.8/site-packages/markdown/__pycache__/serializers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 602 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/representation.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 603 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/model_meta.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 604 -> /usr/local/lib/python3.8/site-packages/rest_framework/utils/__pycache__/urls.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 605 -> /usr/local/lib/python3.8/site-packages/django/contrib/auth/__pycache__/tokens.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 606 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/templates/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 607 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/templates/__pycache__/panel.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 608 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/signals.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 609 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/logging.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 610 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/panels/__pycache__/redirects.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 611 -> /usr/local/lib/python3.8/cProfile.py
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 612 -> /usr/local/lib/python3.8/lib-dynload/_lsprof.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 613 -> /usr/local/lib/python3.8/profile.py
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 614 -> /usr/local/lib/python3.8/pstats.py
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 615 -> /usr/local/lib/python3.8/site-packages/debug_toolbar/__pycache__/middleware.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 616 -> /usr/local/lib/python3.8/site-packages/Pillow.libs/liblzma-78232b57.so.5.4.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 617 -> /usr/local/lib/python3.8/site-packages/Pillow.libs/libXau-52a9ca8d.so.6.0.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 618 -> /usr/local/lib/python3.8/site-packages/Pillow.libs/libXdmcp-4a845f28.so.6.0.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 619 -> /usr/local/lib/python3.8/site-packages/Pillow.libs/libbsd-ac21d37f.so.0.10.0
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 620 -> /usr/local/lib/python3.8/site-packages/django/middleware/__pycache__/clickjacking.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 663 -> /usr/local/lib/python3.8/site-packages/django/contrib/auth/__pycache__/middleware.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 664 -> /usr/local/lib/python3.8/site-packages/django/middleware/__pycache__/common.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 665 -> /usr/local/lib/python3.8/site-packages/corsheaders/__pycache__/middleware.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 67 -> /usr/local/lib/python3.8/site-packages/requests/__pycache__/__version__.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  3 18:23 7 -> '/memfd:memory-usage (deleted)'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 70 -> /usr/lib/libpangoft2-1.0.so.0.4400.7
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 71 -> /lib/ld-musl-x86_64.so.1
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 72 -> /usr/local/lib/python3.8/site-packages/django/contrib/sessions/backends/__pycache__/__init__.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 73 -> /usr/local/lib/python3.8/site-packages/django/contrib/sessions/backends/__pycache__/base.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 74 -> /usr/local/lib/python3.8/site-packages/django/contrib/sessions/__pycache__/exceptions.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 75 -> /usr/local/lib/python3.8/site-packages/django/contrib/sessions/backends/__pycache__/db.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 76 -> /usr/local/lib/python3.8/site-packages/django/middleware/__pycache__/security.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 77 -> /usr/local/lib/python3.8/site-packages/django/contrib/messages/storage/__pycache__/fallback.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 78 -> /usr/local/lib/python3.8/site-packages/rest_framework/schemas/__pycache__/generators.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 79 -> /usr/local/lib/python3.8/site-packages/django/contrib/admindocs/__pycache__/__init__.cpython-38.pyc
lrwx------ 1 nobody nogroup 64 Jan  2 17:19 8 -> 'socket:[1725867362]'
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 80 -> /usr/local/lib/python3.8/site-packages/django/contrib/admindocs/__pycache__/views.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 81 -> /usr/local/lib/python3.8/site-packages/django/contrib/admin/views/__pycache__/decorators.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 82 -> /usr/local/lib/python3.8/site-packages/django/contrib/admindocs/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 83 -> /usr/local/lib/python3.8/site-packages/rest_framework/schemas/__pycache__/inspectors.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 84 -> /usr/local/lib/python3.8/site-packages/rest_framework/schemas/__pycache__/utils.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 85 -> /usr/local/lib/python3.8/site-packages/rest_framework/schemas/__pycache__/openapi.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 86 -> /usr/local/lib/python3.8/site-packages/rest_framework/__pycache__/renderers.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 87 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/scalarfloat.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 88 -> /usr/local/lib/python3.8/site-packages/urllib3/__pycache__/_base_connection.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 89 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/scalarbool.cpython-38.pyc
l-wx------ 1 nobody nogroup 64 Jan  3 18:23 9 -> /var/log/pods/<redacted>/gvisor.log
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 90 -> /usr/local/lib/python3.8/lib-dynload/math.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 904 -> /usr/lib/libintl.so.8.1.6
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 91 -> /usr/local/lib/python3.8/lib-dynload/_datetime.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 92 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/timestamp.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 93 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/emitter.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 94 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/serializer.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 95 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/representer.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 96 -> /usr/local/lib/python3.8/site-packages/coreapi/codecs/__pycache__/jsondata.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 97 -> /usr/local/lib/python3.8/lib-dynload/_heapq.cpython-38-x86_64-linux-gnu.so
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 98 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/resolver.cpython-38.pyc
lr-x------ 1 nobody nogroup 64 Jan  3 18:23 99 -> /usr/local/lib/python3.8/site-packages/ruamel/yaml/__pycache__/loader.cpython-38.pyc

$ ls -l /proc/2392576/fd
total 0
lr-x------ 1 root root 64 Jan  4 22:22 0 -> /dev/null
l-wx------ 1 root root 64 Jan  4 22:22 1 -> 'pipe:[1725864422]'
lrwx------ 1 root root 64 Jan  4 22:22 10 -> 'socket:[1725869557]'
lrwx------ 1 root root 64 Jan  4 22:22 11 -> 'anon_inode:[eventfd]'
l-wx------ 1 root root 64 Jan  4 22:22 2 -> 'pipe:[1725864423]'
lr-x------ 1 root root 64 Jan  4 22:22 3 -> anon_inode:inotify
lr-x------ 1 root root 64 Jan  4 22:22 4 -> 'pipe:[1725869555]'
lrwx------ 1 root root 64 Jan  4 22:22 5 -> 'anon_inode:[eventpoll]'
lr-x------ 1 root root 64 Jan  4 22:22 6 -> 'pipe:[1725869547]'
l-wx------ 1 root root 64 Jan  4 22:22 7 -> 'pipe:[1725869547]'
l-wx------ 1 root root 64 Jan  4 22:22 8 -> 'pipe:[1725869555]'
l-wx------ 1 root root 64 Jan  4 22:22 9 -> /run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json

Jan 04 '24 22:01 markusthoemmes

Thanks for the debugging output. The sandbox process is holding FDs to the destroyed container's rootfs.

@avagin pointed out that the gofer client (pkg/sentry/fsimpl/gofer, which is part of the sandbox process) is not cleaning up the dentry tree when the filesystem is unmounted.

Historically the reason for this was that all file descriptors were owned by the gofer server (part of the gofer process), and when a client disconnected, it would clean up all file descriptors and resources for that client. So cleaning up the dentry cache tree was not required, so we avoided it. However, with directfs the gofer client owns the required file descriptors for each dentry. Hence, failure to clean up cached dentries in the tree is causing this issue. I can send a fix.

Jan 05 '24 15:01 ayushr2

To confirm this theory, could you configure runsc with --directfs=false and see if this issue reproduces?

Jan 05 '24 16:01 ayushr2

@ayushr2 this is running with directfs=false already.

Jan 05 '24 16:01 markusthoemmes

Ah, my bad I missed that.

Even with directfs=false, the gofer donates a host file descriptor for regular files to the sandbox (so that the sandbox can directly read/write to the FD without making RPCs).

I suspect that's what is going on, all the open FDs shown above seem to be of regular files.

Jan 05 '24 16:01 ayushr2

Can you check if this issue reproduces with --ref-leak-mode=panic (which should have the effect of forcing dentry cache cleanup)? Alternatively, you could also patch #9867 and try to repro.

Jan 05 '24 16:01 ayushr2

Sadly, I don't have a good reproducer for this so I can't easily confirm. I'll have to get a new version into the pipeline and let it sit for a bit to see if it happens again.

What doesn't quite check out to me (but might be totally down to my relative fresh-ness with gVisor in K8s) is that under the runsc-sandbox process, the actual processes of the container we've killed (i.e. the Python process) is still running and thus, intuitively, I feel like the FDs are correct to be there.

Does that still sound consistent with your theory?

Jan 05 '24 16:01 markusthoemmes

under the runsc-sandbox process, the actual processes of the container we've killed (i.e. the Python process) is still running

Sorry if I am re-iterating what has already been discussed above; how are you observing that the containerized python process is still running? AFAIK, all processes of all containers in a sandbox run in the sentry context. The sentry allocates a "Task Goroutine" for each application task. So observing these containerized processes from the host should make it look like the sandbox process is running the application code. The sentry does not create host sub-processes for application processes.

So how do you observe that a certain python process in the to-be-deleted container is still running from the host? Or are you using application logs to confirm?

intuitively, I feel like the FDs are correct to be there.

The sentry imposes a "dentry cache", hence the sentry can be holding the FDs, even if the python application has closed it. To confirm if the python process is really behind holding these FDs, you can run ls -l /proc/{python-pid}/fd from within the said container to see what files it really has open. I doubt that the python application will be concurrently holding all these *.pyc files. This looks more like the doing of our dentry cache (which can hold up to 1000 leaf dentries).

Jan 05 '24 17:01 ayushr2

Here's some more info on the container above. Note how the container is in STOPPED state (and it's runsc and runsc-gofer processes are gone) but the sandbox with all the subprocesses is still running. Anecdotally, not sure if this helps narrowing this down or is something else entirely, I've also seen hangs where the container's runsc and runsc-gofer processes where not gone and containerd wasn't hanging umounting but it was hanging sending KILL signals but the processes not budging, apparently.

The mix of these is what lead me to pick the title of the issue 😅

Sandbox: 90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8 Container: 5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f

$ ctr -n k8s.io t ls | grep 5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f
5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f    2392492    STOPPED

ctr -n k8s.io t ls | grep 90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8
90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8    2392492    RUNNING

root     2392474  0.0  0.1 1244020 17396 ?       Sl   Jan02   0:12 /usr/bin/containerd-shim-runsc-v1 -namespace k8s.io -address /run/containerd/containerd.sock -publish-binary /usr/bin/containerd
root     2392488  0.0  0.1 1261928 21736 ?       Ssl  Jan02   0:00  \_ runsc-gofer --systemd-cgroup=true --directfs=false --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json --log-format=json --panic-log=/var/log/pods/<redacted>/gvisor_panic.log --platform=systrap --log-fd=3 gofer --bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8 --gofer-mount-confs=lisafs:none,lisafs:none --io-fds=6,7 --mounts-fd=5 --spec-fd=4 --sync-nvproxy-fd=-1 --sync-userns-fd=-1 --proc-mount-sync-fd=14 --apply-caps=false --setup-root=false
nobody   2392492  2.1  0.4 3547332 77984 ?       Ssl  Jan02  90:55  \_ runsc-sandbox --platform=systrap --systemd-cgroup=true --directfs=false --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json --log-format=json --panic-log=/var/log/pods/<redacted>/gvisor_panic.log --log-fd=3 --panic-log-fd=4 boot --bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8 --controller-fd=10 --cpu-num=8 --dev-io-fd=-1 --gofer-mount-confs=lisafs:none,lisafs:none --io-fds=5,6 --mounts-fd=7 --setup-root=false --spec-fd=11 --start-sync-fd=8 --stdio-fds=12,13,14 --total-host-memory=16768503808 --total-memory=1073741824 --user-log-fd=9 --product-name=<redacted> --proc-mount-sync-fd=21 90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8
nobody   2392527  0.0  0.0  16520    56 ?        Ss   Jan02   0:00  |   \_ [exe]
nobody   2392528  0.0  0.0  16520    56 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2392571  0.0  0.0  18316   140 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2392572  0.0  0.0  18316   140 ?        SN   Jan02   0:00  |       |   \_ [exe]
nobody   2393703  0.0  0.0  16520   104 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2393704  0.0  0.0  16520   104 ?        SN   Jan02   0:13  |       |   \_ [exe]
nobody   2393714  0.0  0.0  16520    60 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2393715  0.0  0.0  16520    60 ?        SN   Jan02   0:46  |       |   \_ [exe]
nobody   2393716  0.0  0.0  16520    76 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2393717  0.0  0.0  16520    76 ?        SN   Jan02   0:14  |       |   \_ [exe]
nobody   2393832  0.0  0.0  16520    60 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2393834  0.0  0.0  16520    60 ?        SN   Jan02   0:00  |       |   \_ [exe]
nobody   2394049  0.0  0.0  16520    60 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2394050  0.0  0.0  16520    60 ?        SN   Jan02   0:00  |       |   \_ [exe]
nobody   2394183  0.0  0.0  16520    60 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2394184  0.0  0.0  16520    60 ?        SN   Jan02   0:00  |       |   \_ [exe]
nobody   2394185  0.0  0.0  16520    60 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2394186  0.0  0.0  16520    60 ?        SN   Jan02   0:00  |       |   \_ [exe]
nobody   2394187  0.0  0.0  16520    60 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2394188  0.0  0.0  16520    60 ?        SN   Jan02   0:00  |       |   \_ [exe]
nobody   2394189  0.0  0.0  16520    60 ?        S    Jan02   0:00  |       \_ [exe]
nobody   2394190  0.0  0.0  16520    60 ?        SN   Jan02   0:00  |           \_ [exe]
root     2392576  0.0  0.1 1253480 20360 ?       Sl   Jan02   0:00  \_ runsc --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/log.json --log-format=json --panic-log=/var/log/pods/<redacted>/gvisor_panic.log --systemd-cgroup=true --platform=systrap --directfs=false wait 90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8

Jan 05 '24 17:01 markusthoemmes

To not clutter things too much, here's the state of the "anecdotal" container I was talking about above: https://gist.github.com/markusthoemmes/7ebf064b44b1a182f552fbe1cc1b9150. I've tried to gather as much info as I could think of on that one as well.

Jan 05 '24 17:01 markusthoemmes

but the sandbox with all the subprocesses is still running

Just to clarify, the [exe] subprocesses you see under runsc-sandbox are not application processes. They are systrap stub processes. The application processes do not run as separate host processes, they run in the sandbox process context. As mentioned earlier, the sandbox allocates a goroutine for each application task which keeps running until something interesting happens (application makes a syscall, gets interrupted, page faults, etc) and the control is transferred to the sentry which handles the event and transfers the control back to the application task goroutine to continue executing from the interrupted IP.

@avagin and I had a chat, some updates:

--ref-leak-mode=panic may lead to some random unrelated panics due to some ref leaks that Andrei is fixing.
This is likely not a dentry cache issue (as I hypothesized earlier). The said dentry cache issue only exists with directfs. When directfs is turned off, all the donated FDs are released here (without destroying the dentry tree). So --directfs=false should not be leaking FDs via the dentry cache.
We suspect that the filesystem Release() itself is not being called due to some leaking reference on the filesystem instance. Given that this bug is not consistently reproducible, we suspect that this leak is due to some race condition.
We should add some logs around filesystem release / mount namespace destruction to confirm if filesystem is being released or not.

Jan 05 '24 18:01 ayushr2

Thanks @ayushr2 and @avagin for taking a look, it's greatly appreciated! Would the new hypothesis also explain the pod stuck like mentioned in https://github.com/google/gvisor/issues/9834#issuecomment-1879040410? It seems like we're having both of these symptoms at the same time so they might be correlated or might be completely different 😅 .

Jan 05 '24 21:01 markusthoemmes

I've looked at comparing differences of the shutdown behavior of "normally behaving" pods and hanging pods today to poke at this some more. Interestingly, the line that's missing in the hanging behavior is "shim disconnected" and sure enough, there's one leaked shim on this respective node! Does this go in favor of the cause you've been speculating above or does this potentially point at a containerd issue?

EDIT: Killing the shim in this case does not unwedge the situation.

normal pod

Jan 10 13:25:46 time="2024-01-10T13:25:46.990027836Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297 pid=76821
Jan 10 13:25:47 time="2024-01-10T13:25:47.336796129Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:web-6467d599b5-bhfhg,Uid:3fbebc13-0b01-4c12-8572-27e50732610c,Namespace:app-c144ed07-7744-41b0-a252-7ecbfee6c762,Attempt:0,} returns sandbox id \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\""
Jan 10 13:25:47 time="2024-01-10T13:25:47.343437719Z" level=info msg="CreateContainer within sandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" for container &ContainerMetadata{Name:tcp-sack-disable,Attempt:0,}"
Jan 10 13:25:47 time="2024-01-10T13:25:47.443995134Z" level=info msg="CreateContainer within sandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" for &ContainerMetadata{Name:tcp-sack-disable,Attempt:0,} returns container id \"2935d81e5623df612628e910824ce6221bb6fd12e0a44d76ff8691fa41e6329d\""
Jan 10 13:25:48 time="2024-01-10T13:25:48.328420249Z" level=info msg="CreateContainer within sandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" for container &ContainerMetadata{Name:web,Attempt:0,}"
Jan 10 13:25:48 time="2024-01-10T13:25:48.416263536Z" level=info msg="CreateContainer within sandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" for &ContainerMetadata{Name:web,Attempt:0,} returns container id \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\""
Jan 10 13:25:48 time="2024-01-10T13:25:48.417194448Z" level=info msg="StartContainer for \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\""
Jan 10 13:25:48 time="2024-01-10T13:25:48.486346165Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485 pid=77015
Jan 10 13:25:48 time="2024-01-10T13:25:48.734819784Z" level=info msg="StartContainer for \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\" returns successfully"
Jan 10 13:26:07 time="2024-01-10T13:26:07.727660023Z" level=info msg="StopContainer for \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\" with timeout 30 (s)"
Jan 10 13:26:07 time="2024-01-10T13:26:07.754940615Z" level=info msg="Stop container \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\" with signal terminated"
Jan 10 13:26:08 time="2024-01-10T13:26:08.074980044Z" level=info msg="shim disconnected" id=20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485 namespace=k8s.io
Jan 10 13:26:08 time="2024-01-10T13:26:08.075087976Z" level=warning msg="cleaning up after shim disconnected" id=20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485 namespace=k8s.io
Jan 10 13:26:08 time="2024-01-10T13:26:08.132746796Z" level=info msg="StopContainer for \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\" returns successfully"
Jan 10 13:26:08 time="2024-01-10T13:26:08.133860348Z" level=info msg="StopPodSandbox for \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\""
Jan 10 13:26:08 time="2024-01-10T13:26:08.134509591Z" level=info msg="Container to stop \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\" must be in running or unknown state, current state \"CONTAINER_EXITED\""
Jan 10 13:26:08 time="2024-01-10T13:26:08.385464748Z" level=warning msg="Ignoring error killing container \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\": \"runsc\" did not terminate successfully: sandbox is not running\n"
Jan 10 13:26:08 time="2024-01-10T13:26:08.429921787Z" level=warning msg="Ignoring error killing container \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\": \"runsc\" did not terminate successfully: sandbox is not running\n"
Jan 10 13:26:08 time="2024-01-10T13:26:08.491732736Z" level=info msg="shim disconnected" id=2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297 namespace=k8s.io
Jan 10 13:26:08 time="2024-01-10T13:26:08.491844148Z" level=warning msg="cleaning up after shim disconnected" id=2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297 namespace=k8s.io
Jan 10 13:26:08 time="2024-01-10T13:26:08.716084407Z" level=info msg="TearDown network for sandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" successfully"
Jan 10 13:26:08 time="2024-01-10T13:26:08.716163793Z" level=info msg="StopPodSandbox for \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" returns successfully"
Jan 10 13:26:09 time="2024-01-10T13:26:09.493819752Z" level=info msg="RemoveContainer for \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\""
Jan 10 13:26:09 time="2024-01-10T13:26:09.507483329Z" level=info msg="RemoveContainer for \"20c6f962ad6bfd62eea81aa2536845631260d37ea2e999e21af0c00813a96485\" returns successfully"
Jan 10 13:26:35 time="2024-01-10T13:26:35.902586939Z" level=info msg="StopPodSandbox for \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\""
Jan 10 13:26:35 time="2024-01-10T13:26:35.985525705Z" level=info msg="TearDown network for sandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" successfully"
Jan 10 13:26:35 time="2024-01-10T13:26:35.985689080Z" level=info msg="StopPodSandbox for \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" returns successfully"
Jan 10 13:26:35 time="2024-01-10T13:26:35.986557729Z" level=info msg="RemovePodSandbox for \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\""
Jan 10 13:26:35 time="2024-01-10T13:26:35.986805068Z" level=info msg="Forcibly stopping sandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\""
Jan 10 13:26:36 time="2024-01-10T13:26:36.083872934Z" level=info msg="TearDown network for sandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" successfully"
Jan 10 13:26:36 time="2024-01-10T13:26:36.095668006Z" level=warning msg="Failed to get podSandbox status for container event for sandboxID \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\": an error occurred when try to find sandbox: not found. Sending the event with nil podSandboxStatus."
Jan 10 13:26:36 time="2024-01-10T13:26:36.095893143Z" level=info msg="RemovePodSandbox \"2b3441530df5090a30594459102fdaacd11c23030f10de327e3286fbe2bc9297\" returns successfully"

hanging pod

Dec 16 22:06:31 time="2023-12-16T22:06:31.694963790Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/1a2642c3a4ffb0f8e24d51979e1ddc8ebb6109d8c48b952242ec86583c8bd99e pid=3673804
Dec 16 22:06:32 time="2023-12-16T22:06:32.471636062Z" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:<redacted>-785b769988-dxxz5,Uid:aaea1146-81d8-4be2-a607-5eaf664f5d23,Namespace:app-b0b43bde-860d-404b-81ad-91c8048ad2d4,Attempt:0,} returns sandbox id \"1a2642c3a4ffb0f8e24d51979e1ddc8ebb6109d8c48b952242ec86583c8bd99e\""
Dec 16 22:06:46 time="2023-12-16T22:06:46.326258542Z" level=info msg="CreateContainer within sandbox \"1a2642c3a4ffb0f8e24d51979e1ddc8ebb6109d8c48b952242ec86583c8bd99e\" for container &ContainerMetadata{Name:<redacted>,Attempt:0,}"
Dec 16 22:06:46 time="2023-12-16T22:06:46.382285753Z" level=info msg="CreateContainer within sandbox \"1a2642c3a4ffb0f8e24d51979e1ddc8ebb6109d8c48b952242ec86583c8bd99e\" for &ContainerMetadata{Name:<redacted>,Attempt:0,} returns container id \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\""
Dec 16 22:06:46 time="2023-12-16T22:06:46.383382537Z" level=info msg="StartContainer for \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\""
Dec 16 22:06:46 time="2023-12-16T22:06:46.450940918Z" level=info msg="starting signal loop" namespace=k8s.io path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654 pid=3674058
Dec 16 22:06:46 time="2023-12-16T22:06:46.769695454Z" level=info msg="StartContainer for \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" returns successfully"
Jan 05 17:32:52 time="2024-01-05T17:32:52.252409222Z" level=info msg="StopContainer for \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" with timeout 30 (s)"
Jan 05 17:32:52 time="2024-01-05T17:32:52.295471813Z" level=info msg="Stop container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" with signal terminated"
Jan 05 17:32:55 time="2024-01-05T17:32:55.160512115Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:32:55 time="2024-01-05T17:32:55.160856356Z" level=error msg="failed to handle container TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}" error="failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:32:56 time="2024-01-05T17:32:56.729668286Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:32:56 time="2024-01-05T17:32:56.770274706Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:32:59 time="2024-01-05T17:32:59.310886184Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:32:59 time="2024-01-05T17:32:59.311230290Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:33:01 time="2024-01-05T17:33:01.730574024Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:33:01 time="2024-01-05T17:33:01.774301890Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:33:04 time="2024-01-05T17:33:04.332275380Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:33:04 time="2024-01-05T17:33:04.332605875Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:33:08 time="2024-01-05T17:33:08.730629387Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:33:08 time="2024-01-05T17:33:08.766274282Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:33:11 time="2024-01-05T17:33:11.299057549Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:33:11 time="2024-01-05T17:33:11.299326517Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:33:19 time="2024-01-05T17:33:19.729993251Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:33:19 time="2024-01-05T17:33:19.757108867Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:33:22 time="2024-01-05T17:33:22.299294820Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:33:22 time="2024-01-05T17:33:22.299593708Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:33:22 time="2024-01-05T17:33:22.378464616Z" level=info msg="Kill container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\""
Jan 05 17:33:38 time="2024-01-05T17:33:38.730174869Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:33:38 time="2024-01-05T17:33:38.773274415Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:33:41 time="2024-01-05T17:33:41.323214832Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:33:41 time="2024-01-05T17:33:41.323537227Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:34:13 time="2024-01-05T17:34:13.730166629Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:34:13 time="2024-01-05T17:34:13.785287279Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:34:16 time="2024-01-05T17:34:16.343265907Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:34:16 time="2024-01-05T17:34:16.343540874Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:35:20 time="2024-01-05T17:35:20.730572316Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:35:20 time="2024-01-05T17:35:20.766965653Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:35:23 time="2024-01-05T17:35:23.311337226Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:35:23 time="2024-01-05T17:35:23.311654494Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:37:31 time="2024-01-05T17:37:31.730522120Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:37:31 time="2024-01-05T17:37:31.765935172Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:37:34 time="2024-01-05T17:37:34.307214039Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:37:34 time="2024-01-05T17:37:34.307557371Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:41:50 time="2024-01-05T17:41:50.730533261Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:41:50 time="2024-01-05T17:41:50.777031189Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:41:53 time="2024-01-05T17:41:53.323929164Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:41:53 time="2024-01-05T17:41:53.324245891Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:46:53 time="2024-01-05T17:46:53.729665501Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:46:53 time="2024-01-05T17:46:53.761647256Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:46:56 time="2024-01-05T17:46:56.310856527Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:46:56 time="2024-01-05T17:46:56.311164365Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:48:22 time="2024-01-05T17:48:22.252997700Z" level=error msg="StopContainer for \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" to be killed: wait container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": context canceled"
Jan 05 17:48:22 time="2024-01-05T17:48:22.253433053Z" level=info msg="StopPodSandbox for \"1a2642c3a4ffb0f8e24d51979e1ddc8ebb6109d8c48b952242ec86583c8bd99e\""
Jan 05 17:48:22 time="2024-01-05T17:48:22.253919924Z" level=info msg="Kill container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\""
Jan 05 17:51:56 time="2024-01-05T17:51:56.730445189Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:51:56 time="2024-01-05T17:51:56.762112322Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:51:59 time="2024-01-05T17:51:59.311656664Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:51:59 time="2024-01-05T17:51:59.311992214Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 17:56:59 time="2024-01-05T17:56:59.729840744Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 17:56:59 time="2024-01-05T17:56:59.770248899Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 17:57:02 time="2024-01-05T17:57:02.322940190Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 17:57:02 time="2024-01-05T17:57:02.323243004Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 18:02:02 time="2024-01-05T18:02:02.733415051Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 18:02:02 time="2024-01-05T18:02:02.769677406Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 18:02:05 time="2024-01-05T18:02:05.306154539Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 18:02:05 time="2024-01-05T18:02:05.306455341Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 18:03:22 time="2024-01-05T18:03:22.253450510Z" level=error msg="StopPodSandbox for \"1a2642c3a4ffb0f8e24d51979e1ddc8ebb6109d8c48b952242ec86583c8bd99e\" failed" error="rpc error: code = DeadlineExceeded desc = failed to stop container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": an error occurs during waiting for container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" to be killed: wait container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": context deadline exceeded"
Jan 05 18:03:22 time="2024-01-05T18:03:22.421075937Z" level=info msg="StopContainer for \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" with timeout 30 (s)"
Jan 05 18:03:22 time="2024-01-05T18:03:22.421568699Z" level=info msg="Skipping the sending of signal terminated to container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" because a prior stop with timeout>0 request already sent the signal"
Jan 05 18:03:52 time="2024-01-05T18:03:52.422075472Z" level=info msg="Kill container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\""
Jan 05 18:07:05 time="2024-01-05T18:07:05.729853592Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 18:07:05 time="2024-01-05T18:07:05.765661493Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 18:07:08 time="2024-01-05T18:07:08.309677808Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 18:07:08 time="2024-01-05T18:07:08.310468093Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 18:12:08 time="2024-01-05T18:12:08.729708388Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 18:12:08 time="2024-01-05T18:12:08.765210416Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 18:12:11 time="2024-01-05T18:12:11.307836905Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 18:12:11 time="2024-01-05T18:12:11.308110309Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 18:17:11 time="2024-01-05T18:17:11.730439246Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 18:17:11 time="2024-01-05T18:17:11.761119608Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 18:17:14 time="2024-01-05T18:17:14.314829398Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 18:17:14 time="2024-01-05T18:17:14.315176288Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 18:18:52 time="2024-01-05T18:18:52.421382140Z" level=error msg="StopContainer for \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" failed" error="rpc error: code = Canceled desc = an error occurs during waiting for container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" to be killed: wait container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": context canceled"
Jan 05 18:18:52 time="2024-01-05T18:18:52.421634607Z" level=info msg="StopPodSandbox for \"1a2642c3a4ffb0f8e24d51979e1ddc8ebb6109d8c48b952242ec86583c8bd99e\""
Jan 05 18:18:52 time="2024-01-05T18:18:52.422122527Z" level=info msg="Kill container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\""
Jan 05 18:22:14 time="2024-01-05T18:22:14.729702990Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 18:22:14 time="2024-01-05T18:22:14.771433591Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 18:22:17 time="2024-01-05T18:22:17.304346336Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 18:22:17 time="2024-01-05T18:22:17.304887376Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 18:27:17 time="2024-01-05T18:27:17.734263260Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 18:27:17 time="2024-01-05T18:27:17.769513050Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 18:27:20 time="2024-01-05T18:27:20.306682270Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 18:27:20 time="2024-01-05T18:27:20.307404351Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 18:32:20 time="2024-01-05T18:32:20.730165337Z" level=info msg="TaskExit event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248}"
Jan 05 18:32:20 time="2024-01-05T18:32:20.785354426Z" level=warning msg="Ignoring error killing container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": \"runsc\" did not terminate successfully: loading container: file does not exist\n"
Jan 05 18:32:23 time="2024-01-05T18:32:23.344511776Z" level=warning msg="failed to cleanup rootfs mount" error="failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy" runtime=io.containerd.runsc.v1
Jan 05 18:32:23 time="2024-01-05T18:32:23.344861924Z" level=error msg="Failed to handle backOff event container_id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" id:\"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" pid:3673819 exit_status:143 exited_at:{seconds:1704475972 nanos:453458248} for b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654" error="failed to handle container TaskExit event: failed to stop container: failed to delete task: failed rootfs umount: failed to unmount target /run/containerd/io.containerd.runtime.v2.task/k8s.io/b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654/rootfs: device or resource busy: unknown"
Jan 05 18:33:52 time="2024-01-05T18:33:52.421774753Z" level=error msg="StopPodSandbox for \"1a2642c3a4ffb0f8e24d51979e1ddc8ebb6109d8c48b952242ec86583c8bd99e\" failed" error="rpc error: code = DeadlineExceeded desc = failed to stop container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": an error occurs during waiting for container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" to be killed: wait container \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\": context deadline exceeded"
Jan 05 18:33:52 time="2024-01-05T18:33:52.806993016Z" level=info msg="StopContainer for \"b8206ab1e1639d030523c7057b398796cc6b9e4c16ba514157fd8fa228160654\" with timeout 30 (s)"
...

Jan 10 '24 16:01 markusthoemmes

Aaaaand one more datapoint in my quest to find the nugget of info that I'm looking for: The goroutine dump of the "leaked" shim:

goroutine dump of leaked shim

SIGQUIT: quit
PC=0x46df01 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex()
        src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fff6bfe6770 sp=0x7fff6bfe6768 pc=0x46df01
runtime.futexsleep(0x7fff6bfe67e8?, 0x440396?, 0x7fff6bfe67e8?)
        GOROOT/src/runtime/os_linux.go:69 +0x30 fp=0x7fff6bfe67c0 sp=0x7fff6bfe6770 pc=0x4355d0
runtime.notesleep(0x158b9c8)
        GOROOT/src/runtime/lock_futex.go:160 +0x87 fp=0x7fff6bfe67f8 sp=0x7fff6bfe67c0 pc=0x40ea47
runtime.mPark(...)
        GOROOT/src/runtime/proc.go:1632
runtime.stoplockedm()
        GOROOT/src/runtime/proc.go:2780 +0x73 fp=0x7fff6bfe6850 sp=0x7fff6bfe67f8 pc=0x440573
runtime.schedule()
        GOROOT/src/runtime/proc.go:3561 +0x3a fp=0x7fff6bfe6888 sp=0x7fff6bfe6850 pc=0x4423ba
runtime.park_m(0xc000007d40?)
        GOROOT/src/runtime/proc.go:3745 +0x11f fp=0x7fff6bfe68d0 sp=0x7fff6bfe6888 pc=0x44293f
runtime.mcall()
        src/runtime/asm_amd64.s:458 +0x4e fp=0x7fff6bfe68e8 sp=0x7fff6bfe68d0 pc=0x46a26e

goroutine 1 [select, 2 minutes]:
runtime.gopark(0xc00006f678?, 0x2?, 0x1?, 0x0?, 0xc00006f61c?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00006f4b8 sp=0xc00006f498 pc=0x43bf4e
runtime.selectgo(0xc00006f678, 0xc00006f618, 0x0?, 0x0, 0x2?, 0x1)
        GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00006f5d8 sp=0xc00006f4b8 pc=0x44bac5
github.com/containerd/containerd/runtime/v2/shim.handleSignals({0xed8d40, 0xc00008aa00}, 0xdb4926?, 0xc00016c600)
        external/com_github_containerd_containerd/runtime/v2/shim/shim_unix.go:77 +0xf8 fp=0xc00006f7e8 sp=0xc00006f5d8 pc=0x724718
github.com/containerd/containerd/runtime/v2/shim.(*Client).Serve(0xc00006fbb8)
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:298 +0x356 fp=0xc00006f9b8 sp=0xc00006f7e8 pc=0x723d96
github.com/containerd/containerd/runtime/v2/shim.run({0xdc00db, 0x16}, 0xdeb800, {0xf0?, 0xc6?, 0x5?})
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:237 +0xb2d fp=0xc00006fe70 sp=0xc00006f9b8 pc=0x72360d
github.com/containerd/containerd/runtime/v2/shim.Run({0xdc00db, 0x16}, 0xc0000400b8?, {0x0, 0x0, 0xc0000061a0?})
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:153 +0xab fp=0xc00006ff00 sp=0xc00006fe70 pc=0x7229eb
gvisor.dev/gvisor/shim/cli.Main(...)
        shim/cli/cli.go:27
main.main()
        shim/main.go:23 +0x2e fp=0xc00006ff40 sp=0xc00006ff00 pc=0xba34ee
runtime.main()
        GOROOT/src/runtime/proc.go:267 +0x2bb fp=0xc00006ffe0 sp=0xc00006ff40 pc=0x43bafb
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x46c0e1

goroutine 2 [force gc (idle), 7 minutes]:
runtime.gopark(0x8f49138fbd4de?, 0x0?, 0x0?, 0x0?, 0x0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00005cfa8 sp=0xc00005cf88 pc=0x43bf4e
runtime.goparkunlock(...)
        GOROOT/src/runtime/proc.go:404
runtime.forcegchelper()
        GOROOT/src/runtime/proc.go:322 +0xb3 fp=0xc00005cfe0 sp=0xc00005cfa8 pc=0x43bdd3
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46c0e1
created by runtime.init.6 in goroutine 1
        GOROOT/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00005d778 sp=0xc00005d758 pc=0x43bf4e
runtime.goparkunlock(...)
        GOROOT/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
        GOROOT/src/runtime/mgcsweep.go:321 +0xdf fp=0xc00005d7c8 sp=0xc00005d778 pc=0x427e5f
runtime.gcenable.func1()
        GOROOT/src/runtime/mgc.go:200 +0x25 fp=0xc00005d7e0 sp=0xc00005d7c8 pc=0x41cf05
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005d7e8 sp=0xc00005d7e0 pc=0x46c0e1
created by runtime.gcenable in goroutine 1
        GOROOT/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x158adc0?, 0xec7500?, 0x0?, 0x0?, 0x0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00005df70 sp=0xc00005df50 pc=0x43bf4e
runtime.goparkunlock(...)
        GOROOT/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x158adc0)
        GOROOT/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00005dfa0 sp=0xc00005df70 pc=0x4256a9
runtime.bgscavenge(0x0?)
        GOROOT/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00005dfc8 sp=0xc00005dfa0 pc=0x425c59
runtime.gcenable.func2()
        GOROOT/src/runtime/mgc.go:201 +0x25 fp=0xc00005dfe0 sp=0xc00005dfc8 pc=0x41cea5
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x46c0e1
created by runtime.gcenable in goroutine 1
        GOROOT/src/runtime/mgc.go:201 +0xa5

goroutine 5 [finalizer wait, 1 minutes]:
runtime.gopark(0x0?, 0xdeb990?, 0x0?, 0xc0?, 0x2000000020?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc0001aee28 sp=0xc0001aee08 pc=0x43bf4e
runtime.runfinq()
        GOROOT/src/runtime/mfinal.go:193 +0x107 fp=0xc0001aefe0 sp=0xc0001aee28 pc=0x41bf07
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001aefe8 sp=0xc0001aefe0 pc=0x46c0e1
created by runtime.createfing in goroutine 1
        GOROOT/src/runtime/mfinal.go:163 +0x3d

goroutine 6 [chan receive]:
runtime.gopark(0x15a53e0?, 0x2?, 0x40?, 0xe7?, 0x431ccb?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00005e700 sp=0xc00005e6e0 pc=0x43bf4e
runtime.chanrecv(0xc000188000, 0xc00005e7b8, 0x1)
        GOROOT/src/runtime/chan.go:583 +0x3cd fp=0xc00005e778 sp=0xc00005e700 pc=0x409a0d
runtime.chanrecv2(0x6fc23ac00?, 0x0?)
        GOROOT/src/runtime/chan.go:447 +0x12 fp=0xc00005e7a0 sp=0xc00005e778 pc=0x409632
github.com/containerd/containerd/runtime/v2/shim.setRuntime.func1()
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:120 +0x5a fp=0xc00005e7e0 sp=0xc00005e7a0 pc=0x7256da
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x46c0e1
created by github.com/containerd/containerd/runtime/v2/shim.setRuntime in goroutine 1
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:119 +0x25

goroutine 7 [select, 33307 minutes, locked to thread]:
runtime.gopark(0xc00005efa8?, 0x2?, 0x60?, 0xee?, 0xc00005efa4?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00005ee38 sp=0xc00005ee18 pc=0x43bf4e
runtime.selectgo(0xc00005efa8, 0xc00005efa0, 0x0?, 0x0, 0x0?, 0x1)
        GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00005ef58 sp=0xc00005ee38 pc=0x44bac5
runtime.ensureSigM.func1()
        GOROOT/src/runtime/signal_unix.go:1014 +0x19f fp=0xc00005efe0 sp=0xc00005ef58 pc=0x462f9f
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x46c0e1
created by runtime.ensureSigM in goroutine 1
        GOROOT/src/runtime/signal_unix.go:997 +0xc8

goroutine 8 [syscall, 1 minutes]:
runtime.notetsleepg(0xffffffffffffffff?, 0xc00005f728?)
        GOROOT/src/runtime/lock_futex.go:236 +0x29 fp=0xc00005f7a0 sp=0xc00005f768 pc=0x40ed29
os/signal.signal_recv()
        GOROOT/src/runtime/sigqueue.go:152 +0x29 fp=0xc00005f7c0 sp=0xc00005f7a0 pc=0x4686e9
os/signal.loop()
        GOROOT/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00005f7e0 sp=0xc00005f7c0 pc=0x719c53
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005f7e8 sp=0xc00005f7e0 pc=0x46c0e1
created by os/signal.Notify.func1.1 in goroutine 1
        GOROOT/src/os/signal/signal.go:151 +0x1f

goroutine 9 [chan receive, 33307 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00005fea8 sp=0xc00005fe88 pc=0x43bf4e
runtime.chanrecv(0xc00016c720, 0xc00005ff90, 0x1)
        GOROOT/src/runtime/chan.go:583 +0x3cd fp=0xc00005ff20 sp=0xc00005fea8 pc=0x409a0d
runtime.chanrecv2(0x0?, 0x0?)
        GOROOT/src/runtime/chan.go:447 +0x12 fp=0xc00005ff48 sp=0xc00005ff20 pc=0x409632
github.com/containerd/containerd/runtime/v2/shim.(*RemoteEventsPublisher).processQueue(0xc0001701e0)
        external/com_github_containerd_containerd/runtime/v2/shim/publisher.go:80 +0x45 fp=0xc00005ffc8 sp=0xc00005ff48 pc=0x721ce5
github.com/containerd/containerd/runtime/v2/shim.NewPublisher.func2()
        external/com_github_containerd_containerd/runtime/v2/shim/publisher.go:56 +0x25 fp=0xc00005ffe0 sp=0xc00005ffc8 pc=0x721ac5
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x46c0e1
created by github.com/containerd/containerd/runtime/v2/shim.NewPublisher in goroutine 1
        external/com_github_containerd_containerd/runtime/v2/shim/publisher.go:56 +0x15b

goroutine 10 [select, 33307 minutes]:
runtime.gopark(0xc00008cdf8?, 0x2?, 0x0?, 0x0?, 0xc00008cdac?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00008cc48 sp=0xc00008cc28 pc=0x43bf4e
runtime.selectgo(0xc00008cdf8, 0xc00008cda8, 0x0?, 0x0, 0x0?, 0x1)
        GOROOT/src/runtime/select.go:327 +0x725 fp=0xc00008cd68 sp=0xc00008cc48 pc=0x44bac5
gvisor.dev/gvisor/pkg/shim.(*watcherV2).run(0xc000011410, {0xed8d40, 0xc00008aa00})
        pkg/shim/oom_v2.go:62 +0x130 fp=0xc00008cfb8 sp=0xc00008cd68 pc=0xb992d0
gvisor.dev/gvisor/pkg/shim.New.func1()
        pkg/shim/service.go:112 +0x2d fp=0xc00008cfe0 sp=0xc00008cfb8 pc=0xb9a12d
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008cfe8 sp=0xc00008cfe0 pc=0x46c0e1
created by gvisor.dev/gvisor/pkg/shim.New in goroutine 1
        pkg/shim/service.go:112 +0x1fe

goroutine 11 [chan receive, 66 minutes]:
runtime.gopark(0xc0000a8900?, 0xed8d40?, 0x0?, 0xaa?, 0x0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc0001afe58 sp=0xc0001afe38 pc=0x43bf4e
runtime.chanrecv(0xc00016c5a0, 0xc000025f78, 0x1)
        GOROOT/src/runtime/chan.go:583 +0x3cd fp=0xc0001afed0 sp=0xc0001afe58 pc=0x409a0d
runtime.chanrecv2(0xc0000a8900?, 0xed8d40?)
        GOROOT/src/runtime/chan.go:447 +0x12 fp=0xc0001afef8 sp=0xc0001afed0 pc=0x409632
gvisor.dev/gvisor/pkg/shim.(*service).processExits(0xc0000a8900, {0xed8d40, 0xc00008aa00})
        pkg/shim/service.go:958 +0xf0 fp=0xc0001affb8 sp=0xc0001afef8 pc=0xba0b10
gvisor.dev/gvisor/pkg/shim.New.func2()
        pkg/shim/service.go:122 +0x28 fp=0xc0001affe0 sp=0xc0001affb8 pc=0xb9a0c8
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001affe8 sp=0xc0001affe0 pc=0x46c0e1
created by gvisor.dev/gvisor/pkg/shim.New in goroutine 1
        pkg/shim/service.go:122 +0x399

goroutine 12 [syscall, 7 minutes]:
syscall.Syscall6(0x46a2d2?, 0xc00008d8e8?, 0x44c865?, 0xc00008d8d0?, 0x44c8a0?, 0xc000007a00?, 0x4?)
        GOROOT/src/syscall/syscall_linux.go:91 +0x30 fp=0xc00008d8b0 sp=0xc00008d828 pc=0x4837b0
syscall.Syscall6(0xe8, 0x4, 0xc00008d9b0, 0x80, 0xffffffffffffffff, 0x0, 0x0)
        <autogenerated>:1 +0x3d fp=0xc00008d8f8 sp=0xc00008d8b0 pc=0x4841dd
golang.org/x/sys/unix.EpollWait(0xc00024ab70?, {0xc00008d9b0?, 0x0?, 0x0?}, 0x0?)
        external/org_golang_x_sys/unix/zsyscall_linux_amd64.go:56 +0x4f fp=0xc00008d968 sp=0xc00008d8f8 pc=0x66336f
github.com/containerd/console.(*Epoller).Wait(0xc000170420)
        external/com_github_containerd_console/console_linux.go:111 +0x5d fp=0xc00008dfc8 sp=0xc00008d968 pc=0x71341d
gvisor.dev/gvisor/pkg/shim.(*service).initPlatform.func1()
        pkg/shim/service_linux.go:107 +0x25 fp=0xc00008dfe0 sp=0xc00008dfc8 pc=0xba2d25
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x46c0e1
created by gvisor.dev/gvisor/pkg/shim.(*service).initPlatform in goroutine 1
        pkg/shim/service_linux.go:107 +0x105

goroutine 13 [chan receive, 66 minutes]:
runtime.gopark(0xc00024ad20?, 0xc000215f18?, 0x7c?, 0x21?, 0xc0001701e0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc000215e88 sp=0xc000215e68 pc=0x43bf4e
runtime.chanrecv(0xc00016c780, 0xc000215f78, 0x1)
        GOROOT/src/runtime/chan.go:583 +0x3cd fp=0xc000215f00 sp=0xc000215e88 pc=0x409a0d
runtime.chanrecv2(0xd54940?, 0xed8d40?)
        GOROOT/src/runtime/chan.go:447 +0x12 fp=0xc000215f28 sp=0xc000215f00 pc=0x409632
gvisor.dev/gvisor/pkg/shim.(*service).forward(0x0?, {0xed8d40, 0xc00008aa00}, {0xecece8, 0xc0001701e0})
        pkg/shim/service.go:1017 +0x52 fp=0xc000215fa8 sp=0xc000215f28 pc=0xba1392
gvisor.dev/gvisor/pkg/shim.New.func3()
        pkg/shim/service.go:128 +0x30 fp=0xc000215fe0 sp=0xc000215fa8 pc=0xb9a070
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000215fe8 sp=0xc000215fe0 pc=0x46c0e1
created by gvisor.dev/gvisor/pkg/shim.New in goroutine 1
        pkg/shim/service.go:128 +0x525

goroutine 18 [select]:
runtime.gopark(0xc0001adf68?, 0x4?, 0x5?, 0xec?, 0xc0001add30?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc0001adbb8 sp=0xc0001adb98 pc=0x43bf4e
runtime.selectgo(0xc0001adf68, 0xc0001add28, 0x1564510?, 0x0, 0x25?, 0x1)
        GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0001adcd8 sp=0xc0001adbb8 pc=0x44bac5
github.com/containerd/ttrpc.(*serverConn).run(0xc00008aa50, {0xed8d40, 0xc00008aa00})
        external/com_github_containerd_ttrpc/server.go:431 +0x51b fp=0xc0001adfb8 sp=0xc0001adcd8 pc=0x6746db
github.com/containerd/ttrpc.(*Server).Serve.func2()
        external/com_github_containerd_ttrpc/server.go:127 +0x28 fp=0xc0001adfe0 sp=0xc0001adfb8 pc=0x6738a8
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001adfe8 sp=0xc0001adfe0 pc=0x46c0e1
created by github.com/containerd/ttrpc.(*Server).Serve in goroutine 16
        external/com_github_containerd_ttrpc/server.go:127 +0x252

goroutine 16 [IO wait, 33307 minutes]:
runtime.gopark(0xc00017c000?, 0xc000040780?, 0xa8?, 0xc?, 0x4e259d?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc000090c38 sp=0xc000090c18 pc=0x43bf4e
runtime.netpollblock(0x11?, 0x407e46?, 0x0?)
        GOROOT/src/runtime/netpoll.go:564 +0xf7 fp=0xc000090c70 sp=0xc000090c38 pc=0x434997
internal/poll.runtime_pollWait(0x7f6ca5529d88, 0x72)
        GOROOT/src/runtime/netpoll.go:343 +0x85 fp=0xc000090c90 sp=0xc000090c70 pc=0x466705
internal/poll.(*pollDesc).wait(0xc0000e7280?, 0x661352?, 0x0)
        GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000090cb8 sp=0xc000090c90 pc=0x4db207
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0000e7280)
        GOROOT/src/internal/poll/fd_unix.go:611 +0x2ac fp=0xc000090d60 sp=0xc000090cb8 pc=0x4e06ec
net.(*netFD).accept(0xc0000e7280)
        GOROOT/src/net/fd_unix.go:172 +0x29 fp=0xc000090e18 sp=0xc000090d60 pc=0x533b29
net.(*UnixListener).accept(0x4442e0?)
        GOROOT/src/net/unixsock_posix.go:172 +0x16 fp=0xc000090e40 sp=0xc000090e18 pc=0x54d8f6
net.(*UnixListener).Accept(0xc0001705a0)
        GOROOT/src/net/unixsock.go:260 +0x30 fp=0xc000090e70 sp=0xc000090e40 pc=0x54c030
github.com/containerd/ttrpc.(*Server).Serve(0xc000170540, {0xed8d40?, 0xc00008aa00}, {0xed5250, 0xc0001705a0})
        external/com_github_containerd_ttrpc/server.go:87 +0x122 fp=0xc000090f60 sp=0xc000090e70 pc=0x673502
github.com/containerd/containerd/runtime/v2/shim.serve.func1()
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:310 +0x6d fp=0xc000090fe0 sp=0xc000090f60 pc=0x72400d
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x46c0e1
created by github.com/containerd/containerd/runtime/v2/shim.serve in goroutine 1
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:308 +0xcd

goroutine 17 [chan receive, 1 minutes]:
runtime.gopark(0x1?, 0x1?, 0x20?, 0x6e?, 0xc0000485b0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc0001abf00 sp=0xc0001abee0 pc=0x43bf4e
runtime.chanrecv(0xc00016ca20, 0xc000029fc0, 0x1)
        GOROOT/src/runtime/chan.go:583 +0x3cd fp=0xc0001abf78 sp=0xc0001abf00 pc=0x409a0d
runtime.chanrecv2(0x0?, 0x0?)
        GOROOT/src/runtime/chan.go:447 +0x12 fp=0xc0001abfa0 sp=0xc0001abf78 pc=0x409632
github.com/containerd/containerd/runtime/v2/shim.(*Client).Serve.func1()
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:294 +0x47 fp=0xc0001abfe0 sp=0xc0001abfa0 pc=0x723e47
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001abfe8 sp=0xc0001abfe0 pc=0x46c0e1
created by github.com/containerd/containerd/runtime/v2/shim.(*Client).Serve in goroutine 1
        external/com_github_containerd_containerd/runtime/v2/shim/shim.go:293 +0x338

goroutine 19 [IO wait]:
runtime.gopark(0xc000092d20?, 0xb?, 0x0?, 0x0?, 0xb?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc000092b48 sp=0xc000092b28 pc=0x43bf4e
runtime.netpollblock(0x481698?, 0x407e46?, 0x0?)
        GOROOT/src/runtime/netpoll.go:564 +0xf7 fp=0xc000092b80 sp=0xc000092b48 pc=0x434997
internal/poll.runtime_pollWait(0x7f6ca5529c90, 0x72)
        GOROOT/src/runtime/netpoll.go:343 +0x85 fp=0xc000092ba0 sp=0xc000092b80 pc=0x466705
internal/poll.(*pollDesc).wait(0xc0000e7380?, 0xc00020c000?, 0x0)
        GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000092bc8 sp=0xc000092ba0 pc=0x4db207
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0000e7380, {0xc00020c000, 0x1000, 0x1000})
        GOROOT/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc000092c60 sp=0xc000092bc8 pc=0x4dc4fa
net.(*netFD).Read(0xc0000e7380, {0xc00020c000?, 0x0?, 0xc000092d00?})
        GOROOT/src/net/fd_posix.go:55 +0x25 fp=0xc000092ca8 sp=0xc000092c60 pc=0x531b05
net.(*conn).Read(0xc000060138, {0xc00020c000?, 0xc000092d48?, 0x408ff3?})
        GOROOT/src/net/net.go:179 +0x45 fp=0xc000092cf0 sp=0xc000092ca8 pc=0x53ea45
net.(*UnixConn).Read(0xc000092d48?, {0xc00020c000?, 0x409060?, 0x1?})
        <autogenerated>:1 +0x25 fp=0xc000092d20 sp=0xc000092cf0 pc=0x550da5
bufio.(*Reader).Read(0xc00016cba0, {0xc00009f320, 0xa, 0xc000092ed4?})
        GOROOT/src/bufio/bufio.go:244 +0x197 fp=0xc000092d58 sp=0xc000092d20 pc=0x50af97
io.ReadAtLeast({0xecc720, 0xc00016cba0}, {0xc00009f320, 0xa, 0xa}, 0xa)
        GOROOT/src/io/io.go:335 +0x90 fp=0xc000092da0 sp=0xc000092d58 pc=0x4d59b0
io.ReadFull(...)
        GOROOT/src/io/io.go:354
github.com/containerd/ttrpc.readMessageHeader({0xc00009f320, 0xa, 0x18?}, {0xecc720?, 0xc00016cba0?})
        external/com_github_containerd_ttrpc/channel.go:53 +0x4f fp=0xc000092de0 sp=0xc000092da0 pc=0x66fd4f
github.com/containerd/ttrpc.(*channel).recv(0xc00009f300)
        external/com_github_containerd_ttrpc/channel.go:101 +0x45 fp=0xc000092e78 sp=0xc000092de0 pc=0x670245
github.com/containerd/ttrpc.(*serverConn).run.func1(0xc00016cc00)
        external/com_github_containerd_ttrpc/server.go:362 +0x13b fp=0xc000092fc8 sp=0xc000092e78 pc=0x674f3b
github.com/containerd/ttrpc.(*serverConn).run.func6()
        external/com_github_containerd_ttrpc/server.go:413 +0x27 fp=0xc000092fe0 sp=0xc000092fc8 pc=0x674dc7
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000092fe8 sp=0xc000092fe0 pc=0x46c0e1
created by github.com/containerd/ttrpc.(*serverConn).run in goroutine 18
        external/com_github_containerd_ttrpc/server.go:332 +0x336

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x8f4d6a15645d9?, 0x2?, 0x10?, 0x2?, 0x0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00005a750 sp=0xc00005a730 pc=0x43bf4e
runtime.gcBgMarkWorker()
        GOROOT/src/runtime/mgc.go:1293 +0xe5 fp=0xc00005a7e0 sp=0xc00005a750 pc=0x41ea85
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x46c0e1
created by runtime.gcBgMarkStartWorkers in goroutine 20
        GOROOT/src/runtime/mgc.go:1217 +0x1c

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x8f4d6a1564790?, 0x2?, 0x55?, 0x1?, 0x0?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc00005af50 sp=0xc00005af30 pc=0x43bf4e
runtime.gcBgMarkWorker()
        GOROOT/src/runtime/mgc.go:1293 +0xe5 fp=0xc00005afe0 sp=0xc00005af50 pc=0x41ea85
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005afe8 sp=0xc00005afe0 pc=0x46c0e1
created by runtime.gcBgMarkStartWorkers in goroutine 20
        GOROOT/src/runtime/mgc.go:1217 +0x1c

goroutine 1563765 [select, 66 minutes]:
runtime.gopark(0xc0001acfb0?, 0x2?, 0x50?, 0xce?, 0xc0001acf6c?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc0001ace10 sp=0xc0001acdf0 pc=0x43bf4e
runtime.selectgo(0xc0001acfb0, 0xc0001acf68, 0xc97d00?, 0x0, 0xd54940?, 0x1)
        GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0001acf30 sp=0xc0001ace10 pc=0x44bac5
github.com/containerd/ttrpc.(*Client).run.func1()
        external/com_github_containerd_ttrpc/client.go:265 +0xb6 fp=0xc0001acfe0 sp=0xc0001acf30 pc=0x6721b6
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001acfe8 sp=0xc0001acfe0 pc=0x46c0e1
created by github.com/containerd/ttrpc.(*Client).run in goroutine 1563764
        external/com_github_containerd_ttrpc/client.go:262 +0xf7

goroutine 44 [syscall, 33307 minutes]:
syscall.Syscall(0x0?, 0x0?, 0xc0000dedd0?, 0x4377db?)
        GOROOT/src/syscall/syscall_linux.go:69 +0x25 fp=0xc00008ec18 sp=0xc00008eba8 pc=0x483725
syscall.read(0xc00005bc80?, {0xc00008ed68?, 0x0?, 0x15bcf00?})
        GOROOT/src/syscall/zsyscall_linux_amd64.go:721 +0x38 fp=0xc00008ec58 sp=0xc00008ec18 pc=0x481698
syscall.Read(...)
        GOROOT/src/syscall/syscall_unix.go:181
github.com/containerd/cgroups/v2.(*Manager).waitForEvents(0xc00007e0e0, 0x70f9a7?, 0x0?)
        external/com_github_containerd_cgroups/v2/manager.go:587 +0x176 fp=0xc00008efb8 sp=0xc00008ec58 pc=0x830f96
github.com/containerd/cgroups/v2.(*Manager).EventChan.func1()
        external/com_github_containerd_cgroups/v2/manager.go:569 +0x28 fp=0xc00008efe0 sp=0xc00008efb8 pc=0x830de8
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x46c0e1
created by github.com/containerd/cgroups/v2.(*Manager).EventChan in goroutine 20
        external/com_github_containerd_cgroups/v2/manager.go:569 +0x9d

goroutine 45 [select, 33307 minutes]:
runtime.gopark(0xc000059768?, 0x2?, 0x0?, 0x0?, 0xc0000596f4?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc000059590 sp=0xc000059570 pc=0x43bf4e
runtime.selectgo(0xc000059768, 0xc0000596f0, 0x0?, 0x0, 0x0?, 0x1)
        GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0000596b0 sp=0xc000059590 pc=0x44bac5
gvisor.dev/gvisor/pkg/shim.(*watcherV2).add.func1()
        pkg/shim/oom_v2.go:101 +0x14e fp=0xc0000597e0 sp=0xc0000596b0 pc=0xb998ce
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000597e8 sp=0xc0000597e0 pc=0x46c0e1
created by gvisor.dev/gvisor/pkg/shim.(*watcherV2).add in goroutine 20
        pkg/shim/oom_v2.go:98 +0xcd

goroutine 1563766 [IO wait, 66 minutes]:
runtime.gopark(0xdebd30?, 0xb?, 0x0?, 0x0?, 0x1c?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc0001a9ba0 sp=0xc0001a9b80 pc=0x43bf4e
runtime.netpollblock(0x481698?, 0x407e46?, 0x0?)
        GOROOT/src/runtime/netpoll.go:564 +0xf7 fp=0xc0001a9bd8 sp=0xc0001a9ba0 pc=0x434997
internal/poll.runtime_pollWait(0x7f6ca55292e0, 0x72)
        GOROOT/src/runtime/netpoll.go:343 +0x85 fp=0xc0001a9bf8 sp=0xc0001a9bd8 pc=0x466705
internal/poll.(*pollDesc).wait(0xc0001d6480?, 0xc0002a7000?, 0x0)
        GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0001a9c20 sp=0xc0001a9bf8 pc=0x4db207
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001d6480, {0xc0002a7000, 0x1000, 0x1000})
        GOROOT/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0001a9cb8 sp=0xc0001a9c20 pc=0x4dc4fa
net.(*netFD).Read(0xc0001d6480, {0xc0002a7000?, 0xc0001a9d78?, 0x410285?})
        GOROOT/src/net/fd_posix.go:55 +0x25 fp=0xc0001a9d00 sp=0xc0001a9cb8 pc=0x531b05
net.(*conn).Read(0xc000060200, {0xc0002a7000?, 0x410285?, 0xc00024c468?})
        GOROOT/src/net/net.go:179 +0x45 fp=0xc0001a9d48 sp=0xc0001a9d00 pc=0x53ea45
net.(*UnixConn).Read(0xc05ce0?, {0xc0002a7000?, 0x7f6ca552ead0?, 0x7f6cec0ba5b8?})
        <autogenerated>:1 +0x25 fp=0xc0001a9d78 sp=0xc0001a9d48 pc=0x550da5
bufio.(*Reader).Read(0xc000189c20, {0xc00009fc20, 0xa, 0x408ebe?})
        GOROOT/src/bufio/bufio.go:244 +0x197 fp=0xc0001a9db0 sp=0xc0001a9d78 pc=0x50af97
io.ReadAtLeast({0xecc720, 0xc000189c20}, {0xc00009fc20, 0xa, 0xa}, 0xa)
        GOROOT/src/io/io.go:335 +0x90 fp=0xc0001a9df8 sp=0xc0001a9db0 pc=0x4d59b0
io.ReadFull(...)
        GOROOT/src/io/io.go:354
github.com/containerd/ttrpc.readMessageHeader({0xc00009fc20, 0xa, 0xc0001a9e80?}, {0xecc720?, 0xc000189c20?})
        external/com_github_containerd_ttrpc/channel.go:53 +0x4f fp=0xc0001a9e38 sp=0xc0001a9df8 pc=0x66fd4f
github.com/containerd/ttrpc.(*channel).recv(0xc00009fc00)
        external/com_github_containerd_ttrpc/channel.go:101 +0x45 fp=0xc0001a9ed0 sp=0xc0001a9e38 pc=0x670245
github.com/containerd/ttrpc.(*Client).run.func2()
        external/com_github_containerd_ttrpc/client.go:294 +0xb5 fp=0xc0001a9fe0 sp=0xc0001a9ed0 pc=0x671db5
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001a9fe8 sp=0xc0001a9fe0 pc=0x46c0e1
created by github.com/containerd/ttrpc.(*Client).run in goroutine 1563764
        external/com_github_containerd_ttrpc/client.go:286 +0x15f

goroutine 1563764 [select, 76 minutes]:
runtime.gopark(0xc0001faf90?, 0x2?, 0xc0?, 0x63?, 0xc0001faf64?)
        GOROOT/src/runtime/proc.go:398 +0xce fp=0xc0001fae08 sp=0xc0001fade8 pc=0x43bf4e
runtime.selectgo(0xc0001faf90, 0xc0001faf60, 0xc0001fafd0?, 0x0, 0xc00004600e?, 0x1)
        GOROOT/src/runtime/select.go:327 +0x725 fp=0xc0001faf28 sp=0xc0001fae08 pc=0x44bac5
github.com/containerd/ttrpc.(*Client).run(0xc0001d6680)
        external/com_github_containerd_ttrpc/client.go:330 +0x1ee fp=0xc0001fafc8 sp=0xc0001faf28 pc=0x671c2e
github.com/containerd/ttrpc.NewClient.func2()
        external/com_github_containerd_ttrpc/client.go:94 +0x25 fp=0xc0001fafe0 sp=0xc0001fafc8 pc=0x670945
runtime.goexit()
        src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0001fafe8 sp=0xc0001fafe0 pc=0x46c0e1
created by github.com/containerd/ttrpc.NewClient in goroutine 13
        external/com_github_containerd_ttrpc/client.go:94 +0x1d6

rax    0xca
rbx    0x0
rcx    0x46df03
rdx    0x0
rdi    0x158b9c8
rsi    0x80
rbp    0x7fff6bfe67b0
rsp    0x7fff6bfe6768
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x442820
r13    0xc00005efa4
r14    0x158b0a0
r15    0x0
rip    0x46df01
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

Sadly, I'm not able to see anything fishy in here. I'm saying this shim leaked because it doesn't have any child processes. That in itself might be me just being wrong.

Jan 10 '24 20:01 markusthoemmes

I've made some progress on this: I was able to reproduce this sortakinda reliably (not reliable on my dev clusters yet) by force-exceeding the ephemeral-storage in the respective pod. The exec to the pod drops and I can't exec into the pod again but it remains intact because the kubelet can't evict it.

We haven't adjusted the overlay2 setting, so we should be running with overlay2=root:self. Does this ring any bells @ayushr2? I'll share a proper reproducer once I can reliably do it.

Edit: I can cautiously confirm that using overlay2=none fixes the issue. I'll keep poking though.

Jan 17 '24 11:01 markusthoemmes

Interesting, thanks for the investigation!

We have internal tests which test exceeding the ephemeral storage limits in GKE with overlay2=root:self and the pod gets evicted by the kubelet. Happy to look at the reproducer.

Jan 17 '24 15:01 ayushr2

Thanks @ayushr2. Here we go, I've gotten it to reproduce with the below PodSpec. It's been a little finicky so I left seemingly unrelated parts in it as well. FWIW, I've tested this on the latest gVisor release as well, just in case. Same behavior there.

So as a reminder:

K8s version: 1.28.2
containerd version: 1.7.8
gvisor version: release-20240109.0

apiVersion: v1
kind: Pod
metadata:
  name: repro
spec:
  restartPolicy: Always
  containers:
  - name: repro
    image: shlinkio/shlink@sha256:c70cf1b37087581cfcb7963d74d6c13fbee8555a7b10aa4af0493e70ade41202
    env:
    - name: INITIAL_API_KEY
      value: foobar
    - name: DEFAULT_DOMAIN
      value: foo.bar
    resources:
      limits:
        cpu: "1"
        ephemeral-storage: 4G
        memory: 2Gi
      requests:
        cpu: 200m
        ephemeral-storage: 400M
        memory: "858993459"

You can trip the issue by running

$ sleep 60 # Not sure if this is really doing anything, but it seems to have improved the likelihood of it happening
$ kubectl exec repro -- dd if=/dev/zero of=./big_file bs=4k iflag=fullblock,count_bytes count=10G

Sometimes, this works fine, leaving the pod in an Error state. Sometimes, the pod doesn't hang but goes into a ContainerStatusUnknown state (which feels wrong too) and sometimes the container just stays Running (even though it can't be execed into anymore). If you try to delete it then, the same hang as introduced above happens.

For some reason that is still beyond me, it happens much more consistently when doing the same thing to the pods of the following deployment

apiVersion: v1
data:
  DEFAULT_DOMAIN: Zm9vLmJhcg==
  GEOLITE_LICENSE_KEY: ""
  INITIAL_API_KEY: Zm9vYmFy
kind: Secret
metadata:
  name: shlinkio-shlink-secrets
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shlinkio-shlink
spec:
  minReadySeconds: 10
  progressDeadlineSeconds: 730
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app-component: shlinkio-shlink
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        env-var-hash: 1c3af241bb8862b3fc57e28a31797c39
      creationTimestamp: null
      labels:
        app-component: shlinkio-shlink
    spec:
      automountServiceAccountToken: false
      containers:
      - env:
        - name: CNB_PROCESS_TYPE
          value: web
        - name: KUBERNETES_PORT
        - name: KUBERNETES_PORT_443_TCP
        - name: KUBERNETES_PORT_443_TCP_ADDR
        - name: KUBERNETES_PORT_443_TCP_PORT
        - name: KUBERNETES_PORT_443_TCP_PROTO
        - name: KUBERNETES_SERVICE_HOST
        - name: KUBERNETES_SERVICE_PORT
        - name: KUBERNETES_SERVICE_PORT_HTTPS
        - name: PORT
          value: "8080"
        envFrom:
        - secretRef:
            name: shlinkio-shlink-secrets
        image: shlinkio/shlink@sha256:c70cf1b37087581cfcb7963d74d6c13fbee8555a7b10aa4af0493e70ade41202
        imagePullPolicy: IfNotPresent
        name: shlinkio-shlink
        ports:
        - containerPort: 8080
          name: http-8080
          protocol: TCP
        readinessProbe:
          failureThreshold: 9
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8080
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "1"
            ephemeral-storage: 4G
            memory: 2Gi
          requests:
            cpu: 200m
            ephemeral-storage: 400M
            memory: "858993459"
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      enableServiceLinks: false
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app-component: shlinkio-shlink
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway

Jan 17 '24 16:01 markusthoemmes

I have been trying to use the following GKE Reproducer:

$ gcloud container clusters create repro --num-nodes=1 --location=us-central1-a --cluster-version=1.28.3-gke.1203001

$ gcloud container node-pools create gvisor --cluster=repro --num-nodes=1 --location=us-central1-a --sandbox=type=gvisor --image-type=cos_containerd --machine-type=e2-standard-2
...
NAME    MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
gvisor  e2-standard-2  100           1.28.3-gke.1203001

$ kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-node-tools/master/gvisor/enable-gvisor-flags.yaml
daemonset.apps/enable-gvisor-flags created

$ cat repro.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: repro
  annotations:
    dev.gvisor.flag.platform: "systrap"
    dev.gvisor.flag.directfs: "false"
spec:
  runtimeClassName: gvisor
  restartPolicy: Always
  containers:
  - name: repro
    image: shlinkio/shlink@sha256:c70cf1b37087581cfcb7963d74d6c13fbee8555a7b10aa4af0493e70ade41202
    env:
    - name: INITIAL_API_KEY
      value: foobar
    - name: DEFAULT_DOMAIN
      value: foo.bar
    resources:
      limits:
        cpu: "1"
        ephemeral-storage: 4G
        memory: 2Gi
      requests:
        cpu: 200m
        ephemeral-storage: 400M
        memory: "858993459"

$ kubectl apply -f repro.yaml
pod/repro created

$ sleep 60 && kubectl exec repro -- dd if=/dev/zero of=./big_file bs=4k iflag=fullblock,count_bytes count=10G
command terminated with exit code 137

$ kubectl get pods
NAME    READY   STATUS                   RESTARTS   AGE
repro   0/1     ContainerStatusUnknown   1          2m

I have only been able to repro the ContainerStatusUnknown and Error cases. Across 12 runs, I got ContainerStatusUnknown 8 times and Error 4 times. But I have not been able to get the Running case (which reproduces your issue). Let me know if I am doing something wrong.

$ kubectl version
Client Version: v1.28.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3-gke.1203001

$ containerd --version
containerd github.com/containerd/containerd 1.7.7 8c087663b0233f6e6e2f4515cee61d49f14746a8

$ /home/containerd/usr/local/sbin/runsc --version
runsc version google-573904262    # This maps to 20231009.0_RC01
spec: 1.1.0-rc.2

Let me try the deployment...

Jan 18 '24 19:01 ayushr2

No luck with the deployment OR using the latest runsc from master.

FWIW, both /run/containerd/io.containerd.runtime.v2.task/k8s.io/5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f/rootfs and /run/containerd/io.containerd.runtime.v2.task/k8s.io/90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8/rootfs are empty on ls -l

@markusthoemmes Could you check the stuck container's rootfs with ls -al? Because with --overlay2=root:self, we create an hidden file named .gvisor.filestore.{sandbox-ID}. So ls -l by itself will not show this file.

This file is deleted when the container is destroyed. From your logs above, you can see that the sandbox is still holding an FD to this file:

lrwx------ 1 nobody nogroup 64 Jan  3 18:23 47 -> '/run/containerd/io.containerd.runtime.v2.task/k8s.io/5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f/rootfs/.gvisor.filestore.90022bca840ef2fac36200ae8f3511800e883f07558b146290c3b6a9822090b8 (deleted)'

It shows that the file is deleted (which means ls -al may also show nothing, but need to confirm, so we know if Container.Destroy() code was executed). However, the file will only be "released" in the host kernel when all open FDs on it are closed. Since sandbox is holding an FD on it, rootfs can not be umounted (hence EBUSY error). So maybe we are leaking this filestore FD somewhere, let me look more closely.

Meanwhile, can you try reproducing with --overlay2=none?

Jan 18 '24 20:01 ayushr2

Actually, the other FDs shown in this comment are also from the container rootfs. They don't have the /run/containerd/io.containerd.runtime.v2.task/k8s.io/5b6ae772c71b57b0a00297b775a1900bcd30fcd9a6c8ccca2e49573829ae636f/rootfs/ path prefix because they were opened after the rootfs was pivot_root(2)-ed.

The filestore FD is closed when the sentry overlayfs mount (and hence the tmpfs upper layer) are unmounted inside the sentry. Given this FD is still open, it means that the rootfs mount is not being unmounted inside the sentry (hence leaking all these host FDs).

So I don't think --overlay2=none will fix this issue, but could you still please give it a shot.

Jan 18 '24 20:01 ayushr2

gvisor gvisor copied to clipboard

Pods stuck in Terminating state due to process not being killed

Description

Pod state

container state in crictl

pod state in crictl

container state in ctr

pod state in ctr

Logs grepping for the pod

Logs grepping for the container

Steps to reproduce

runsc version

docker version (if using docker)

uname

kubectl (if using Kubernetes)

repo state (if built from source)

runsc debug logs (if available)

gvisor
gvisor copied to clipboard