Occasionally nerdctl stop never exits

[prakashmirji]

We have been using nerdctl for about 5 months. We are seeing an issue with nerdctl occasionally. Here are the details;

Env:

1. nerdctl version: 0.11.1
2. system: SLES 15 SP2
3. contained: 1.5.1

what commands are failing? nerdctl stop we then issue `nerdctl wait command to wait for the container to stop.

Occasionally, we see that stop command never exits, and then our follow-up wait command fails.

Have you seen this behavior? If so - what is the root cause? Where to look for debug/trace logs for nerdctl.

[AkihiroSuda]

• Does nerdctl kill work?
• Please try the latest containerd and nerdctl
• Please provide full reproducer

[kcsraju]

Even nerdclt rm -f hangs forever, occasionally.

# nerdctl --namespace k8s.io rm -f 286cbb5e94b8
WARN[0000] failed to send SIGKILL error="unknown error after kill: /usr/bin/nvidia-container-runtime did not terminate successfully: exit status 1: 2022/06/16 08:30:07 Error running [/usr/bin/nvidia-container-runtime --root /run/containerd/runc/k8s.io --log /run/containerd/io.containerd.runtime.v2.task/k8s.io/286cbb5e94b8b3abbb7ba3867a9ad9510da04c7987befe5a6e268e4cc952c9b2/log.json --log-format json kill 286cbb5e94b8b3abbb7ba3867a9ad9510da04c7987befe5a6e268e4cc952c9b2 9]: error creating runtime: error constructing OCI specification: error getting OCI specification file path: error getting working directory: getwd: no such file or directory\n: unknown"

we are using nerdctl v0.11.1 and container v.1.5.1. For product delivery reasons we cannot update to newer nerdctl right away. And this problem only happens occasionally. Like twice in the last month or so and we don't have a consistent way to reproduce the issue.

@AkihiroSuda we would appreciate any insights you could provide that could help us narrow down the issue.