nerdctl
nerdctl copied to clipboard
containerd
nerdctl oci hook failed with wasm image (`containerd-shim-wasmtime-v1`)
Description
To reproduce, you can install a wasmtime shim on your PATH named containerd-shim-wasmtime-v1 from https://github.com/containerd/runwasi/releases/tag/containerd-shim-wasmtime%2Fv0.5.0.
And then try to run with the following image: ghcr.io/containerd/runwasi/wasi-demo-app:latest and you will see
sudo nerdctl run --rm --runtime=io.containerd.wasmtime.v1 --debug ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm /wasi-demo-app.wasm echo 'hello'
DEBU[0000] verifying process skipped
DEBU[0000] generated log driver: binary:///usr/local/bin/nerdctl?_NERDCTL_INTERNAL_LOGGING=%2Fvar%2Flib%2Fnerdctl%2F1935db59
DEBU[0000] detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf
WARN[0000] cannot set cgroup manager to "systemd" for runtime "io.containerd.wasmtime.v1"
DEBU[0000] remote introspection plugin filters filters="[type==io.containerd.snapshotter.v1, id==overlayfs]"
FATA[0000] failed to create shim task: failed to create container: hook command exited with non-zero exit code: 1: unknown
Steps to reproduce the issue
Describe the results you received and expected
The expected result should be "hello" printed and the container exsited.
What version of nerdctl are you using?
nerdctl version 2.0.4
Are you using a variant of nerdctl? (e.g., Rancher Desktop)
None
Host information
No response
Note: works with ctr
sudo ctr run --rm --runtime=io.containerd.wasmtime.v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm
@AkihiroSuda tag expert? This is unlikely to be looked at by casual contributors IMHO.
--net=none seems to avoid the hook issue, but still fails due to another error
$ sudo nerdctl run --net=none --rm --runtime=io.containerd.wasmedge.v1 ghcr.io/containerd/runwasi/wasi-demo-app:latest testwasm
WARN[0000] cannot set cgroup manager to "systemd" for runtime "io.containerd.wasmedge.v1"
FATA[0000] failed to create shim task: failed to create container: intermediate process error cgroup error: realtime is not supported on v2 yet
For the ocihooks part, we do not get a state.Pid in onCreateRuntime for some reason, so, networking is not happy and does bail out with:
both state.Pid and the netNs annotation are unset
It looks like for normal container, we start with onPostStop, then onCreateRuntime - while with wasm we get onCreateRuntime directly.
I need to read up again on oci lifecycle to refresh on this thing.
This one maybe is on us.
As for the rest: https://github.com/youki-dev/youki/blob/main/crates/libcgroups/src/v2/cpu.rs#L24
My money here is that containerd-shim-wasmtime-v1 does not support cgroup v2 (more accurately, that it depends on a crate that does not support the option it needs).
@Mossaka maybe opening a discussion with the folks at wasm about intermediate process error cgroup error: realtime is not supported on v2 yet to get their reading on this?
@AkihiroSuda maybe we keep this ticket around (or we close as third-party), but at this point it does not feel like there is anything we can do.
Apologies @Mossaka I did not notice you were a maintainer over there. So, I guess my question is for you then: ^ :P
About the oci hooks part, it seems like when we receive onCreateRuntime from wasm, we get status=creating.
If I am reading the spec correctly: https://github.com/opencontainers/runtime-spec/blob/main/runtime.md#create
We should instead receive created.
Furthermore, at status=created, I believe we should have a pid (of the container process):
pid (int, REQUIRED when status is created or running on Linux, OPTIONAL on other platforms) is the ID of the container process. For hooks executed in the runtime namespace, it is the pid as seen by the runtime. For hooks executed in the container namespace, it is the pid as seen by the container.
But I do not see anything in the state.Pid (and neither in state.Annotations[labels.PIDFile] which should be there?
Because we do not have a pid, we then bail out with an error, because we cannot figure out networking.
Can confirm, for me it fails for both wasmedge and wasmtime shims with:
WARN[0000] cannot set cgroup manager to "systemd" for runtime "io.containerd.wasmedge.v1" DEBU[0000] remote introspection plugin filters filters="[type==io.containerd.snapshotter.v1, id==overlayfs]" FATA[0000] failed to create shim task: failed to create container: hook command exited with non-zero exit code: 1: unknown
Whereas with ctr it runs perfectly
My money here is that containerd-shim-wasmtime-v1 does not support cgroup v2 (more accurately, that it depends on a crate that does not support the option it needs).
Sorry for the late reply. The Runwasi wasmtime shim depends on youki's cgroup implementation and I believe they do support cgroup v2.
@Mossaka maybe opening a discussion with the folks at wasm about
intermediate process error cgroup error: realtime is not supported on v2yet to get their reading on this?
@utam0k do you have any thoughts on this error? I skimed through the cgroup v2 documentation and it does mention that "WARNING: cgroup2 cpu controller doesn’t yet support the (bandwidth) control of realtime processes. ", so perhaps youki does the implementation in the right way.