pod failed: open /var/lib/docker/image/overlay2/layerdb/mounts/containerd://.../mount-id: no such file or directory

[ludenus]

I am trying to run kubectl flame on a new k8s cluster and it fails to launch

% kubectl flame `kubectl get pods -n dev-load| grep sharding | awk '{print $1}'` -n=dev-load -t 10s -f ./flamegraph.svg --lang java
Verifying target pod ... ✔
Launching profiler ... ❌
2022/01/12 12:32:20 pod failed

% kubectl version

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T21:16:14Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:32:32Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

flame pod contains error message like this:

{"type":"progress","data":{"time":"2022-01-12T10:32:43.529897521Z","stage":"started"}}
{"type":"error","data":{"reason":"open /var/lib/docker/image/overlay2/layerdb/mounts/containerd://166013c88125d0123a7d957165e87ca0d3cb28ee6c42327d8b905e1c8796e58c/mount-id: no such file or directory"}}

everything works just fine with older versions of k8s

% kubectl flame `kubectl get pods -n jenkins| grep hub-list | awk '{print $1}'` -n=jenkins -t 10s -f ./flamegraph.svg --lang java
Verifying target pod ... ✔
Launching profiler ... ✔
Profiling ... ✔
FlameGraph saved to: ./flamegraph.svg 🔥
√    
% kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T21:16:14Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4", GitCommit:"3cce4a82b44f032d0cd1a1790e6d2f5a55d20aae", GitTreeState:"clean", BuildDate:"2021-08-11T18:10:22Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

% kubectl flame `kubectl get pods -n solid| grep sharding | awk '{print $1}'` -n=solid -t 10s -f ./flamegraph.svg --lang java
Verifying target pod ... ✔
Launching profiler ... ✔
Profiling ... ✔
FlameGraph saved to: ./flamegraph.svg 🔥
√    
% kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T21:16:14Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

QUESTION: Is there a workaround or some special security or filesystem settings in k8s cluster I need to check?

[azlev]

Looks like your new cluster uses containerd, not docker.

[ludenus]

@azlev yes, you are right, have just checked - new cluster uses containerd runtime, old ones use docker.

Are there any plans to adjust flame plugin for containerd or it will be impossible to use flame on newer k8s versions?

[lczupryn-tibco]

+1

[baasumo]

+1

Any plans to support containerd soon, as containerd will be the default runtime in k8s moving forward, making this tool impossible to leverage.

[mkrzywanski]

+1

[baasumo]

Any updates on this issue? As containerd becomes the default runtime, this makes this issue critical for this project.

[govindaraj-indihood]

+1

[chetankme]

+1

[manjutapali]

Any updates on this issue?