Error: unable to start container process: error during container init

[sephethus]

trafficstars

Version 0.4.2 GKE CloudBuild based Helm deployment.

Here's the error:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/aece9c11-6a55-443b-8851-1fa6a053f6a4/volumes/kubernetes.io~empty-dir/secrets-init-volume" to rootfs at "/secrets-init/bin": open o_path procfd: open /run/containerd/io.containerd.runtime.v2.task/k8s.io/secrets-init/rootfs/secrets-init/bin: not a directory: unknown

Here's what I'm doing to start the container as one of three other containers inside the pod, the other two depend on this to get at the Google Secrets Manager secrets from the env values specified as they are in your readme, the workload identity service account should be working correctly so I think the issue is specifically mount path related:

- name: secrets-init
  image: ghcr.io/doitintl/secrets-init:latest
  command:
    - sh
  args:
    - -c
    - "cp /usr/local/bin/secrets-init /secrets-init/bin/"
  volumeMounts:
  - mountPath: /secrets-init/bin
    name: secrets-init-volume

I defined the volumes in the yaml as well:

- mountPath: /secrets-init/bin
  name: secrets-init-volume

[alexei-led]

@sephethus please check the full example

The valid flow is the following:

1. create a new emptyDir volume in a Pod, call it secrets-init-volume for example
2. add a secrets-init container to the initContainers section and copy the secrets-init binary to the secrets-init-volume volume
3. Modify the target container: mount the same secrets-init-volume (with copied secrets-init binary) and change the container's ENTRYPOINT or CMD to the secrets-init binary. Put the original ENTRYPOINT/CMD as arguments to the secrets-init.

[dmateusp]

I could reproduce that issue, if I use the example from the blog post it uses this version: doitintl/secrets-init:v0.2.1 which works fine, this version fails with the error above: doitintl/secrets-init:0.4.7

[alexei-led]

fixed

[caseycrites]

FYI that I'm still able to reproduce this behavior with version 0.5.0. I downgraded to v0.2.1 as mentioned above and things are now working.

[alexei-led]

@caseycrites can you please attach log message

[caseycrites]

I'll see if I can bring it back up today, but from what I remember, it's the exact same message as OP.

[caseycrites]

here's the exact error message, does look to be the same as OP:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/af20fbba-b03a-4019-85ae-f40671642b9e/volumes/kubernetes.io~empty-dir/secrets-init-volume" to rootfs at "/secrets-init/bin": open o_path procfd: open /run/containerd/io.containerd.runtime.v2.task/k8s.io/secrets-init/rootfs/secrets-init/bin: not a directory: unknown

[alexei-led]

@caseycrites please attach your pod YAML file here