OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown

[yanpeipan]

machine: nvidia drive px2

/apollo/data/core/core_%e.%p
dev-aarch64-20170927_1111: Pulling from apolloauto/apollo
a1c981565bcf: Already exists 
4ecd22b74242: Already exists 
8d4841161f63: Already exists 
a93b04770247: Already exists 
Digest: sha256:10fc9daa7699f30650df05b9f3aff9762a236433e089e8755dcb0f6cfb7baab0
Status: Downloaded newer image for apolloauto/apollo:dev-aarch64-20170927_1111
RTNETLINK answers: Device or resource busy
modprobe: FATAL: Module nvidia not found in directory /lib/modules/4.4.38-rt49-tegra
modprobe: FATAL: Module nvidia-uvm not found in directory /lib/modules/4.4.38-rt49-tegra
[WARNING] Failed to find device with pattern "ttyUSB*" ...
[ OK ] Found device: /dev/ttyS0.
[ OK ] Found device: /dev/ttyS3.
[ OK ] Found device: /dev/ttyS2.
[ OK ] Found device: /dev/ttyS1.
[ OK ] Found device: /dev/can3.
[ OK ] Found device: /dev/can2.
[ OK ] Found device: /dev/can1.
[ OK ] Found device: /dev/can0.
[WARNING] Failed to find device with pattern "ram*" ...
[WARNING] Failed to find device with pattern "loop*" ...
[ OK ] Found device: /dev/nvidia-uvm-tools.
[ OK ] Found device: /dev/nvidia-uvm.
[ OK ] Found device: /dev/nvidia0.
[ OK ] Found device: /dev/nvidiactl.
bb0567aef16ae92397e23fb577cabf78a3821e1bb6cef9d598c7004947618eec
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown

docker info

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.12.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.38-rt49-tegra
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 6.504GiB
Name: nvidia
ID: C4UG:ILPT:DQFX:ZUZA:R4RE:45XP:KKV3:6XMX:X4CK:D56J:C4PL:BOFZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

[andrewrynhard]

@yanpeipan I am seeing the same issue in a Kubernetes cluster. Any luck finding out the root cause here?

[AndrianD]

#subscribe

[thaJeztah]

This could be a duplicate of https://github.com/docker/for-linux/issues/238, https://github.com/moby/moby/issues/36467, which was a bug in Runc, and fixed through https://github.com/opencontainers/runc/pull/1727 in RunC 1.0.0-rc5 (included in Docker 18.03 through https://github.com/moby/moby/pull/36449).

Can you try updating to Docker 18.03.1 and see if you still run into this?

[samlyhuang]

I also see this problem

ubuntu@qa:~$ uname -a

Linux qa 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@qa:~$ docker -v

Docker version 18.03.0-ce, build 0520e24

ubuntu@qa:~$ sudo docker exec -it ihr360-one-click-provider -a

OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown

ubuntu@qa:~$ sudo docker exec -it ihr360-one-click-provider sh

OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown

this problem has fixed now?

[ruslanguns]

Me too.... looking forward for a solution ! thanks in advance

[eltiare]

I'm having this issue running Linux containers on Windows. Are these even checked anymore? :(

[tizki]

I also see this issue:

time="2018-08-06T05:05:48.591754629Z" level=error msg="Error running exec 673c1fc689e85c542d12188b7bec2819638bfedb9c05b894c072d1c6d5f5a7d2 in container: OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "process_linux.go:86: executing setns process caused \"exit status 21\"": unknown"

my environment details:

Centos 7 Linux 4.4.144-1.el7.elrepo.x86_64 #1 SMP Wed Jul 25 13:54:40 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux Docker version 18.03.0-ce, build 0520e24

[rks92]

Bump. I'm getting this issue on OSX Sierra

[egrueter-dev]

Same here. Using Docker version 18.06.1-ce, build e68fc7a

[nauer]

Got this within a docker-compose run on Ubuntu 18.04:

OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: "Snakefile.BuildChoMine": executable file not found in $PATH": unknown

Using Docker version 18.06.1-ce When I execute it from inside the container it works as usual.

[ronakganatra9]

The same issue is occurring in my Windows.

[eltiare]

I fixed this on Windows by running Docker as an administrator at least once. I don't recall if I had to do it every time though.

[ronakganatra9]

I'm having this issue running Linux containers on Windows. Are these even checked anymore? :(

same issue is in my system also.

[loomsen]

edit not applicable

[thaJeztah]

@loomsen if that stat is inside the container; also check if the binary may be dynamically linked, and may be missing a dependency. Linux may show a no such file or directory if a dependency (linked library) is missing

[loomsen]

@loomsen if that stat is inside the container; also check if the binary may be dynamically linked, and may be missing a dependency. Linux may show a no such file or directory if a dependency (linked library) is missing

No, the stat is outside. But this only happens if I use a couple of programs with their newest version, just checked, a downgrade of mitogen works. So I guess my comment is kind of obsolete?

[tkwant]

use /bin/sh instead of /bin/bash

[Wangjunyu]

thanks @tkwant it works for me

[DyingLight123]

use /bin/sh or bash

[HolyLow]

have the same problem with 18.06.1-ce

[gezichenshan]

@tkwant thanks. Why some containers just have sh and no bash? But some has both.

[jvelasquezm1]

The alpine only have the sh, it depends on how is created the container

[coderLyonMa]

have the same problem with 18.06.1-ce and it really made me crazy

[Younghae]

My environment is Vmware in Windows 10. Remove all Local volumes

1. Check all local volumes : sudo docker volume ls
2. Delete all local volumes : sudo ./byfn.sh -m down In my case it's working.

[PriyankaGupta05]

Error: failed to start container "kubernetes-demo-container": Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: "which shell": executable file not found in $PATH": unknown :( same error

[AbhimanyuG]

Oddly, it worked for me after reboot.

[tpaksu]

I fixed it with adding a dot before /bin/sh on windows minikube virtualbox setup. Final command that worked:

kubectl exec -it [pod_name] -c [container_name] -- ./bin/sh

[kalyandevopsengineer]

I ran into the same error while running the command docker exec -it jenkins bash on an already running container with name jenkins. I pulled the jenkins/jenkins image from repository.

The fix is: 1.Create a normal user/group with uid 1000. 2. add sudo permissions in visudo file. 3. Switch user as new user. 4. Execute the docker exec command as new user.

I am able to ssh into container.

[braian125]

use /bin/sh instead of /bin/bash

On Windows CMD (after switching to bash) docker exec -it //bin//sh

On Git Bash winpty docker exec -it //bin//sh

[prateekn]

getting the same error with sh as well

➜  ~ kubectl -n kube-system exec -ti coredns-67fd67489b-9q8r5  -- /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:344: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory": unknown
command terminated with exit code 126
➜  ~ kubectl -n kube-system exec -ti coredns-67fd67489b-9q8r5  -- /bin/sh  
OCI runtime exec failed: exec failed: container_linux.go:344: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory": unknown
command terminated with exit code 126