Docker inspect hangs, docker exec hangs and docker kill doesn't work

[mmontagna]

• [x] This is a bug report
• [ ] This is a feature request
• [x] I searched existing issues before opening this one

Expected behavior

• docker inspect does not hang and returns information.
• docker exec succeeds or fails and doesn't hang.
• docker kill succeeds or fails and doesn't hang

Actual behavior

docker inspect, docker exec and docker kill hang when run on some containers.

Steps to reproduce the behavior

We are seeing this happen occasionally on some of our servers on random containers.

We initially thought it might be related to #38478 (https://github.com/moby/moby/issues/38478#issuecomment-451526797) but this probably a different issue as we don't see any permission denied messages when attempting to kill the container the command just hangs.

If we kill the containerd-shim process the container stops and the docker daemon appears to recover.

We also do not believe it is related to https://github.com/moby/moby/pull/37831 as our kernel version is 4.4.0-1069-aws which shouldn't be effected?

Output of docker version:

Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:57 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 5
 Running: 4
 Paused: 0
 Stopped: 1
Images: 10
Server Version: 18.09.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: syslog
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-1069-aws
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.42GiB
Name: ecs-XXX
ID: 72SQ:FNFB:BAFB:TAJB:D6FI:MKJ7:ZYXR:X6JX:PZMP:TLU7:RSQW:TWQK
Docker Root Dir: /tmp/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, etc.)

This is on AWS EC2/ECS hosts.