for-aws
for-aws copied to clipboard
Healthcheck on 44554 doesn't detect hung/broken dockerd
Expected behavior
When dockerd hangs or is otherwise broken, the healthcheck service running on 44554 should report it is down so that AWS can restart the machine.
Actual behavior
The service on 44554 returns 'LGTM' despite:
- portainer claims the swarm is down - 'endpoint is unreachable'
- port 22 is not connectable - suggesting that the shell-aws container is down
- port 2375 connects but times out
- port 2376 is not connectable
Information
docker-diagnose on a worker produces no output.
docker info on a worker looks ok, it still reports the manager's IP address
Investigation is restricted due to not being able to ssh in to the manager machine.