After Docker daemon restart, containers do not gracefully recover

[greenmoss]

Describe the problem

• Bring up st2-docker
• Restart the Docker service
• Some of the containers do not recover properly, and throw stack traces

Versions

• Host OS: Ubuntu 20.04
• docker: version 20.10.2, build 2291f61
• docker-compose: version 1.29.2, build 5becea4c
• stackstorm version: 3.6.0

To Reproduce Bring up cluster docker-compose up -d

Look for exited containers

docker-compose ps -a | grep Exit
st2-docker_st2chatops_1            /st2chatops-startup.sh           Exit 0
st2-docker_st2makesecrets_1        /makesecrets.sh                  Exit 0

Stop/restart docker service service docker restart

Look for exited containers

docker-compose ps -a | grep Exit
st2-docker_mongo_1                 docker-entrypoint.sh mongod      Exit 0
st2-docker_rabbitmq_1              docker-entrypoint.sh rabbi ...   Exit 0
st2-docker_redis_1                 docker-entrypoint.sh redis ...   Exit 0
st2-docker_st2actionrunner_1       /opt/stackstorm/st2/bin/st ...   Exit 0
st2-docker_st2chatops_1            /st2chatops-startup.sh           Exit 0
st2-docker_st2makesecrets_1        /makesecrets.sh                  Exit 0
st2-docker_st2scheduler_1          /opt/stackstorm/st2/bin/st ...   Exit 0
st2-docker_st2web_1                /bin/bash -c if [ ${ST2WEB ...   Exit 0
st2-docker_st2workflowengine_1     /opt/stackstorm/st2/bin/st ...   Exit 0

Follow/tail Docker logs

docker-compose logs -f --tail 20

Note all the errors. It seems all of them are some variant of pymongo.errors.ServerSelectionTimeoutError: mongo:27017: [Errno -2] Name or service not known.

To recover:

docker-compose down
docker-compose up -d

[stevemuskiewicz]

@greenmoss Although I have encountered the same issue, I'm not sure there is a whole lot that can be done here. The problem is that the (standalone) docker service does not have any knowledge of the dependencies/startup order between containers (as defined in docker-compose.yml), it only knows it has N containers that were running before and need to be restarted.

I'm sure there is some additional tooling out there that could be used to integrate the docker-compose files into the container startup process, but this is definitely not the "out of the box" behavior.

Probably also why the readme for this repo states:

This docker-compose is provided as a way to allow "get up and running" quickly with StackStorm using Docker (based on st2-dockerfiles). It is not designed to be used in production, but rather a way to test out StackStorm and facilitate pack development.