cli
cli copied to clipboard
docker
restart policy not working when using nodejs
Description
Docker doesn't follow the restart policy (always and unless-stopped) when a nodejs script exit on an error. I tried the same with python, and it works correctly.
Reproduce
- use following
script.js,Dockerfileanddocker-compose.ymlfile (docker compose is used for convenience, but it changes nothing to the behavior)
// script.js
setTimeout(() => {
console.log("Exiting with error code 1");
process.exit(1);
}, 20000); // this timer purpose is to be sure that the script throw an error after the initial time window where docker doesn't consider the script started correctly (time window of 10 second according to what I've read on forums, but I can't find the info in the official documentation
# Use the official Node.js 14 image
FROM node:14
# Set the working directory in the container
WORKDIR /usr/src/app
# Copy the script file into the container
COPY script.js ./
# Command to run when the container starts
CMD ["node", "script.js"]
version: '3.8'
services:
myscript:
build: .
image: myscript-image
restart: always
- run
docker compose up
Expected behavior
docker should restart automatically after the exiting of the node script. Instead of this, we see the following output log
Attaching to app-myscript-1
app-myscript-1 | Exiting with error code 1
app-myscript-1 exited with code 0
When I manually check the script exit code outside of docker (by executing node script.js ; echo $?), I get 1. But inside of docker, the error seems to not be propagating correctly.
docker version
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:08:15 2023
OS/Arch: linux/arm64
Context: default
docker info
Client: Docker Engine - Community
Version: 24.0.7
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 5
Running: 3
Paused: 0
Stopped: 2
Images: 15
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc version: v1.1.10-0-g18a0cb0
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-rpi6-rpi-v8
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 3.704GiB
Name: raspberrypi
ID: 7cc59d5d-4493-4dbc-8cd6-6bef4680b6ec
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No memory limit support
WARNING: No swap limit support
Additional Info
Thank you for your help. I will gladly answer to any requests. I haven't tried yet to try this setup with another version of docker engine.
Thanks for reporting; Are you seeing the same if you run the container without compose (using docker run and setting the equivalent options (--restart=always) ?
Thanks for reporting; Are you seeing the same if you run the container without compose (using
docker runand setting the equivalent options (--restart=always) ?
Thank you for your reply !
Yes, I runned this :
docker build -t myscript-image .
docker run --restart always --name myscript-container myscript-image
I just tried sudo docker inspect myscript-container -f '{{.State.ExitCode}}' and it returns me 1, which is correct. But the container still doesn't restart.
Here are the most relevant part of the inspect json :
[
{
"Id": "7627dbe3d70a679ebd69cc13bcefb94aae0fe845eee22f188e7f821c03bcab8f",
"Created": "2024-01-11T21:56:40.259185173Z",
"Path": "docker-entrypoint.sh",
"Args": [
"node",
"script.js"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 23743,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-01-11T21:57:23.459691613Z",
"FinishedAt": "2024-01-11T21:57:22.857571257Z"
},
"Image": "sha256:e65590cf5094cc67d3379dca4010d154781b38e88dd86dd82d83e45a521ce46e",
"ResolvConfPath": "/var/lib/docker/containers/7627dbe3d70a679ebd69cc13bcefb94aae0fe845eee22f188e7f821c03bcab8f/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/7627dbe3d70a679ebd69cc13bcefb94aae0fe845eee22f188e7f821c03bcab8f/hostname",
"HostsPath": "/var/lib/docker/containers/7627dbe3d70a679ebd69cc13bcefb94aae0fe845eee22f188e7f821c03bcab8f/hosts",
"LogPath": "/var/lib/docker/containers/7627dbe3d70a679ebd69cc13bcefb94aae0fe845eee22f188e7f821c03bcab8f/7627dbe3d70a679ebd69cc13bcefb94aae0fe845eee22f188e7f821c03bcab8f-json.log",
"Name": "/myscript-container",
"RestartCount": 2,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
27,
245
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware",
"/sys/devices/virtual/powercap"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"Mounts": [],
"Config": {
"Hostname": "7627dbe3d70a",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NODE_VERSION=14.21.3",
"YARN_VERSION=1.22.19"
],
"Cmd": [
"dumb-init",
"node",
"script.js"
],
"Image": "myscript-image",
"Volumes": null,
"WorkingDir": "/usr/src/app",
"Entrypoint": [
"docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {}
},
}
]
Oh, right, I THINK I see what's happening here; the container is started in "attached" mode. When the container exits, the CLI itself exits because the connection with the container is terminated. But if you run docker ps, you'll see that the container is still running;
docker run --restart always --name myscript-container myscript-image
Exiting with error code 1
In another shell;
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e04390152e5 myscript-image "docker-entrypoint.s…" 20 seconds ago Up 19 seconds myscript-container
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e04390152e5 myscript-image "docker-entrypoint.s…" 22 seconds ago Up Less than a second myscript-container
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e04390152e5 myscript-image "docker-entrypoint.s…" 24 seconds ago Up 3 seconds myscript-container
In that case you can re-attach to the container with docker container attach (which will then exit again when the container quits);
docker container attach myscript-container
Exiting with error code 1
I guess technically it might be possible for an option that makes CLI continue polling for the container to re-appear (not sure what the best approach for that would be though).
Indeed ! Thank you so much ! I've tried (with docker CLI and docker compose), and both way correctly handle the restart in detached mode. I haven't thought about this at all ! I usually use "attach" mode when I'm doing prototyping.
Maybe it could be helpful to add this behavior in the documentation of docker attach and docker restart policy (here and here).
I am still not sure why it does restart correctly with docker compose when using python though
Thank you again :)
Great that we found the "cause" of what you saw.
I gotta be honest that I never really thought about this case. Probably also because in most (if not all) cases I run an "attached" container, I don't set a restart policy (attached containers usually are either "interactive" (run a shell in a container, run some commands, and exit), or a "one-off" task (run a container with a command, have the command run, and exit the container once it completes).
With compose I can see this being more common, because in that case you may have set a restart policy (must continue running), but if you "run" the container manually / attached for debugging, of course the same options are applied. For compose, perhaps it makes sense to have a "re-attach" option or something along those lines.
And perhaps it makes sense to have something similar for docker run, although we'd have to think about that (what would happen with an interactive container; you're in a shell inside the container, and it exits; can we successfully re-attach and continue where we left off?)
I agree that we should consider adding some mention of this in the documentation (the cli may exit / detach, but the container will continue running / restarted in the background (detached)).
Let me at least /cc @dvdksn for the docs
I see, it's interesting. I've learned a lot about docker through this :).
Btw, I just realized I was wrong earlier when I said that the restart policy was working with python. Actually, what happened is that the minimal setup I had that was using python was in a docker compose with another docker service , which explains why the shell was staying attached to the docker compose up command.
So there is no unexpected/asymetric behavior between node and python regarding the restart policy :).
Also, it means that by adding a dummy service in a docker compose, it would be possible to run docker compose up in attach mode and staying attached across restarts of other services.
Moreover, by writing this, I realize that docker compose is usually being used for orchestrating multiple services together, which means that the shell will continue to stay attach anyway, even without a dummy service. This dummy service would only be useful in case only one service is defined in the docker compose file.
Quick update; improvements to our docs were just merged (and should be published soon);
- https://github.com/docker/docs/pull/19087
Thank you everyone ! It was really helpful to understand more this situation :)