immich icon indicating copy to clipboard operation
immich copied to clipboard

[Bug]: Handle SIGTERM

Open bt90 opened this issue 2 years ago • 5 comments

The node processes started by Immich currently don't handle the SIGTERM signal. This is required to let Docker gracefully shutdown the containers.

Ideally this should close any open connections and also be logged.

bt90 avatar Feb 06 '23 15:02 bt90

Affected:

  • [ ] machine-learning
  • [ ] microservices
  • [ ] server
  • [ ] web

bt90 avatar Feb 06 '23 16:02 bt90

Just a comment to give this issue a bit more attention. Especially for my Podman installation where I run containers using systemd, I have to hack a bit to manage the units.

Maybe this is a solution: https://stackoverflow.com/questions/43584760/gunicorn-graceful-stopping-with-docker-compose (for ML)

For the services, catching the signals in the script might work?

I'm just thinking out loud here. I'm a mere beginner when it comes to containers.

bertmelis avatar Mar 22 '23 09:03 bertmelis

The container itself should already be setup correctly. It's just the node logic which is missing here.

bt90 avatar Mar 22 '23 10:03 bt90

You mean the containers itself stop with SIGTERM but the business logic isn't gracefully aborted?

Then I have a bug in my setup because some of my containers still fallback to SIGKILL after the standard 10s timeout.

bertmelis avatar Mar 22 '23 11:03 bertmelis

I should phrase it differently. The signal from the docker daemon is correctly handed to the node process which ignores it at the moment. The daemon ultimately waits 10s until it kills the processes in a non-graceful manner using SIGKILL.

tl;dr immich processes need to handle SIGTERM

bt90 avatar Mar 22 '23 11:03 bt90

https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#handling-kernel-signals

Looks like we need to add an init process to our docker containers. Maybe we can get away with a simple change to our compose file?

https://docs.docker.com/compose/compose-file/05-services/#init

bt90 avatar Jul 20 '23 13:07 bt90