Add liveness, readiness, and startup probes

[ammerzon]

The more complex the services get, the better we should ensure that all the individual components start up and run correctly. If we implement liveness, readiness, and startup probes, Kubernetes will help us to catch bugs and possibly schedule restarts in case of failures.

Definition of Done

• [ ] All services have liveness, readiness, and (if required) startup probes

Tasks

• [ ] Add healthz endpoints to HTTP-based services
• [ ] Add liveness probes to all pods that don't have them yet
• [ ] Add readiness probes to pods that are used in Kubernetes services
• [ ] Add startup probes for slow-starting services
• [ ] Implement open todo's

Additional context

• The user-auth service may need startup probes to initialize the database.
• See: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

[MfCrizz]

Idea: Also use the Health Checks provided by the AWS Load Balancer. There are annotations to configure them. See:

• https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/guide/ingress/annotations/#annotations