Add support for reloading application running in a container instead of rolling update

[rasheedamir]

Suggested by Marton Szucs in slack channel:

For applications that supports updating configuration at runtime, like pgbouncer or nginx, it would be nice to just reload the application inside a running container. Instead of restarting the whole Pod using a rolling update.

This is how I do it manually:

• Change configmap/secret and apply it to the cluster
• Exec into the container that has mounted the configmap/secret as a file.
• Check if the mounted file is updated inside the container
• reload application with kill -SIGHUP 1 or nginx -s reload or some other application specific command.

Questions:

• Will it be safe?
• Look into how does nginx ingress controller does it right now? It indeed never does rolling update but does have new configs!

[rasheedamir]

Yes it will be awesome to have this feature in place! e.g. in case of fluentd assume we have 50 nodes then it will take too much for rolling update of all pods in the daemonset; but if have restart of process then it will happen at sametime in all

[waseem-h]

Here's how vmware does it in their fluentd operator: https://github.com/vmware/kube-fluentd-operator/blob/master/config-reloader/fluentd/reloader.go#L26

Basically we have to enable rpc endpoint for fluentfd as mentioned here: https://docs.fluentd.org/v1.0/articles/rpc#configuration

In general, we can add support for reloading applications via RPC endpoint

• An application will contain annotation with the RPC endpoint to call
• Instead of rolling update, we can call the RPC endpoint mentioned (in all pods)

[grzesuav]

Question : how it will guarantee HA of application? Will it respect deployment settings during restarting?

[mrmuli]

I would also like to put in consideration for applications with required bootstrapping on start or restart. Unless this will be implemented towards very specifically defined apps or those running behind process managers. I'm not sure how this would work out, seems very diverse?

[ankon]

We're using https://github.com/weaveworks/watch for this purpose:

1. Mount the configmap as volume into a side-car running watch
2. Watch the mount point/files in it
3. Trigger the reload by issueing a curl command to the actual app