locksmith: document how to do health checks and draining

[philips]

Draining Work Before Reboot

For a variety of reasons a user of CoreOS may want to run a program before rebooting. Perhaps to drain work off of the machine, to remove the machine from DNS, or to tell a load balancer to stop sending traffic.

This is as simple as writing a systemd unit file that does something like:

[Unit]
Description=Drain the host
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target


[Service]
ExecStart=/usr/bin/rkt run quay.io/example/drain-lb
Type=oneshot

You can run any arbitrary program in the ExecStart line, it just must exit.

Health Locksmith Unlocks

For a variety of reasons you may want to do a health check of something before locksmith unlocks. Perhaps you want to ensure that your machine reconnects to the kubernetes API, can reach a service discovery system, etc.

This is documented in the mailing list post here:

https://groups.google.com/forum/#!searchin/coreos-dev/locksmith/coreos-dev/onuyn-72UYs/ELXfxc3Ax-oJ

[mischief]

what are you supposed to do if this extra before-unlock check fails?

[philips]

@mischief Don't fail. It should run until it successfully does its work.

[robszumski]

There are some Tectonic use-cases that make sense for this as well. For example, if you accidentally booted an old version of CoreOS and we're executing an install workflow, we need to inhibit the update until that process completes.

[jeevarathinam-dhanapal]

It didn't work for me.

Created a service and tried commands "reboot/systemctl reboot/systemctl reboot.target". Below service was not triggered before the actual reboot

[Unit]
Description=Drain K8s pod in the node
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target

[Service]
ExecStartPre=command1
ExecStart=command2
Type=oneshot

Tried removing "DefaultDependencies" and added install section with wantedBy. But coreos reboot doesn seem to triggere this service before reboot

[jeevarathinam-dhanapal]

1. After/Before are not hard dependencies. It not necessary the service has to run successfully. These should be accompanied with requires/requiredby
2. Even with RequiredBy in the “host drain” service, systems not running the service. It’s because the reboot.target already has its own “Requires”. Before the “host drain” is executed system is already shutdown

The best approach would be creating dropins for shutdown.target with “Requires/After” pointing to “host drain” service