Simplify node state tracking model

[invidian]

After looking at #37 and #30, I realized that existing FLUO model for tracking at what stage of the reboot process each node is is overly complex and perhaps could be simplified.

Right now, pkg/constants identifies 3 different labels used:

        LabelRebootNeeded = Prefix + "reboot-needed"
        LabelBeforeReboot = Prefix + "before-reboot"
        LabelAfterReboot = Prefix + "after-reboot"

And 5 different annotations:

        AnnotationRebootNeeded = Prefix + "reboot-needed"
        AnnotationRebootInProgress = Prefix + "reboot-in-progress"
        AnnotationOkToReboot = Prefix + "reboot-ok"
        AnnotationRebootPaused = Prefix + "reboot-paused"
        AnnotationAgentMadeUnschedulable = Prefix + "agent-made-unschedulable"

I think we should drop use of annotations, as they are meant to be used for carrying 3rd party node-unique information or property, while labels are meant to be used for handling selectable data. This will allow operator and agent to operate only on labels, which should simplify the process of selecting specific nodes.

Another thing to simplify would be to make use of label's value to reduce the amount of labels in use. As a rule of thumb, every distinct state source should have it's own label assigned for publishing their value, so they can be used in parallel if needed.

I identify the following state sources:

• update_engine - If we want to be able to cancel the reboot process at every stage, each stage must take update_engine status into account. Simply put: If we run before-reboot hooks and node crashes and reboots itself, we will not drain the node, repeat the reboot and run after-reboot hooks. Update engine status also presents node-point of view.
• operator - Right now operator decides which nodes will be rebooted, which runs which hooks etc, so operator must be able to track progress of those operations. However, from operator point of view, node (agent) can only be in single state at a time:
  • Idle
  • Paused (issued by user)
  • Running before-drain (currently before-reboot) hooks
  • Running before-reboot hooks (#30)
  • Reboot in progress (confirming before-reboot hooks executed successfully and allow to proceed to before-uncordon when reboot is no longer needed)
  • Running before-uncordon (currently after-reboot) hooks

Each of those stages happens one after another, so operator should be able to select all nodes from state X, check appropriate conditions for them (right now extra annotations + probably update_engine status) and then promote them to the X+1 stage, streamlining the reboot process.

With such strategy, label will be available for scheduling hook daemons using labels for appropriate nodes.

With new model, primarily 2 labels will be used + user defined labels/annotations for confirming stages when needed via hooks. Hooks can then do all sorts of integrations, similar to kured (#72) like sending slack notifications, waiting for volumes detachment etc etc.

Perhaps AnnotationAgentMadeUnschedulable annotation can stay as well as a safety knob for cordoning and uncordoning the node, but this remains an implementation detail of the agent.

[invidian]

update_engine - If we want to be able to cancel the reboot process at every stage, each stage must take update_engine status into account. Simply put: If we run before-reboot hooks and node crashes and reboots itself, we will not drain the node, repeat the reboot and run after-reboot hooks. Update engine status also presents node-point of view.

Perhaps this could be abstracted by the agent state, so we can also include information like acknowledging approval of reboot.