[feature] automatically detect and apply configuration updates

[ddiss]

lrbd could use RADOS watch/notify functionality to automatically detect and apply configuration changes.

• change the one-shot systemd service to run "rados watch "
• ensure that all updates to the config object are atomic
• notify all watchers via "rados notify "
• apply the new configuration locally on notification

[swiftgist]

A couple of issues. The lrbd.conf object is written to each pool. So, there is technically more than one object, but we should be able to watch just one. Lrbd is not doing anything special rados-wise to guarantee that updates are atomic. I've been following the "admin makes changes, applies to first gateway, verifies correctness, applies to remaining gateways" and not thinking that iSCSI configuration changes were all that frequent. The last issue is that while lrbd handles additions and modifications well, renames of specific structures are an issue. I simply do not have all the logic in there to handle the reverse steps of the targetcli commands in those cases.

This could probably serve as a documented example though for those admins that are tweaking kernel parameters repeatedly and comfortable knowing that writing the configuration equates restarting the service.

[ddiss]

Thanks for the feedback, Eric. A few comments on the points you made:

• watching and handling notifications across multiple objects from the systemd service should be possible
• regarding applying/verifying a config on a single gateway node before roll-out: I agree it'd be unfortunate to lose this ability, so maybe a -no-notification argument would be suitable here
• for structure renames etc, it'd probably be good to detect and warn admins if a config change can't be applied without reboot