Dynamic Host Volumes Deferred Binding

[ahzay]

Proposal

Dynamic Host Volumes (#15489) could allow for deferred binding (similar to CSI create/publish distinction) by adding an optional flag in the fingerprint. If the plugin's fingerprint.deferred_bind (or similar) is enabled, then the create/delete RPC calls to the DHV plugin would only happen during the task pre/post hooks. "Creating" the volume in nomad would only store it in the state, and not actually call the plugin binary.

Use-cases

Dynamic Host Volumes currently always bind to a node during create. This is not suitable for plugins backed by remote storage. Binding during create makes it pretty much useless to build those kinds of network-backed plugins, for which volumes have no reason to be bound to any specific node until scheduling time. I find this to be a very desirable application for DHV because it is much simpler than CSI, it would be much easier to maintain a wide variety of plugins, networked or otherwise. Also, having this deferred bind "mode" hard-coded at the fingerprint level means simpler plugin code, a plugin dev could have two binaries, one for each mode.

Attempted Solutions

Have not attempted a solution yet. I have been looking at the codebase to figure out where the best place would be to insert these deferred calls, so far I'm thinking around somewhere in the task hooks, perhaps creating a new hook similar to the csi_hook. I would appreciate any input from people more familiar with it.