Nomad masks / confuses error from CSI plugin

[sorenisanerd]

When any of ControllerPublishVolume, NodePublishVolume, and NodeStageVolume returns a FailedPrecondition error from a CSI plugin, it gets logged with an error message derived from the CSI spec. Example:

2024-08-23T16:32:05-07:00  Setup Failure    failed to setup alloc: pre-run hook "csi_hook" failed: mounting volumes: volume "85cd0978-05bf-11ef-bd6d-0242c0a82802" is already published on another node and does not have MULTI_NODE volume capability: rpc error: code = FailedPrecondition desc = volume "85cd0978-05bf-11ef-bd6d-0242c0a82802" is already staged at [/local/csi/staging/vol-madeupid1/rw-file-system-single-node-writer]

The spec certainly says that in case of a call to PublishVolume for an already mounted volume etc. etc., the correct return code is FailedPrecondition. The spec doesn't actually say that FailedPrecondition is only for that particular error condition, although it's a little awkwardly phrased. Regardless, a FailedPrecondition response to NodePublishVolume or NodeStageVolume means something entirely different and the error shown to users is confusing.