CSI plugin is not deregistered when a new version of the plugin job changes the CSI plugin id

[Jamesits]

Nomad version

1.7.6

Operating system and Environment details

Ubuntu 22.04

Issue

When a CSI job has a new version with a different CSI plugin id, Nomad is unable to unregister the storage plugin with the old id cleanly.

Reproduction steps

0. Submit a working CSI storage plugin job
1. Submit a new version of the job with job.group[].task[].csi_plugin.id changed directly

Expected Result

Web UI -> Storage -> Plugins shows only the new id

Actual Result

Web UI -> Storage -> Plugins have both the old id and new id showing up, and you have no way to remove the old one

I've tried changing the CSI plugin id back, stopping all the plugin jobs, purging the related jobs completely. It worked for one cluster, but does not on another cluster of the same configuration (created with near identical Terraform plans). This might be caused by an unknown issue.

Job file (if appropriate)

N/A

Nomad Server logs (if appropriate)

N/A

Nomad Client logs (if appropriate)

N/A

Related https://github.com/hashicorp/nomad/issues/7306

[jrasell]

Hi @Jamesits and thanks for raising this issue. It looks like we are not correctly handling the ID update, to perform destructive actions on the original CSI ID. I'll add this to our roadmap.