Unable to deploy cinder-csi-plugin 1.21 with kops >=1.24

[ederst]

/kind bug

1. What kops version are you running? The command kops version, will display this information.

1.24.0-beta.3

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

1.21.x

3. What cloud provider are you using?

openstack

4. What commands did you run? What is the simplest way to reproduce this issue?

Set kubernetes version to 1.21.x, use the external cloud provider and CSI drivers (enabled per default i guess) and rollout a cluster.

5. What happened after the commands executed?

The CSI cinder plugin throws the following error:

cinder-csi-plugin Error: required flag(s) "nodeid" not set
cinder-csi-plugin Usage:
cinder-csi-plugin   Cinder [flags]
cinder-csi-plugin
cinder-csi-plugin Flags:
cinder-csi-plugin       --add_dir_header                   If true, adds the file directory to the header of the log messages
cinder-csi-plugin       --alsologtostderr                  log to standard error as well as files
cinder-csi-plugin       --cloud-config string              CSI driver cloud config
cinder-csi-plugin       --cluster string                   The identifier of the cluster that the plugin is running in.
cinder-csi-plugin       --endpoint string                  CSI endpoint
cinder-csi-plugin   -h, --help                             help for Cinder
cinder-csi-plugin       --log-flush-frequency duration     Maximum number of seconds between log flushes (default 5s)
cinder-csi-plugin       --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
cinder-csi-plugin       --log_dir string                   If non-empty, write log files in this directory
cinder-csi-plugin       --log_file string                  If non-empty, use this log file
cinder-csi-plugin       --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
cinder-csi-plugin       --logtostderr                      log to standard error instead of files (default true)
cinder-csi-plugin       --nodeid string                    node id
cinder-csi-plugin       --one_output                       If true, only write logs to their native severity level (vs also writing to each lower severity level)
cinder-csi-plugin       --skip_headers                     If true, avoid header prefixes in the log messages
cinder-csi-plugin       --skip_log_headers                 If true, avoid headers when opening log files
cinder-csi-plugin       --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
cinder-csi-plugin       --user-agent stringArray           Extra data to add to gophercloud user-agent. Use multiple times to add more than one component.
cinder-csi-plugin   -v, --v Level                          number for the log level verbosity
cinder-csi-plugin       --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
cinder-csi-plugin

The cinder-csi-plugin version adapts to the kubernetes version, so it is missing the --nodeid argument, which is deprecated and (will be) removed in a later version. From another cluster of ours running k8s 1.22.x (and thus cinder-csi-plugin 1.22.x):

 cinder-csi-plugin Flag --nodeid has been deprecated, This flag would be removed in future. Currently, the value is ignored by the driver

6. What did you expect to happen?

I expected the cinder-csi-plugin to start successfully.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

IMHO not important, because there is nothing special going on.

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

See error above.

9. Anything else do we need to know?

I think the bug was introduced with this change: https://github.com/kubernetes/kops/commit/4ca48dcd5be0c7ece275078f6d8a2caac9117a3c (PR: https://github.com/kubernetes/kops/pull/13603, fyi @zetaab)

So maybe there needs to be some kind of logic in the template to make --nodeid appear for cinder-csi-plugin versions <1.22 or maybe a separate addon manifest for kubernetes clusters with versions <1.22. At least if kops 1.24.x should support those k8s versions.