[BUG]: NVMeTCP Linux requirements and Best Practices need to be documented and/or incorporated into CSI drivers

[dancohen21]

Bug Description

There are a collection of improvements required to support NVMe TCP as documented in the following documentation that are required by for high availability and performance that are currently not reflected in the CSI/CSM documentation or missed by the automation functions that the CSI driver, and associated libraries, provides.

referenced documenation:

• Dell Host Connectivity Guide
• NVMe/TCP Boot from SAN with Dell PowerStore Storage Arrays

1. The Dell Linux host connectivity guide recommends on page 214 https://elabnavigator.dell.com/vault/pdf/Linux.pdf?key=1725374107988

By default, the Linux controller enters a reconnect state when it loses connection with the target. The default timeout for reconnecting is 10 minutes. However, a PowerStore node reboot may take more than 10 minutes. It is recommended to set ctrl-loss-tmo = -1 to keep the controller constantly reconnecting.

Consider, that this ctrl-loss-tmo = -1 parameter can be applied easily using Linux UDEV rules. /etc/udev/rules.d/72-nvmf-ctrl_loss_tmo.rules ACTION=="add|change", SUBSYSTEM=="nvme", KERNEL=="nvme*", ATTR{ctrl_loss_tmo}="-1"

this configuration can be applied to an OpenShift cluster via machine config with

apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 99-nvmf-ctrl_loss_tmo spec: config: ignition: version: 3.2.0 storage: files: - contents: source: data:text/plain;base64,QUNUSU9OPT0iYWRkfGNoYW5nZSIsIFNVQlNZU1RFTT09Im52bWUiLCBLRVJORUw9PSJudm1lKiIsIEFUVFJ7Y3RybF9sb3NzX3Rtb309Ii0xIgo= filesystem: root mode: 420 path: /etc/udev/rules.d/72-nvmf-ctrl_loss_tmo.rules

2. enable nvmf-autoconnect service | this ensures that if a OpenShift node reboots the node reconnects to the nvme-tcp storage targets

this configuration can be applied to an OpenShift cluster via machine config with

apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 99-start-nvmf-autoconnect spec: config: ignition: version: 3.2.0 systemd: units: - name: nvmf-autoconnect.service enabled: true state: started

3. The CSI documentation should recommend customers use the round-robin io-policy for nvme-tcp and nvme-fc in accordance with the Host connectivity guide. The CSI documentation should provide a working udev file and associated machineconfig to apply this configuration.

example from NVMe/TCP Boot from SAN with Dell PowerStore Storage Arrays

To set the host server I/O policy to round-robin for NVMe devices, complete the following procedure.

Note: The subsystem number may be different for your host server. For example, this document uses nvme-subsys0. Verify the correct subsystem number for your host server.

Run the following commands to change the I/O policy from NUMA to round-robin for the specified nvme-subsys.

cat /sys/class/nvme-subsystem/nvme-subsys0/iopolicy numa echo round-robin > /sys/class/nvme-subsystem/nvme-subsys0/iopolicy cat /sys/class/nvme-subsystem/nvme-subsys0/iopolicy round-robin Create udev rules so any newly added NVMe devices use round-robin. cd /etc/udev/rules.d vi 71-nvmf-iopolicy-dell.rules ACTION=="add", SUBSYSTEM=="nvme-subsystem", ATTR{model}=="dellemc-powerstore",ATTR{iopolicy}="round-robin" /sbin/udevadm control --reload-rules /sbin/udevadm trigger --type=devices --action=change

4. is /etc/nvme/discovery.conf required to be populated?

Logs

none

Screenshots

No response

Additional Environment Information

https://bugzilla.redhat.com/show_bug.cgi?id=1984987

Steps to Reproduce

to reproduce with default settings.... shutdown a PowerStore Node to simulate a PowerStore node being unavailable during a PowerStore non-disruptive code upgrade (NDU). Wait 13 minutes and while monitoring the output of 'nvme list-subsys' you will see the host stop trying to connect to the unavailable path

Expected Behavior

with the ctrl-loss-tmo = -1 configuration parameter applied, the host will never stop reconnecting to the storage system

CSM Driver(s)

All versions using nvme-tcp

Installation Type

csm-operator 1.5.1

Container Storage Modules Enabled

No response

Container Orchestrator

OpenShift 4.14

Operating System

OpenShift Linux - RHCOS based on RHEL 9.2