NAS-128368 / 24.10 / nvme-of: Remove/Add disk on discovery change event

[ixhamza]

Jira Ticket: NAS-128368

NAS-128368 reports three different issues related to NVMe-oF hotplugging:

Problem:

NAS-128368 reports three different issues related to NVMe-oF hotplugging:

1. When a drive is physically removed while part of the pool, the Linux/zfs still sees it as ONLINE.
2. A drive is not visible to the OS upon insertion until the next drive is inserted, i.e., drive addition by OS is lagged by one event. For example, if a drive is inserted into Slot-1, it won't be added by Linux until another drive is inserted into a different slot. However, the addition of the next drive will then be lagged by the drive inserted after that.
3. If fenced is holding a drive, the new drive still points to the previous NVMe subsystem, showing the previous serial number of the last drive inserted.

Solution:

1. The discovery controller receives the udev event, and by default, the NVMe-oF service runs the connect-all command, which is insufficient to remove the drive. To address this, we check the drives that are part of the enclosure and disconnect those not matched with the discovery log page.
2. It takes up to 3 seconds after receiving the discovery event for the drives to be ready for connection on the client side. To accommodate this, we added a timeout of five seconds to ensure the drive connects properly.
3. This issue is fixed by a Linux kernel patch: https://github.com/truenas/linux/pull/178. We now create a new subsystem if the previous one does not have any valid controllers.

Additionally, by default, systemd does not queue the events and ignores the service start if the service is already running. To resolve this, we added a simple queue to handle at most one event and created a wrapper to unblock systemd for the next udev events.

[ixhamza]

@yocalebo - I have added a detailed explanation.

[yocalebo]

@ixhamza thanks! I'll start review here soon.

[ixhamza]

@yocalebo - would be appreciated if you could review this, please.

[bugclerk]

This PR has been merged and conversations have been locked. If you would like to discuss more about this issue please use our forums or raise a Jira ticket.