Resize NVMe sector size on snap-03

[Firefishy]

The NVMe disks in snap-03 are using 512 sectors instead of 4096 byte sectors. The sector size should likely be changed if boosts performance.

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format  2 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format  3 : Metadata Size: 8   bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format  4 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good

Changing the disk sector size formats the disk, the sector size needs to be carefully coordinated with mdadm.

Outline steps:

1. Confirm array idle via /proc/mdstats
2. mdadm --fail
3. mdadm --remove
4. nvme format --lbaf=2
5. mdadm --add

(repeat disk by disk across all disks)