openzfs
Attach Multiple Devices via zpool attach
Describe the feature would like to see added to OpenZFS
I'd like to see the zpool attach command expanded to allow more than one device to be attached at the same time. For example:
zpool attach tank disk1 disk2 disk3
Would add disk2 and disk3 to disk1 to form a three-way mirror from what was previously a single disk.
How will this feature improve OpenZFS?
While very much a niche feature, it would avoid the need to run zpool attach once for each disk being added, and if it can be done atomically (i.e- add all disks then begin resilvering) would ensure that all new devices start from the same point in the resilvering process compared to a second, third etc. disk being added later.
This would be an improvement especially in cases where multiple disks are being added to increase capacity and/or redundancy.
Additional Context
This should not break the current constraints on the existing command, as the device to attach to must still come first, and the same checks can be performed for each new device to ensure they are not already in use, have enough capacity etc., failing if one or more do not pass all checks.
To give an example use case, until recently I had a pool consisting of two concatenated 3tb disks (no redundancy) as I didn't have the physical space to add more drives. After adding some additional bays for hard drives, I added a pair of 4tb drives. Since these have higher capacity it didn't make sense to just add one to each existing disk, as they'd still be limited to 3tb each, so I attached both 4tb disk to one 3tb drive, and once resilvering was complete I detached the 3tb drive and attached it to the other 3tb drive to create 2x3tb + 2x4tb pair of mirrors as the final result. So the process looked something like:
tank
3tb1
3tb2
tank
3tb1
mirror-1
3tb2
4tb1
4tb2
tank
3tb1
mirror-1
4tb1
4tb2
tank
mirror-2
3tb1
3tb2
mirror-1
4tb1
4tb2
Since I now have several spare bays I could see myself wanting to do the same again if I decide to upgrade capacity by swapping the 3tb drives for more 4tb (or larger) drives, in which case it would be a lot easier to simply add both 4tb drives to mirror-2 at the same time then detach the 3tb drives once resilvering is complete, as opposed to replacing each drive individually (which temporarily reduces redundancy and has a bigger performance hit).
To be clear, not saying that adding the drives individually is a major burden or anything, and I know that ZFS' resilvering should cope with the drives being added at different times in the process, it just seems like it would be neater to be able to do this atomically with a single command before starting the resilver.
tbh i think this would just be a nice-to-have if we could have resilver_defer improvements like #14505, so it would just cleanly start them all rather than queuing them up at 0.01% complete.
It'd be nice, but also probably inconvenient to implement cleanly, as the older ZFS commands are gross, so implementing this in a way where the older kernel modules wouldn't silently ignore additional attach arguments would be somewhat challenging.
I suppose you could also just implement it as a wrapper around a for loop for issuing multiple attaches and then a forced zpool resilver at the end to make them all resilver at once, but that could have weird knock-on effects with unexpected outcomes without something like #14505 implementing that without weirdness.
tbh i think this would just be a nice-to-have if we could have
resilver_deferimprovements like #14505, so it would just cleanly start them all rather than queuing them up at 0.01% complete.
To be honest I'd be happy just to not get the weird double resilver I encountered in my use case; after I added the second disk (soon after the first, matter of seconds at most) both disks showed as resilvering with the same write activity so I just assumed that was fine and when it finished they'd both be done, but I ended up with a deferred resilver on the second disk (which took a weirdly long time to really start in earnest, like 8 hours before it got up to speed) and it just ended up doing another full resilver to that disk so there was no benefit at all to adding it early.
Really it's my own stupid fault and I should have a) run zpool attach -s because I scrubbed the pool recently already so the faster sequential resilver/copy followed by a scrub should have been fine, and b) once I realised I couldn't add two at once I should have just waited for the first before adding the second, but I didn't realise it would end up doubling up like this, I guess I assumed it would either do a bit of extra resilvering on the second drive, or initiate a scrub to check it.
But it definitely feels like ZFS should be able to handle adding two disks at once a lot better; while it's a bit niche, in a system with bays to spare it's way easier to upgrade a pair by adding a whole new one to the same mirror then detaching the old disk(s), plus it maintains redundancy and most of the performance that way compared to replacing one at a time.
It'd be nice, but also probably inconvenient to implement cleanly, as the older ZFS commands are gross, so implementing this in a way where the older kernel modules wouldn't silently ignore additional attach arguments would be somewhat challenging.
Could the multiple arguments be passed with some kind of separator? Older kernel modules wouldn't know to split that so would just see an extra long device name it doesn't recognise, and this wouldn't affect the case of sending a single argument which would work as expected?
It used to. But then someone thought it would be good to improve it so it didn't restart when you were 99% done resilvering one disk, and did it very badly.
This isn't really related, but it made me think of (my) this other issue report #14268 (which took a lot of time to find among 1100 other issues).
