zfs
zfs copied to clipboard
Published
20 hours ago •
openzfs
openzfs
Resilver does not stop when detaching relevant drive
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Debian |
| Distribution Version | Bullseye, 11.4 |
| Kernel Version | 5.15.49 |
| Architecture | x86-64 |
| OpenZFS Version | 2.1.5 |
Describe the problem you're observing
For some reason, a drive got promoted from a spare to a live drive. I don't know why (I think , but I'm not complaining about that.
[email protected]:~# zpool status -P
pool: data
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 20 09:56:55 2022
712G scanned at 741M/s, 488G issued at 507M/s, 27.2T total
946K resilvered, 1.75% done, 15:18:47 to go
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900347V-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805793B-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806164R-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806149E-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806972T-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805552E-part1 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S5B1NC0N400948T-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806011F-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806005X-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900484P-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806988M-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806133J-part1 ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900225W-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805792M-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805805V-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806932Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805600Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805995T-part1 ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
spare-0 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S5B1NC0N400960Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900896W-part1 ONLINE 0 0 0 (resilvering)
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900476X-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900226A-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806076Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806153Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900558E-part1 ONLINE 0 0 0
spares
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900896W-part1 INUSE currently in use
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H802575K-part1 AVAIL
errors: No known data errors
so I detach the drive and it looks like resilver is still running? AIUI zpool detach detached the device we were resilvering so there should be no resilvering going on.
[email protected]:~# zpool detach data /dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900896W-part1
[email protected]:~# zpool status
pool: data
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 20 09:56:55 2022
732G scanned at 748M/s, 503G issued at 514M/s, 27.2T total
946K resilvered, 1.81% done, 15:06:37 to go
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900347V ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805793B ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806164R ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806149E ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806972T ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805552E ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S5B1NC0N400948T ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806011F ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806005X ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900484P ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806988M ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806133J ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900225W ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805792M ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805805V ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806932Z ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805600Z ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805995T ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S5B1NC0N400960Z ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900476X ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900226A ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806076Z ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806153Z ONLINE 0 0 0
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900558E ONLINE 0 0 0
spares
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900896W AVAIL
ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H802575K AVAIL
errors: No known data errors
Even removing the device does not stop resilver:
[email protected]:~# zpool remove data /dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900896W-part1
[email protected]:~# zpool status -P
pool: data
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 20 09:56:55 2022
1.46T scanned at 851M/s, 1.13T issued at 661M/s, 27.2T total
946K resilvered, 4.16% done, 11:28:13 to go
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900347V-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805793B-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806164R-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806149E-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806972T-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805552E-part1 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S5B1NC0N400948T-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806011F-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806005X-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900484P-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806988M-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806133J-part1 ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900225W-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805792M-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805805V-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806932Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805600Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H805995T-part1 ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S5B1NC0N400960Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900476X-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900226A-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806076Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H806153Z-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H900558E-part1 ONLINE 0 0 0
spares
/dev/disk/by-ktname/ata-Samsung_SSD_850_EVO_2TB_S2RLNX0H802575K-part1 AVAIL
errors: No known data errors
Describe how to reproduce the problem
- Create a zpool with a spare
- Offline a live device
- Promote a spare to a live drive
- Online the device you previously offlined
- Detach the resilvering spare
- Observe that the old resilver is still active and the offlined-onlined device is awaiting for it's turn to resilver
Demo:
Click to expand
[email protected]:~# zpool offline data /dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508257X-part1
[email protected]:~# zpool status
pool: data
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 646G in 04:00:17 with 0 errors on Wed Jul 13 05:15:07 2022
config:
NAME STATE READ WRITE CKSUM
data DEGRADED 0 0 0
raidz1-0 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508256R ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505897W ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501310F ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505907M ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501125V ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501122E ONLINE 0 0 0
raidz1-1 DEGRADED 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501126H ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508263V ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505900A ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508257X OFFLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505904X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501341A ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501124K ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508245Y ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505917X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508222F ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505899X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505909D ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501121J ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505911H ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508255W ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508249D ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505906T ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501060W ONLINE 0 0 0
spares
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y AVAIL
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508239J AVAIL
errors: No known data errors
[email protected]:~# zpool status -P
pool: data
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 646G in 04:00:17 with 0 errors on Wed Jul 13 05:15:07 2022
config:
NAME STATE READ WRITE CKSUM
data DEGRADED 0 0 0
raidz1-0 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508256R-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505897W-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501310F-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505907M-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501125V-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501122E-part1 ONLINE 0 0 0
raidz1-1 DEGRADED 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501126H-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508263V-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505900A-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508257X-part1 OFFLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505904X-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501341A-part1 ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501124K-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508245Y-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505917X-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508222F-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505899X-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505909D-part1 ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501121J-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505911H-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508255W-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508249D-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505906T-part1 ONLINE 0 0 0
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501060W-part1 ONLINE 0 0 0
spares
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y-part1 AVAIL
/dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508239J-part1 AVAIL
errors: No known data errors
[email protected]:~# zpool replace data /dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508257X-part1 /dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y-part1
[email protected]:~# zpool status
pool: data
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 20 10:51:13 2022
11.1G scanned at 1.86G/s, 1.21G issued at 206M/s, 14.6T total
0B resilvered, 0.01% done, 20:34:35 to go
config:
NAME STATE READ WRITE CKSUM
data DEGRADED 0 0 0
raidz1-0 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508256R ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505897W ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501310F ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505907M ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501125V ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501122E ONLINE 0 0 0
raidz1-1 DEGRADED 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501126H ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508263V ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505900A ONLINE 0 0 0
spare-3 DEGRADED 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508257X OFFLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y-part1 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505904X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501341A ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501124K ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508245Y ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505917X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508222F ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505899X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505909D ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501121J ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505911H ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508255W ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508249D ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505906T ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501060W ONLINE 0 0 0
spares
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y INUSE currently in use
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508239J AVAIL
errors: No known data errors
[email protected]:~# zpool online data /dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508257X-part1
[email protected]:~# zpool status
pool: data
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 20 10:51:13 2022
58.9G scanned at 1.79G/s, 27.9G issued at 866M/s, 14.6T total
0B resilvered, 0.19% done, 04:53:35 to go
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508256R ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505897W ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501310F ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505907M ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501125V ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501122E ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501126H ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508263V ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505900A ONLINE 0 0 0
spare-3 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508257X ONLINE 0 0 0 (awaiting resilver)
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y-part1 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505904X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501341A ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501124K ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508245Y ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505917X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508222F ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505899X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505909D ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501121J ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505911H ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508255W ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508249D ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505906T ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501060W ONLINE 0 0 0
spares
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y INUSE currently in use
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508239J AVAIL
errors: No known data errors
[email protected]:~# zpool detach data /dev/disk/by-ktname/ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y-part1
[email protected]:~# zpool status
pool: data
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 20 10:51:13 2022
77.7G scanned at 1.41G/s, 41.5G issued at 772M/s, 14.6T total
0B resilvered, 0.28% done, 05:28:55 to go
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508256R ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505897W ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501310F ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505907M ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501125V ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501122E ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501126H ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508263V ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505900A ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508257X ONLINE 0 0 0 (awaiting resilver)
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505904X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501341A ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501124K ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508245Y ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505917X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508222F ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505899X ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505909D ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501121J ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505911H ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508255W ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508249D ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505906T ONLINE 0 0 0
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K501060W ONLINE 0 0 0
spares
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K505905Y AVAIL
ata-Samsung_SSD_860_EVO_2TB_S45KNB0K508239J AVAIL
errors: No known data errors
Include any warning/errors/backtraces from the system logs
Nothing relevant in dmesg
I am also having this problem.
Started replacement of a good drive with another good drive, detached the new good drive during resilver.
12h later, still resilvering.
I'm seeing the same issue. I took a drive in a raidz2 offline, replaced it and it started resilvering. I took the resilvering drive offline but it still insists the resilvering is continuing even though no drives shows as resilvering and any drive that would be is offline. So not quite sure what it is actually doing but it does seem to be reading from the other drives at least…just not sure what it is doing with that data afterwards.
| Type | Version/Name |
|---|---|
| Distribution Name | Debian |
| Distribution Version | Proxmox VE 7.2-11 |
| Kernel Version | 5.15.60-2-pve |
| Architecture | x86-64 |
| OpenZFS Version | 2.1.6-pve1 |
Same here on Proxmox VE 7.3-6 Kernel: SMP PVE 5.15.102-1 OpemnZFS Version: zfs-2.1.9-pve1
`root@pve2:~# zpool status pool: tank state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Mar 20 14:01:53 2023 1.47T scanned at 659M/s, 542G issued at 37.0M/s, 2.42T total 0B resilvered, 21.89% done, 14:51:02 to go config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
ata-ST4000VN008-2DR166_ZGY4MMRK-part1 ONLINE 0 0 0
logs
nvme-SanDisk_Extreme_Pro_500GB_182379420073-part4 ONLINE 0 0 0
cache
nvme-SanDisk_Extreme_Pro_500GB_182379420073-part5 ONLINE 0 0 0
errors: No known data errors `
Update:
After 3 days of resilvering towards nirvana, the status of the pool is back in order. I think you can live with it in case of need.
I also seem to have hit this issue on my system. I added a third drive to a two way mirror, then decided I wanted to use it for something else.
| Type | Version/Name |
|---|---|
| Distribution Name | Ubuntu |
| Distribution Version | 22.04.4 LTS |
| Kernel Version | 5.15.0-117-generic |
| Architecture | ppc64le |
| OpenZFS Version | zfs-2.1.5-1ubuntu6~22.04.4 |
I used sudo zpool offline corona ata-ST4000DM004-2CV104_ZTT1R5GF to first offline the drive, resulting in this state:
zpool status corona
pool: corona
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Jul 28 03:45:41 2024
1.08T scanned at 78.5M/s, 196G issued at 13.9M/s, 1.08T total
198G resilvered, 17.70% done, 18:39:43 to go
config:
NAME STATE READ WRITE CKSUM
corona DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-ST4000DM004-2CV104_ZTT0QQAT ONLINE 0 0 0
ata-ST4000DM004-2CV104_ZTT1B4TA ONLINE 0 0 0
ata-ST4000DM004-2CV104_ZTT1R5GF OFFLINE 0 0 0 (resilvering)
Then sudo zpool detach corona ata-ST4000DM004-2CV104_ZTT1R5GF to remove it, but the resilver is still chugging along:
zpool status corona
pool: corona
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sun Jul 28 03:45:41 2024
1.08T scanned at 75.9M/s, 279G issued at 19.1M/s, 1.08T total
198G resilvered, 25.20% done, 12:19:40 to go
config:
NAME STATE READ WRITE CKSUM
corona ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST4000DM004-2CV104_ZTT0QQAT ONLINE 0 0 0
ata-ST4000DM004-2CV104_ZTT1B4TA ONLINE 0 0 0
errors: No known data errors
I can also see a ton of unusual read disk activity with zpool iostat; this pool has a couple webroot directories for apache it in, and stopping apache doesn't stop the activity:
zpool iostat -vly 1 1
capacity operations bandwidth total_wait disk_wait syncq_wait asyncq_wait scrub trim
pool alloc free read write read write read write read write read write read write wait wait
----------------------------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
corona 1.08T 2.54T 423 0 195M 0 21ms - 15ms - - - - - 6ms -
mirror-0 1.08T 2.54T 423 0 195M 0 21ms - 15ms - - - - - 6ms -
ata-ST4000DM004-2CV104_ZTT0QQAT - - 202 0 90.4M 0 22ms - 15ms - - - - - 7ms -
ata-ST4000DM004-2CV104_ZTT1B4TA - - 221 0 104M 0 20ms - 14ms - - - - - 6ms -
----------------------------------------------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
