zpool cannot remove vdev

[gchmurka123]

zfs-2.1.4-1 zfs-kmod-2.1.5-1

when i try remove mirror-1 vdev i have error: cannot remove mirror-1: invalid config; all top-level vdevs must have the same sector size and not be raidz.

How to reproduce:

1. create pool with 1 mirror (2 part of hard drive), default ashift

# zpool create vservers mirror  wwn-0x5000cca097c66efd-part3 wwn-0x5000cca097c254b8-part3

# zpool status
  pool: vservers
 state: ONLINE
        vservers                          ONLINE       0     0     0
          mirror-0                        ONLINE       0     0     0
            wwn-0x5000cca097c66efd-part3  ONLINE       0     0     0
            wwn-0x5000cca097c254b8-part3  ONLINE       0     0     0

errors: No known data errors

# zpool list 
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
vservers  3.61T  3.25G  3.61T        -         -     0%     0%  1.00x    ONLINE  -

2. add a mirror-1 consisting of 2 files (no ashift set = default)

# dd if=/dev/zero of=/1 bs=1M count=1000
# dd if=/dev/zero of=/2 bs=1M count=1000

# zpool add vservers mirror /1 /2

# zpool status
  pool: vservers
 state: ONLINE
  scan: resilvered 16.0M in 00:00:01 with 0 errors on Thu Dec 22 10:39:07 2022
config:

        NAME                              STATE     READ WRITE CKSUM
        vservers                          ONLINE       0     0     0
          mirror-0                        ONLINE       0     0     0
            wwn-0x5000cca097c66efd-part3  ONLINE       0     0     0
            wwn-0x5000cca097c254b8-part3  ONLINE       0     0     0
          mirror-1                        ONLINE       0     0     0
            /1                            ONLINE       0     0     0
            /2                            ONLINE       0     0     0

errors: No known data errors

3. Try remove mirror-1

zpool remove vservers mirror-1
cannot remove mirror-1: invalid config; all top-level vdevs must have the same sector size and not be raidz.

My disk has:

        Logical  Sector size:                   512 bytes
        Physical Sector size:                  4096 bytes

my pool alocation:

# zpool list 
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
vservers  3.61T  3.25G  3.61T        -         -     0%     0%  1.00x    ONLINE  -

mu pool ashift:

# zpool get ashift vservers 
NAME      PROPERTY  VALUE   SOURCE
vservers  ashift    0       default

[gmelikov]

Please show zdb -C output, it will show actual ashifts.

[gchmurka123]

# zdb -C

    vdev_children: 2
    vdev_tree:
        type: 'root'
        id: 0
         children[0]:
            type: 'mirror'
            ashift: 12
             children[0]:
                type: 'disk'
                path: '/dev/disk/by-id/wwn-0x5000cca097c66efd-part3'
               children[1]:
                type: 'disk'
                 path: '/dev/disk/by-id/wwn-0x5000cca097c254b8-part3'
        
        
           children[1]:
            type: 'mirror'
            id: 1
            ashift: 9
             children[0]:
                type: 'file'
                path: '/1'
              children[1]:
                type: 'file'
                path: '/2'

vdev mirror-1 was created in files which stored in ext4 fs (default blocksize=4k), I don't understand why zpool used ashift=9

The expected behavior would be that when adding vdev to a pool, zpool would automatically match the best pool ashift. Only when the user forces ashift (zpool add -o ashift=x poolname devices) zpool should use another

[gmelikov]

zpool add will use best ashift per disk now, not per pool. So that's the reason, unfortunately works as expected now (zpool remove itself, at least).

[ghost]

scenario:

• user adds vdev with ashift=x
• user adds vdev with ashift=y
• user is trying to remove one vdev
• system denies, because ashifts are mixed in pool (x,y)

Current approach blocks removing of vdev if ashift will differ, that means user will have to rebuild whole pool I think, "zpool add" should at least warn before adding vdev that would cause mixed ashift scenario

[yshui]

I am having this problem right now. Seems illogical to allow mixed ashift when adding a disk, then complaining about it when trying to remove.

Since mixed ashift pool apparently works, is there just a check that needs to be removed from zpool-remove? Or is it actually not possible to remove vdev from a mixed pool?

EDIT: I see... removal operation needs to keep a block level mapping table for the removed vdev. I assume that's why it requires all vdev to have the same ashift. And it doesn't support removal of raidz vdevs either.

Seems like a big trap for inexperienced user (like me) to fall into.

[samvv]

Well this was awkward ... I was adding the wrong device to my zpool and suddenly I have to rebuild my entire pool from scratch because of this bug.

I really appreciate the time and effort that is put into this project but I really hope this bug can be fixed.

[AllKind]

It's in the works: https://github.com/openzfs/zfs/pull/15509

[XANi]

We've had that happen without user manually setting ashift, just by adding a hard disk

Sooo it is unfixable and requires re-creating everything ?

zpool add will use best ashift per disk now, not per pool.

why it works like that, it's a massive footgun. We just had junior admin following the usual instructions that happened to use the replacement drive that got ashift 12 instead of rest of the pool's 9 and now it is unfixable.