LSILogicParallel disk is still listed by lsblk after hot adding\removing

[Winnie81]

Description

LSILogicParallel disk is still listed by lsblk after hot adding\removing

Environment and steps to reproduce

1. Deploy a Flatcar 3033.2.0 or 3033.2.1 vm.
2. Hot add a LsiLogicParallel controller and a disk with this controller.
3. Run command lsblk, it could see sda and sdb disks.
4. Hot remove the LsiLogicParallel disk.
5. Recan all scsi devices: for i in find /sys/ -iname rescan;do echo 1 >$i; done;for i in find /sys/ -iname scan;do echo "- - -" >$i; done;
6. Run command lsblk. ----> sdb is still there.

Expected behavior

Only sda is shown after step 6.

For 2983.2.0, after step 6, sdb is not shown.

[t-lo]

Thank you for reporting, @Winnie81 ! Could you please attach the journal and dmesg output? It would help us a lot to understand what's happening.

[Winnie81]

Thank you for reporting, @Winnie81 ! Could you please attach the journal and dmesg output? It would help us a lot to understand what's happening.

Hi, journal and dmesg are [attached journal.zip dmesg.txt .

[t-lo]

Thank you for providing the logs Winnie, this is very helpful.

[Winnie81]

@t-lo Is there any progress for this issue? Thanks!

[Winnie81]

Thank you for providing the logs Winnie, this is very helpful.

Hi, Could you please update the newest progress for this issue? Thanks a lot.

[jepio]

Hi, @Winnie81 We currently have no idea what this is caused by and have not made any progress on this issue.

[Winnie81]

This issue exists in 3033.3.0 and 3139.2.0, hope it could be fixed. Thanks a lot.

[Winnie81]

Hi, @Winnie81 We currently have no idea what this is caused by and have not made any progress on this issue.

I'd like to know do you have any plan for this issue? Thanks a lot.

[Winnie81]

Since this issue blocks some of our testing, hope it could be fixed. Thanks a lot.

[jepio]

Hi @Winnie81,

I took the time to investigate this again. The change comes from this commit https://github.com/util-linux/util-linux/commit/c90a8f1a3cdf116e050708098f0188349b55abc3, part of util-linux 2.37. You will note that in both cases (flatcar 2983.2.0 and 3033.2.0) the /dev/sda node is present after hotunplug, but has a size == 0. Old lsblk filters this out from it's output, and new lsblk does not. This matches the behavior I am seeing with Ubuntu 22.04 VMs, but I am testing with the LSILogic SCSI controller in Qemu.

So since this is upstream behavior, you might want to adjust your test case to account for this. Or perhaps report this as an issue against the util-linux project.

[Winnie81]

Hi @Winnie81,

I took the time to investigate this again. The change comes from this commit util-linux/util-linux@c90a8f1, part of util-linux 2.37. You will note that in both cases (flatcar 2983.2.0 and 3033.2.0) the /dev/sda node is present after hotunplug, but has a size == 0. Old lsblk filters this out from it's output, and new lsblk does not. This matches the behavior I am seeing with Ubuntu 22.04 VMs, but I am testing with the LSILogic SCSI controller in Qemu.

So since this is upstream behavior, you might want to adjust your test case to account for this. Or perhaps report this as an issue against the util-linux project.

Thanks for your update. From my testing with Flatcar 2983.2.0, after hot remove, its size doesn't change: core@localhost ~ $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 8.5G 0 disk
|-sda1 8:1 0 128M 0 part
|-sda2 8:2 0 2M 0 part
|-sda3 8:3 0 1G 0 part
| -usr 254:0 0 1016M 1 crypt /usr |-sda4 8:4 0 1G 0 part |-sda6 8:6 0 128M 0 part /usr/share/oem |-sda7 8:7 0 64M 0 part -sda9 8:9 0 6.2G 0 part / sdb 8:16 0 5G 0 disk
sr0 11:0 1 1024M 0 rom

If rescan it, sdb will gone.

[jepio]

I understand that on 2983.2.0 sdb does not show up in lsblk output after rescan. Please check if ls -l /dev/sdb still shows that the underlying device node exists after the rescan. In my testing this is the case regardless of Flatcar version (and also on other distros), and so the sdb entry being shown in lsblk output is purely dependent on the util-linux version. If you could confirm that we are seeing the same, that would be great.

In that case it is a matter of the linked commit in util-linux, and needs to be reported to util-linux.

[Winnie81]

I understand that on 2983.2.0 sdb does not show up in lsblk output after rescan. Please check if ls -l /dev/sdb still shows that the underlying device node exists after the rescan. In my testing this is the case regardless of Flatcar version (and also on other distros), and so the sdb entry being shown in lsblk output is purely dependent on the util-linux version. If you could confirm that we are seeing the same, that would be great.

In that case it is a matter of the linked commit in util-linux, and needs to be reported to util-linux.

Yes, as described in https://github.com/flatcar-linux/Flatcar/issues/608#issuecomment-1143078433, for 2983.2.0, after rescan, sdb doesn't exist.

[jepio]

@Winnie81

I've just created an ESXI environment with a Flatcar 2983.2.0 VM and using LSILogicParallel controller. The attached harddrive is called sdb.

After hot-removing the harddrive from the controller, and running the rescan command, sdb disappears from the lsblk output but ls -l /dev/sdb confirms that the device node still exists. I also see:

$ cat /sys/block/sdb/size
0

So based on that, I can't see any bug to be fixed here. The change in lsblk output is due to the util-linux update.

Are we seeing different things?

[Winnie81]

@Winnie81

I've just created an ESXI environment with a Flatcar 2983.2.0 VM and using LSILogicParallel controller. The attached harddrive is called sdb.

After hot-removing the harddrive from the controller, and running the rescan command, sdb disappears from the lsblk output but ls -l /dev/sdb confirms that the device node still exists. I also see:

$ cat /sys/block/sdb/size
0

So based on that, I can't see any bug to be fixed here. The change in lsblk output is due to the util-linux update.

Are we seeing different things?

Sorry for the late response. I missed your reply.

As https://github.com/flatcar-linux/Flatcar/issues/608#issue-1115887651 described, this issue started from 3033.2.0.

And also please refer to our above discussions about the different behaviors between 2983 and its later visions.

[jepio]

lsblk changed their output, you will need to adapt your test to filter out devices with size 0. In util-linux 2.38.1 there is a new cli flag (--noempty) that does this for you https://github.com/util-linux/util-linux/commit/554e866b94fc9965fef089cf480abea474dc9e34, but since this flag is not available in older versions you will need to handle size 0 devices anyway.

There's nothing further that I can do here.