carina icon indicating copy to clipboard operation
carina copied to clipboard

Published 20 hours ago • verified publisher icon carina-io

容器内lsblk结果和宿主机上结果不一致

Open zwForrest opened this issue 3 years ago • 2 comments

What happened: 查看csi-carina-node的日志，发现pod内的lsblk结果和宿主机不一致，从结果上来看没问题，但是日志会出现不是空盘的磁盘会有pv创建等错误日志 出现，因为磁盘已经被初始化了而出现错误。

pod内视角

[root@csi-carina-node-qhgnv /]# lsblk --pairs --paths --bytes --all --output NAME,FSTYPE,MOUNTPOINT,SIZE,STATE,TYPE,ROTA,RO,PKNAME
NAME="/dev/nbd3" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd15" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/rbd0" FSTYPE="" MOUNTPOINT="/var/lib/kubelet/pods/aab3b273-4521-43bc-805d-4f075622b1d6/volumes/kubernetes.io~csi/pvc-5813f718-2bb3-4a33-839b-68aa665f615f/mount" SIZE="10737418240" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd1" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd13" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/vdd" FSTYPE="" MOUNTPOINT="" SIZE="137438953472" STATE="" TYPE="disk" ROTA="1" RO="0" PKNAME=""
NAME="/dev/vdd1" FSTYPE="" MOUNTPOINT="" SIZE="137437904896" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vdd"
NAME="/dev/mapper/vg_hdd-thin--pvc--f9a35054--5652--4e4e--8a0c--5e7252eebe0a_tmeta" FSTYPE="" MOUNTPOINT="" SIZE="4194304" STATE="running" TYPE="lvm" ROTA="1" RO="0" PKNAME="/dev/vdd1"
NAME="/dev/mapper/vg_hdd-thin--pvc--f9a35054--5652--4e4e--8a0c--5e7252eebe0a-tpool" FSTYPE="" MOUNTPOINT="" SIZE="1073741824" STATE="running" TYPE="lvm" ROTA="1" RO="0" PKNAME="/dev/dm-1"
NAME="/dev/mapper/vg_hdd-thin--pvc--f9a35054--5652--4e4e--8a0c--5e7252eebe0a" FSTYPE="" MOUNTPOINT="" SIZE="1073741824" STATE="running" TYPE="lvm" ROTA="1" RO="1" PKNAME="/dev/dm-3"
NAME="/dev/mapper/vg_hdd-volume--pvc--f9a35054--5652--4e4e--8a0c--5e7252eebe0a" FSTYPE="" MOUNTPOINT="/var/lib/kubelet/pods/d5cd3f31-d6db-4d01-82b3-3ee42b647530/volumes/kubernetes.io~csi/pvc-f9a35054-5652-4e4e-8a0c-5e7252eebe0a/mount" SIZE="1073741824" STATE="running" TYPE="lvm" ROTA="1" RO="0" PKNAME="/dev/dm-3"
NAME="/dev/mapper/vg_hdd-thin--pvc--f9a35054--5652--4e4e--8a0c--5e7252eebe0a_tdata" FSTYPE="" MOUNTPOINT="" SIZE="1073741824" STATE="running" TYPE="lvm" ROTA="1" RO="0" PKNAME="/dev/vdd1"
NAME="/dev/mapper/vg_hdd-thin--pvc--f9a35054--5652--4e4e--8a0c--5e7252eebe0a-tpool" FSTYPE="" MOUNTPOINT="" SIZE="1073741824" STATE="running" TYPE="lvm" ROTA="1" RO="0" PKNAME="/dev/dm-2"
NAME="/dev/mapper/vg_hdd-thin--pvc--f9a35054--5652--4e4e--8a0c--5e7252eebe0a" FSTYPE="" MOUNTPOINT="" SIZE="1073741824" STATE="running" TYPE="lvm" ROTA="1" RO="1" PKNAME="/dev/dm-3"
NAME="/dev/mapper/vg_hdd-volume--pvc--f9a35054--5652--4e4e--8a0c--5e7252eebe0a" FSTYPE="" MOUNTPOINT="/var/lib/kubelet/pods/d5cd3f31-d6db-4d01-82b3-3ee42b647530/volumes/kubernetes.io~csi/pvc-f9a35054-5652-4e4e-8a0c-5e7252eebe0a/mount" SIZE="1073741824" STATE="running" TYPE="lvm" ROTA="1" RO="0" PKNAME="/dev/dm-3"
NAME="/dev/mapper/vg_hdd-pvc--8450d57d--8079--460d--a8e9--96c0a290a776" FSTYPE="" MOUNTPOINT="" SIZE="128849018880" STATE="running" TYPE="lvm" ROTA="1" RO="0" PKNAME="/dev/vdd1"
NAME="/dev/nbd11" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/vdb" FSTYPE="" MOUNTPOINT="" SIZE="214748364800" STATE="" TYPE="disk" ROTA="1" RO="0" PKNAME=""
NAME="/dev/vdb1" FSTYPE="" MOUNTPOINT="/var/lib/kubelet/pods/73fa2701-5844-4109-a362-531081a5d332/volume-subpaths/config/dashboard/1" SIZE="214746267648" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vdb"
NAME="/dev/nbd8" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd6" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd4" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/rbd1" FSTYPE="" MOUNTPOINT="/var/lib/kubelet/pods/54814ea4-44a1-41a7-a880-4058e841144f/volumes/kubernetes.io~csi/pvc-9df1c024-60d6-4537-87c4-13d76f5f1615/mount" SIZE="10737418240" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/loop0" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="loop" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd2" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd14" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/vde" FSTYPE="" MOUNTPOINT="" SIZE="137438953472" STATE="" TYPE="disk" ROTA="1" RO="0" PKNAME=""
NAME="/dev/vde1" FSTYPE="" MOUNTPOINT="" SIZE="137437904896" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vde"
NAME="/dev/nbd0" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd12" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/vdc" FSTYPE="" MOUNTPOINT="" SIZE="1073741824000" STATE="" TYPE="disk" ROTA="1" RO="0" PKNAME=""
NAME="/dev/vdc1" FSTYPE="" MOUNTPOINT="" SIZE="1073739726848" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vdc"
NAME="/dev/nbd9" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd10" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/vda" FSTYPE="" MOUNTPOINT="" SIZE="21474836480" STATE="" TYPE="disk" ROTA="1" RO="0" PKNAME=""
NAME="/dev/vda2" FSTYPE="" MOUNTPOINT="/var/log/carina" SIZE="20946354176" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vda"
NAME="/dev/vda1" FSTYPE="" MOUNTPOINT="" SIZE="524288000" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vda"
NAME="/dev/nbd7" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd5" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""

宿主机视角

[root@tj1-test04 ~]# lsblk --pairs --paths --bytes --all --output NAME,FSTYPE,MOUNTPOINT,SIZE,STATE,TYPE,ROTA,RO,PKNAME
NAME="/dev/nbd3" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd15" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd1" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd13" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd11" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/vdb" FSTYPE="" MOUNTPOINT="" SIZE="214748364800" STATE="" TYPE="disk" ROTA="1" RO="0" PKNAME=""
NAME="/dev/vdb1" FSTYPE="ext4" MOUNTPOINT="/home" SIZE="214746267648" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vdb"
NAME="/dev/nbd8" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd6" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd4" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/rbd1" FSTYPE="ext4" MOUNTPOINT="/home/kubelet/pods/f2544a37-8136-4c87-95bb-f4feb55c6668/volumes/kubernetes.io~csi/pvc-25997d72-ad5a-4d78-99a9-b6c09bd59d00/mount" SIZE="8589934592" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd2" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd14" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd0" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd12" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/vdc" FSTYPE="" MOUNTPOINT="" SIZE="1073741824000" STATE="" TYPE="disk" ROTA="1" RO="0" PKNAME=""
NAME="/dev/vdc1" FSTYPE="ext4" MOUNTPOINT="/home/work/ssd1" SIZE="1073739726848" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vdc"
NAME="/dev/nbd9" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd10" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/vda" FSTYPE="" MOUNTPOINT="" SIZE="21474836480" STATE="" TYPE="disk" ROTA="1" RO="0" PKNAME=""
NAME="/dev/vda2" FSTYPE="ext4" MOUNTPOINT="/" SIZE="20946354176" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vda"
NAME="/dev/vda1" FSTYPE="ext4" MOUNTPOINT="/boot" SIZE="524288000" STATE="" TYPE="part" ROTA="1" RO="0" PKNAME="/dev/vda"
NAME="/dev/nbd7" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""
NAME="/dev/nbd5" FSTYPE="" MOUNTPOINT="" SIZE="" STATE="" TYPE="disk" ROTA="0" RO="0" PKNAME=""

比如其中的/dev/vdc1磁盘，在容器内是显示空磁盘，但是实际上是已经有挂载点，正常不应该被自动发现而出现需要初始化的情况。

初始化日志

2022-02-23T14:23:20.860+0800    info    devicemanager/manager.go:309    eligible vg_ssd device /dev/vdc1

有这个日志说明就发现这个磁盘为可添加磁盘， 还有初始化错误日志

2022-02-23T14:23:20.905+0800    info    devicemanager/manager.go:162    vg:vg_ssd ,pvs:[/dev/vdc1]
2022-02-23T14:23:20.905+0800    info    exec/exec.go:303        Running command: pvs --noheadings --separator=, --units=b --nosuffix --unbuffered --nameprefixes
2022-02-23T14:23:20.948+0800    info    exec/exec.go:303        Running command: pvcreate /dev/vdc1
2022-02-23T14:23:20.991+0800    error   volume/volume.go:363    create pv failed /dev/vdc1
2022-02-23T14:23:20.991+0800    error   devicemanager/manager.go:169    add new disk failed vg: vg_ssd, disk: /dev/vdc1, error: exit status 5

What you expected to happen: 正常不应该由不同的磁盘lsblk结果，这可能会有意向不到的结果

How to reproduce it:

Anything else we need to know?:

Environment:

  • CSI Driver version: 0.9.1
  • Kubernetes version (use kubectl version):
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

zwForrest avatar Feb 23 '22 07:02 zwForrest

这是个问题，忒在获取磁盘信息前，先同步一下磁盘信息

antmoveh avatar Feb 24 '22 01:02 antmoveh

I had the same problem. It can't get FSTYPE info in container. It need mount udev data. I will take a PR to fix this bug.

fanhaouu avatar Sep 13 '22 10:09 fanhaouu