nativestor icon indicating copy to clipboard operation
nativestor copied to clipboard

Published 20 hours ago • verified publisher icon alauda

Unable to detect loop devices in AWS

Open leelavg opened this issue 3 years ago • 1 comments

Hello again,

  • I created an OCP cluster in AWS and performed all pre-req for carving a PVC from topolvm-operator
  • Below are the manifests that I deployed
# bat ../sample/tl-all-disks.yaml --plain
apiVersion: topolvm.cybozu.com/v2
kind: TopolvmCluster
metadata:
  name: topolvmcluster-sample
  namespace: topolvm-system
spec:
  topolvmVersion: "alaudapublic/topolvm:2.0.0"
  storage:
    useAllNodes: true
    useAllDevices: true
    useLoop: true
    volumeGroupName: test
    className: hdd

# bat --plain ../sample/tl-sc.yaml 
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: topolvm-provisioner-hdd
provisioner: topolvm.cybozu.com
# Intentionally set to Immediate just to test the binding
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  # "csi.storage.k8s.io/fstype": "xfs"
  "topolvm.cybozu.com/device-class": "hdd"

# bat --plain ../sample/tl-pvc.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tl-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: topolvm-provisioner-hdd
  • Verified #35 as well and found some new issues, refer below log
# Right after applying CR
2021-10-29 03:33:32.566717 I | discover: updating device configmap
2021-10-29 03:33:32.570730 W | sys: skipping device "loop0". diskType is empty
2021-10-29 03:33:33.057123 I | sys: skipping device "nvme0n1" because it has child, considering the child instead.
2021-10-29 03:33:34.058689 I | sys: skipping device "nvme1n1" because it contains a filesystem "LVM2_member"
2021-10-29 03:33:34.058709 I | sys: skipping device "nvme0n1p1" because it size less than 1G
2021-10-29 03:33:34.058713 I | sys: skipping device "nvme0n1p2" because it size less than 1G
2021-10-29 03:33:34.058717 I | sys: skipping device "nvme0n1p3" because it size less than 1G
2021-10-29 03:33:34.058722 I | sys: skipping device "nvme0n1p4" because it contains a filesystem "xfs"
2021-10-29 03:33:34.058728 I | sys: skipping device "nvme3n1" because it contains a filesystem "LVM2_member"
2021-10-29 03:33:34.058732 I | sys: skipping device "nvme2n1" because it contains a filesystem "LVM2_member"
2021-10-29 03:33:34.058736 I | sys: skipping device "nvme4n1" because it contains a filesystem "LVM2_member"


[...]


# After carving a block PVC
2021-10-29 04:09:23.261666 I | discover: updating device configmap
2021-10-29 04:09:23.560866 W | sys: skipping device "dm-0". unsupported diskType lvm
2021-10-29 04:09:23.760660 I | sys: skipping device "nvme1n1" because it has child, considering the child instead.
2021-10-29 04:09:24.059618 I | sys: skipping device "nvme0n1" because it has child, considering the child instead.
2021-10-29 04:09:24.958649 I | sys: skipping device "loop0" because it size less than 1G
2021-10-29 04:09:24.958669 I | sys: skipping device "nvme0n1p1" because it size less than 1G
2021-10-29 04:09:24.958673 I | sys: skipping device "nvme0n1p2" because it size less than 1G
2021-10-29 04:09:24.958676 I | sys: skipping device "nvme0n1p3" because it size less than 1G
2021-10-29 04:09:24.958685 I | sys: skipping device "nvme0n1p4" because it contains a filesystem "xfs"
2021-10-29 04:09:24.958688 I | sys: skipping device "nvme3n1" because it contains a filesystem "LVM2_member"
2021-10-29 04:09:24.958693 I | sys: skipping device "nvme2n1" because it contains a filesystem "LVM2_member"
2021-10-29 04:09:24.958697 I | sys: skipping device "nvme4n1" because it contains a filesystem "LVM2_member"
  • Refer below o/p as well
# After applying CR
sh-4.4# lsblk /dev/loop0 -O
sh-4.4# file /dev/loop0
/dev/loop0: block special (7/0)

# After creating block PVC
sh-4.4# lsblk
NAME                                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0                                             7:0    0     1G  0 loop 
nvme1n1                                         259:0    0    10G  0 disk 
`-test-98e009ff--6448--4e7e--a6d1--a738797d8862 253:0    0     1G  0 lvm  
nvme0n1                                         259:1    0   120G  0 disk 
|-nvme0n1p1                                     259:2    0     1M  0 part 
|-nvme0n1p2                                     259:3    0   127M  0 part 
|-nvme0n1p3                                     259:4    0   384M  0 part /boot
`-nvme0n1p4                                     259:5    0 119.5G  0 part /sysroot
nvme3n1                                         259:6    0    10G  0 disk 
nvme2n1                                         259:7    0    10G  0 disk 
nvme4n1                                         259:8    0    10G  0 disk 

sh-4.4# lsblk /dev/loop0 --bytes --nodeps --pairs --paths --output "SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME"
SIZE="1073741824" ROTA="0" RO="0" TYPE="loop" PKNAME="" NAME="/dev/loop0" KNAME="/dev/loop0"

sh-4.4# lsblk /dev/dm-0 --bytes --nodeps --pairs --paths --output "SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME"
SIZE="1073741824" ROTA="0" RO="0" TYPE="lvm" PKNAME="" NAME="/dev/mapper/test-98e009ff--6448--4e7e--a6d1--a738797d8862" KNAME="/dev/dm-0"

# Same for `loop0` as well
sh-4.4# sfdisk -l /dev/dm-0
Disk /dev/dm-0: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
sh-4.4#

Issues:

  1. [Minor] Log should state less than 2G not less than 1G
  2. Just after applying CR, discovery failed to consider loop0 as a loop device and didn't store it in lvmd configmap due to no o/p in lsblk, any other way to consider it as a loop device
  3. After creating a block volume, discovery is trying to see dm-0 is usable or not, however is it possible not to step over it at all (modify lsblk maybe?), cause overtime there'll be many VGs created (file/block) and it might not be ideal to step over all VGs every-time the discovery logic runs?

Let me know if more info is required.

leelavg avatar Oct 29 '21 04:10 leelavg

Hello again,

  • I created an OCP cluster in AWS and performed all pre-req for carving a PVC from topolvm-operator
  • Below are the manifests that I deployed
# bat ../sample/tl-all-disks.yaml --plain
apiVersion: topolvm.cybozu.com/v2
kind: TopolvmCluster
metadata:
  name: topolvmcluster-sample
  namespace: topolvm-system
spec:
  topolvmVersion: "alaudapublic/topolvm:2.0.0"
  storage:
    useAllNodes: true
    useAllDevices: true
    useLoop: true
    volumeGroupName: test
    className: hdd

# bat --plain ../sample/tl-sc.yaml 
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: topolvm-provisioner-hdd
provisioner: topolvm.cybozu.com
# Intentionally set to Immediate just to test the binding
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
  # "csi.storage.k8s.io/fstype": "xfs"
  "topolvm.cybozu.com/device-class": "hdd"

# bat --plain ../sample/tl-pvc.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tl-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: topolvm-provisioner-hdd

# Right after applying CR
2021-10-29 03:33:32.566717 I | discover: updating device configmap
2021-10-29 03:33:32.570730 W | sys: skipping device "loop0". diskType is empty
2021-10-29 03:33:33.057123 I | sys: skipping device "nvme0n1" because it has child, considering the child instead.
2021-10-29 03:33:34.058689 I | sys: skipping device "nvme1n1" because it contains a filesystem "LVM2_member"
2021-10-29 03:33:34.058709 I | sys: skipping device "nvme0n1p1" because it size less than 1G
2021-10-29 03:33:34.058713 I | sys: skipping device "nvme0n1p2" because it size less than 1G
2021-10-29 03:33:34.058717 I | sys: skipping device "nvme0n1p3" because it size less than 1G
2021-10-29 03:33:34.058722 I | sys: skipping device "nvme0n1p4" because it contains a filesystem "xfs"
2021-10-29 03:33:34.058728 I | sys: skipping device "nvme3n1" because it contains a filesystem "LVM2_member"
2021-10-29 03:33:34.058732 I | sys: skipping device "nvme2n1" because it contains a filesystem "LVM2_member"
2021-10-29 03:33:34.058736 I | sys: skipping device "nvme4n1" because it contains a filesystem "LVM2_member"


[...]


# After carving a block PVC
2021-10-29 04:09:23.261666 I | discover: updating device configmap
2021-10-29 04:09:23.560866 W | sys: skipping device "dm-0". unsupported diskType lvm
2021-10-29 04:09:23.760660 I | sys: skipping device "nvme1n1" because it has child, considering the child instead.
2021-10-29 04:09:24.059618 I | sys: skipping device "nvme0n1" because it has child, considering the child instead.
2021-10-29 04:09:24.958649 I | sys: skipping device "loop0" because it size less than 1G
2021-10-29 04:09:24.958669 I | sys: skipping device "nvme0n1p1" because it size less than 1G
2021-10-29 04:09:24.958673 I | sys: skipping device "nvme0n1p2" because it size less than 1G
2021-10-29 04:09:24.958676 I | sys: skipping device "nvme0n1p3" because it size less than 1G
2021-10-29 04:09:24.958685 I | sys: skipping device "nvme0n1p4" because it contains a filesystem "xfs"
2021-10-29 04:09:24.958688 I | sys: skipping device "nvme3n1" because it contains a filesystem "LVM2_member"
2021-10-29 04:09:24.958693 I | sys: skipping device "nvme2n1" because it contains a filesystem "LVM2_member"
2021-10-29 04:09:24.958697 I | sys: skipping device "nvme4n1" because it contains a filesystem "LVM2_member"
  • Refer below o/p as well
# After applying CR
sh-4.4# lsblk /dev/loop0 -O
sh-4.4# file /dev/loop0
/dev/loop0: block special (7/0)

# After creating block PVC
sh-4.4# lsblk
NAME                                            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0                                             7:0    0     1G  0 loop 
nvme1n1                                         259:0    0    10G  0 disk 
`-test-98e009ff--6448--4e7e--a6d1--a738797d8862 253:0    0     1G  0 lvm  
nvme0n1                                         259:1    0   120G  0 disk 
|-nvme0n1p1                                     259:2    0     1M  0 part 
|-nvme0n1p2                                     259:3    0   127M  0 part 
|-nvme0n1p3                                     259:4    0   384M  0 part /boot
`-nvme0n1p4                                     259:5    0 119.5G  0 part /sysroot
nvme3n1                                         259:6    0    10G  0 disk 
nvme2n1                                         259:7    0    10G  0 disk 
nvme4n1                                         259:8    0    10G  0 disk 

sh-4.4# lsblk /dev/loop0 --bytes --nodeps --pairs --paths --output "SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME"
SIZE="1073741824" ROTA="0" RO="0" TYPE="loop" PKNAME="" NAME="/dev/loop0" KNAME="/dev/loop0"

sh-4.4# lsblk /dev/dm-0 --bytes --nodeps --pairs --paths --output "SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME"
SIZE="1073741824" ROTA="0" RO="0" TYPE="lvm" PKNAME="" NAME="/dev/mapper/test-98e009ff--6448--4e7e--a6d1--a738797d8862" KNAME="/dev/dm-0"

# Same for `loop0` as well
sh-4.4# sfdisk -l /dev/dm-0
Disk /dev/dm-0: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
sh-4.4#

Issues:

  1. [Minor] Log should state less than 2G not less than 1G
  2. Just after applying CR, discovery failed to consider loop0 as a loop device and didn't store it in lvmd configmap due to no o/p in lsblk, any other way to consider it as a loop device
  3. After creating a block volume, discovery is trying to see dm-0 is usable or not, however is it possible not to step over it at all (modify lsblk maybe?), cause overtime there'll be many VGs created (file/block) and it might not be ideal to step over all VGs every-time the discovery logic runs?

Let me know if more info is required.

loop0 didn't add to lvmd configmap because the size (1G) is less than 2G , the log is not correct, i will make a fix.

for issue 3. we will think about it.

little-guy-lxr avatar Nov 01 '21 05:11 little-guy-lxr