NDM looping constantly causing high cpu usage with `Error: unreachable state`

[magnetised]

What steps did you take and what happened: I've just installed openebs as part of k0s on an aws ec2 instance with 2 disks, the host disk and a separate ebs data partition. Everything seems to be working fine but one of the ndm pods is at a constant 20% cpu usage. looking at the logs it seems to be in some loop querying the host/node disks

Looking at another server with the same ndm version but a simpler, single-disk setup, the exact same thing is happening.

What did you expect to happen: I expected the ndm process to not be constantly using cpu in a constant loop.

The output of the following commands will help us better understand what's going on: [Pasting long output into a GitHub gist or other pastebin is fine.]

• kubectl get pods -n openebs

NAME                                           READY   STATUS    RESTARTS      AGE
openebs-localpv-provisioner-6ccc9d6fc9-kcnhs   1/1     Running   9 (19h ago)   20h
openebs-ndm-jpvpw                              1/1     Running   0             26m
openebs-ndm-operator-7bd6898d96-vz54r          1/1     Running   9 (19h ago)   20h

• kubectl get blockdevices -n openebs -o yaml

apiVersion: v1
items:
- apiVersion: openebs.io/v1alpha1
  kind: BlockDevice
  metadata:
    annotations:
      internal.openebs.io/uuid-scheme: gpt
    creationTimestamp: "2022-07-05T13:22:38Z"
    generation: 20
    labels:
      kubernetes.io/hostname: ip-172-31-18-163.eu-west-1.compute.internal
      ndm.io/blockdevice-type: blockdevice
      ndm.io/managed: "true"
    name: blockdevice-01fd0d0d966998648102985c5f12e22a
    namespace: openebs
    resourceVersion: "64236"
    uid: 9d3e2ec3-57b5-4303-829c-e0cfa51f2f07
  spec:
    capacity:
      logicalSectorSize: 512
      physicalSectorSize: 512
      storage: 137437888000
    details:
      compliance: ""
      deviceType: partition
      driveType: SSD
      firmwareRevision: ""
      hardwareSectorSize: 512
      logicalBlockSize: 512
      model: Amazon Elastic Block Store
      physicalBlockSize: 512
      serial: vol033aa51d4508ed1b0
      vendor: ""
    devlinks:
    - kind: by-id
      links:
      - /dev/disk/by-id/nvme-nvme.1d0f-766f6c3033336161353164343530386564316230-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001-part1
      - /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol033aa51d4508ed1b0-part1
      - /dev/disk/by-id/wwn-nvme.1d0f-766f6c3033336161353164343530386564316230-416d617a6f6e20456c617374696320426c6f636b2053746f7265-00000001-part1
    - kind: by-path
      links:
      - /dev/disk/by-path/pci-0000:00:1f.0-nvme-1-part1
    filesystem:
      fsType: xfs
      mountPoint: /var/openebs
    nodeAttributes:
      nodeName: ip-172-31-18-163.eu-west-1.compute.internal
    partitioned: "No"
    path: /dev/nvme1n1p1
  status:
    claimState: Unclaimed
    state: Inactive
kind: List
metadata:
  resourceVersion: ""

• kubectl get blockdeviceclaims -n openebs -o yaml

apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""

• kubectl logs <ndm daemon pod name> -n openebs

just including two loops, it goes on like this permanently.

https://gist.github.com/magnetised/c1f2bef4242b663721d87898f8416d65

• lsblk from nodes where ndm daemonset is running

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme1n1     259:0    0  128G  0 disk
└─nvme1n1p1 259:4    0  128G  0 part /var/openebs
nvme0n1     259:1    0  128G  0 disk
├─nvme0n1p1 259:2    0    1M  0 part
└─nvme0n1p2 259:3    0  128G  0 part /

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• OpenEBS version

openebs.io/version=3.0.0 node-disk-manager:1.7.0

• Kubernetes version (use kubectl version):

Client Version: v1.24.2
Kustomize Version: v4.5.4
Server Version: v1.23.6+k0s

• Kubernetes installer & version:

K0s version v1.23.6+k0s.0

• Cloud provider or hardware configuration:

AWS EC2 instance

• Type of disks connected to the nodes (eg: Virtual Disks, GCE/EBS Volumes, Physical drives etc)

host root partition nvme0n1 open ebs volume nvme1n1 with a single partition nvme1n1p1 mounted at /var/openebs

• OS (e.g. from /etc/os-release):

NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

[artem-zinnatullin]

Exact same issue with vanilla k0s v1.27.2+k0s.0 installation with open ebs extension enabled (openebs/node-disk-manager:1.9.0), consumes over 60% of CPU on IDLE with no PVs no nothing, this is really bad.

[gervaso]

Hi, we had the same issue on-premise and it was caused by the presence of "/dev/sr1" on the vm, so I think you should update the filter to remove unusable devices.