lvm-localpv Volumes go missing.

Volumes go missing.

Open davidkarlsen opened this issue 2 years ago • 31 comments

What steps did you take and what happened: Created a pvc, which got formatted and mounted just fine. Then booted the machine, after bootup the volume is not to be found! This causes loss of data and is very severe.

What did you expect to happen: Not loosing volumes!

The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other Pastebin is fine.)

kubectl logs -f openebs-lvm-controller-0 -n kube-system -c openebs-lvm-plugin

I0919 21:07:44.486525       1 main.go:136] LVM Driver Version :- 0.8.0 - commit :- 929ae4439f2da71a2d6ee5bda6a33dd2f7d424fc
I0919 21:07:44.486615       1 main.go:137] DriverName: local.csi.openebs.io Plugin: controller EndPoint: unix:///var/lib/csi/sockets/pluginproxy/csi.sock NodeID:  SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I0919 21:07:44.486646       1 driver.go:48] enabling volume access mode: SINGLE_NODE_WRITER
I0919 21:07:44.496097       1 controller.go:212] waiting for k8s & lvm node informer caches to be synced
I0919 21:07:44.596266       1 controller.go:216] synced k8s & lvm node informer caches
I0919 21:07:44.596300       1 controller.go:218] initializing csi provisioning leak protection controller
I0919 21:07:44.596895       1 leak_protection.go:129] "starting up csi pvc controller"
I0919 21:07:44.596916       1 shared_informer.go:240] Waiting for caches to sync for CSI Provisioner
I0919 21:07:44.597275       1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
I0919 21:07:44.697073       1 shared_informer.go:247] Caches are synced for CSI Provisioner 
I0919 21:07:44.991733       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:44.999709       1 grpc.go:81] GRPC response: {}
I0919 21:07:45.001288       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.001331       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.002064       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0919 21:07:45.002142       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0919 21:07:45.002973       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.003009       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0919 21:07:45.325452       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:45.325666       1 grpc.go:81] GRPC response: {}
I0919 21:07:45.326637       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.326779       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.327324       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0919 21:07:45.327354       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0919 21:07:45.328054       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.328135       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0919 21:07:45.673459       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.673695       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.675001       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:45.675122       1 grpc.go:81] GRPC response: {}
I0919 21:07:45.675962       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.676000       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
....
I0919 22:20:15.437512       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco06","openebs.io/nodename":"alt-ksx-g-c01oco06"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.437667       1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.438064       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-eos-g-c01oco03","openebs.io/nodename":"alt-eos-g-c01oco03"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.438195       1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.438548       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ebs-g-c01oco01","openebs.io/nodename":"alt-ebs-g-c01oco01"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.438674       1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.439082       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco09","openebs.io/nodename":"alt-ksx-g-c01oco09"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.439263       1 grpc.go:81] GRPC response: {"available_capacity":2171097579520}
I0919 22:20:15.439687       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-eos-g-c01oco02","openebs.io/nodename":"alt-eos-g-c01oco02"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.439817       1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.440257       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco06","openebs.io/nodename":"alt-ksx-g-c01oco06"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.440385       1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.440940       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco04","openebs.io/nodename":"alt-ksx-g-c01oco04"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.441111       1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.441526       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco07","openebs.io/nodename":"alt-ksx-g-c01oco07"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.441646       1 grpc.go:81] GRPC response: {"available_capacity":2171097579520}
I0919 22:20:15.441994       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ebs-g-c01oco01","openebs.io/nodename":"alt-ebs-g-c01oco01"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.442129       1 grpc.go:81] GRPC response: {"available_capacity":322118352896}

kubectl logs -f openebs-lvm-node-[xxxx] -n kube-system -c openebs-lvm-plugin
kubectl get pods -n kube-system

k -n openebs get pods
NAME                                           READY   STATUS    RESTARTS   AGE
openebs-admission-server-894884484-gcc5f       1/1     Running   0          29h
openebs-apiserver-6494d57b96-xfltt             1/1     Running   0          77m
openebs-localpv-provisioner-5787758b59-tzs8l   1/1     Running   1          29h
openebs-lvm-localpv-controller-0               5/5     Running   0          76m
openebs-lvm-localpv-node-7x2kw                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-98kxr                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-9r2kd                 2/2     Running   20         5d4h
openebs-lvm-localpv-node-h949v                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-jff5w                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-kccx5                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-ln7r4                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-pzxw4                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-qxwn9                 2/2     Running   11         5d4h
openebs-lvm-localpv-node-qzxzb                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-trfvk                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-w25kv                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-wxv6w                 2/2     Running   6          5d4h
openebs-lvm-localpv-node-xkmzj                 2/2     Running   6          5d4h
openebs-ndm-5rq88                              1/1     Running   5          10d
openebs-ndm-ftnwd                              1/1     Running   3          10d
openebs-ndm-k9z5w                              1/1     Running   3          10d
openebs-ndm-mxzss                              1/1     Running   3          10d
openebs-ndm-operator-7bd7455b5b-khrmt          1/1     Running   0          29h
openebs-ndm-p79zz                              1/1     Running   3          10d
openebs-ndm-p7f77                              1/1     Running   3          10d
openebs-ndm-phfjw                              1/1     Running   3          10d
openebs-ndm-zsqmp                              1/1     Running   3          10d
openebs-provisioner-9f948db4-ljndj             1/1     Running   1          29h
openebs-snapshot-operator-68cf9946d6-vc5ls     2/2     Running   0          77m

I0919 21:47:33.032487       1 main.go:136] LVM Driver Version :- 0.8.0 - commit :- 929ae4439f2da71a2d6ee5bda6a33dd2f7d424fc
I0919 21:47:33.032679       1 main.go:137] DriverName: local.csi.openebs.io Plugin: agent EndPoint: unix:///plugin/csi.sock NodeID: alt-ksx-g-c01oco03 SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I0919 21:47:33.032725       1 driver.go:48] enabling volume access mode: SINGLE_NODE_WRITER
I0919 21:47:33.033388       1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//plugin/csi.sock", Net:"unix"}
I0919 21:47:33.036307       1 builder.go:107] Creating event broadcaster
I0919 21:47:33.036946       1 builder.go:107] Creating event broadcaster
I0919 21:47:33.037042       1 snapshot.go:165] Starting Snap controller
I0919 21:47:33.037049       1 snapshot.go:168] Waiting for informer caches to sync
I0919 21:47:33.036950       1 volume.go:265] Starting Vol controller
I0919 21:47:33.037584       1 volume.go:268] Waiting for informer caches to sync
I0919 21:47:33.055970       1 volume.go:209] Got add event for Vol pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
I0919 21:47:33.056056       1 volume.go:209] Got add event for Vol pvc-63ebce02-7077-48ea-88b3-301b1cf7909d
I0919 21:47:33.067860       1 builder.go:116] Creating event broadcaster
I0919 21:47:33.068028       1 lvmnode.go:191] Starting Node controller
I0919 21:47:33.068466       1 lvmnode.go:194] Waiting for informer caches to sync
I0919 21:47:33.071142       1 lvmnode.go:127] Got add event for lvm node openebs/alt-ksx-g-c01oco03
I0919 21:47:33.137317       1 snapshot.go:172] Starting Snap workers
I0919 21:47:33.137350       1 snapshot.go:179] Started Snap workers
I0919 21:47:33.137728       1 volume.go:272] Starting Vol workers
I0919 21:47:33.137738       1 volume.go:279] Started Vol workers
I0919 21:47:33.137830       1 volume.go:98] lvm volume already provisioned
I0919 21:47:33.137840       1 volume.go:337] Successfully synced 'openebs/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca'
I0919 21:47:33.137832       1 volume.go:98] lvm volume already provisioned
I0919 21:47:33.137904       1 volume.go:337] Successfully synced 'openebs/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d'
I0919 21:47:33.168649       1 lvmnode.go:199] Starting Node workers
I0919 21:47:33.168667       1 lvmnode.go:206] Started Node workers
I0919 21:47:33.489540       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:47:33.489636       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a060 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:47:33.497190       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:47:33.497209       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:47:33.754135       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:47:33.755194       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:47:35.327404       1 grpc.go:72] GRPC call: /csi.v1.Node/NodeGetInfo requests {}
I0919 21:47:35.334632       1 grpc.go:81] GRPC response: {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco03","openebs.io/nodename":"alt-ksx-g-c01oco03"}},"node_id":"alt-ksx-g-c01oco03"}
I0919 21:47:36.692782       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:36.762805       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:36.927823       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 21:47:36.927901       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 21:47:36.927997       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 21:47:37.493233       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:37.498794       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:37.499905       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 21:47:37.499975       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 21:47:37.500016       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 21:47:37.818527       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:47:37.823301       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:47:37.824462       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 21:47:37.824478       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 21:47:37.824495       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 21:47:38.411354       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:47:38.417422       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:47:38.418586       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 21:47:38.418630       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 21:47:38.418687       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 21:47:38.520481       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:38.525817       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:38.526895       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) ...
I0919 21:48:33.198169       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:48:33.198235       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006224c9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:48:33.205156       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:48:33.205174       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:48:40.776987       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:48:40.793493       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:48:40.794577       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 21:48:40.794632       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 21:48:40.794757       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 21:48:41.690835       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:48:41.697249       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:48:41.698425       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 21:48:41.698442       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 21:48:41.698463       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 

I0919 21:58:33.193143       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:58:33.193387       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026bce8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:58:33.200591       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:58:33.200709       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:59:33.197599       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:59:33.197854       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006320f9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:59:33.203510       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:59:33.203600       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:59:55.071045       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:59:55.077098       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:59:55.078420       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 21:59:55.078595       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 21:59:55.078709       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 21:59:56.077218       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:59:56.081863       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:59:56.083050       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 21:59:56.083154       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 21:59:56.083251       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:00:33.194333       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:00:33.194546       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000131069 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:00:33.200632       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:00:33.200795       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:01:33.192396       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:01:33.192458       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006a808a BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:01:33.199002       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:01:33.199019       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:01:57.130951       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:01:57.136043       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:01:57.137124       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:01:57.137250       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:01:57.137358       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:01:58.139894       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:01:58.145250       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:01:58.146369       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:01:58.146436       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:01:58.146499       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:02:33.198541       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:02:33.198810       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000632248 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:02:33.219623       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:02:33.219775       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:03:33.197462       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:03:33.197666       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006221bb BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:03:33.204957       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:03:33.205147       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:03:59.214157       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:03:59.221739       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:03:59.223076       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:03:59.223095       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:03:59.223124       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:04:00.225863       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:04:00.230920       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:04:00.232305       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:04:00.232440       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:04:00.232540       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:04:33.193373       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:04:33.193540       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000043278 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:04:33.199806       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:04:33.199958       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:05:33.201598       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:05:33.201968       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0003e00aa BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:05:33.209690       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:05:33.209786       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:06:01.309472       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:06:01.315687       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:06:01.316691       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:06:01.316707       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:06:01.316734       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:06:02.317247       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:06:02.321985       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:06:02.323009       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:06:02.323099       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:06:02.323181       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:06:33.192108       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:06:33.192174       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212090 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:06:33.203863       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:06:33.203885       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:07:33.197599       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:07:33.197883       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:07:33.204772       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:07:33.204789       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:08:03.336415       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:08:03.343673       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:08:03.344630       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:08:03.344645       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:08:03.344675       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:08:04.353996       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:08:04.364752       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:08:04.365823       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:08:04.365942       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:08:04.366035       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:08:33.195376       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:08:33.195672       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006ac7e8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:08:33.205158       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:08:33.205327       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:09:33.195347       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:09:33.195593       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0005901da BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:09:33.204502       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:09:33.204669       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:10:05.398785       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:10:05.405224       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:10:05.406422       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:10:05.406541       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:10:05.406634       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:10:06.407071       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:10:06.412408       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:10:06.413575       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:10:06.413705       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:10:06.413795       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:10:33.195388       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:10:33.195460       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000622978 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:10:33.209769       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:10:33.209910       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:11:33.195346       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:11:33.195445       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0000421f0 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:11:33.206672       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:11:33.206754       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:12:07.452479       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:12:07.457948       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:12:07.459041       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:12:07.459053       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:12:07.459073       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:12:08.468111       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:12:08.473579       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:12:08.474930       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:12:08.475057       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:12:08.475153       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:12:33.199283       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:12:33.199554       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001ab668 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:12:33.208417       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:12:33.208537       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:13:33.194398       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:13:33.194604       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001aa000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:13:33.202748       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:13:33.202825       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:14:09.561523       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:14:09.566213       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:14:09.567230       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:14:09.567328       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:14:09.567438       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:14:10.571841       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:14:10.577746       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:14:10.579041       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:14:10.579115       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:14:10.579204       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:14:33.194217       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:14:33.194488       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001300e9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:14:33.203751       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:14:33.203929       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:15:33.197368       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:15:33.197592       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a0c9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:15:33.204602       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:15:33.204716       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:15:33.204919       1 lvmnode.go:139] Got update event for lvm node openebs/alt-ksx-g-c01oco03
I0919 22:15:33.227390       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:15:33.227723       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a6f0 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:15:33.237650       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:15:33.237836       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:16:11.616127       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:16:11.621182       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:16:11.622289       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:16:11.622376       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:16:11.622450       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:16:12.629181       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:16:12.634759       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:16:12.638315       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:16:12.638330       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:16:12.638354       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:16:33.194321       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:16:33.194582       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001ab10d BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:16:33.200940       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:16:33.201043       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:17:33.195250       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:17:33.195449       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000800000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:17:33.203085       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:17:33.203146       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:18:13.705268       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:18:13.710264       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:18:13.711379       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:18:13.711399       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:18:13.711423       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:18:14.712131       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:18:14.717477       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:18:14.718574       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:18:14.718671       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:18:14.718739       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:18:33.197552       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:18:33.197616       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212598 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:18:33.203316       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:18:33.203334       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:19:33.198358       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:19:33.198425       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000130000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:19:33.204133       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:19:33.204295       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:20:15.780801       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:20:15.785438       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:20:15.786634       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:20:15.786799       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:20:15.786892       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:20:16.798260       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:20:16.805115       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:20:16.806421       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:20:16.806537       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:20:16.806623       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 

E0919 22:22:17.863627       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
E0919 22:22:17.863684       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
I0919 22:22:18.863589       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:22:18.869062       1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:22:18.870243       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:22:18.870264       1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
E0919 22:22:18.870295       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
) 
I0919 22:22:33.195255       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:22:33.195326       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006ac5ea BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:22:33.201762       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:22:33.201935       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:23:33.197380       1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:23:33.197443       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0000421f8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:23:33.204502       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:23:33.204519       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:24:19.937883       1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:24:19.942809       1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:24:19.943870       1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) 
...
I0919 22:25:33.199426       1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000130079 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:25:33.220225       1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:25:33.220325       1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'

kubectl get lvmvol -A -o yaml

apiVersion: v1
items:

- apiVersion: local.openebs.io/v1alpha1
  kind: LVMVolume
  metadata:
    creationTimestamp: "2021-09-19T16:58:40Z"
    finalizers:
    - lvm.openebs.io/finalizer
    generation: 3
    labels:
      kubernetes.io/nodename: alt-ksx-g-c01oco03
    name: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
    namespace: openebs
    resourceVersion: "1304748995"
    uid: 303e29b0-39f7-413a-9a1a-cecd32757061
  spec:
    capacity: "1073741824"
    ownerNodeID: alt-ksx-g-c01oco03
    shared: "yes"
    thinProvision: "no"
    vgPattern: ^datavg$
    volGroup: datavg
  status:
    state: Ready
- apiVersion: local.openebs.io/v1alpha1
  kind: LVMVolume
  metadata:
    creationTimestamp: "2021-09-19T17:48:47Z"
    finalizers:
    - lvm.openebs.io/finalizer
    generation: 3
    labels:
      kubernetes.io/nodename: alt-ksx-g-c01oco03
    name: pvc-63ebce02-7077-48ea-88b3-301b1cf7909d
    namespace: openebs
    resourceVersion: "1304943732"
    uid: 79c320fc-4a6f-4c0e-b02e-0d433d9cf74f
  spec:
    capacity: "1073741824"
    ownerNodeID: alt-ksx-g-c01oco03
    shared: "yes"
    thinProvision: "no"
    vgPattern: ^datavg$
    volGroup: datavg
  status:
    state: Ready

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

 pvscan --cache
[root@alt-ksx-g-c01oco03 ~]# lvscan 
  ACTIVE            '/dev/datavg/openebs' [5.00 GiB] inherit
  ACTIVE            '/dev/rootvg/swap_lv' [2.00 GiB] inherit
  ACTIVE            '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
  ACTIVE            '/dev/rootvg/var_lv' [150.00 GiB] inherit
  ACTIVE            '/dev/rootvg/home_lv' [1000.00 MiB] inherit
  ACTIVE            '/dev/rootvg/root_lv' [<6.84 GiB] inherit

k get pvc alertmanager-main-db-alertmanager-main-2  -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: local.csi.openebs.io
    volume.kubernetes.io/selected-node: alt-ksx-g-c01oco03
  creationTimestamp: "2021-09-19T16:58:40Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    alertmanager: main
    app: alertmanager
    app.kubernetes.io/instance: main
    app.kubernetes.io/managed-by: prometheus-operator
    app.kubernetes.io/name: alertmanager
    app.kubernetes.io/version: 0.21.0
  name: alertmanager-main-db-alertmanager-main-2
  namespace: openshift-monitoring
  resourceVersion: "1304749087"
  uid: 630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: openebs-lvmpv-ext4
  volumeMode: Filesystem
  volumeName: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

k describe pod alertmanager-main-2   |tail -10
  Warning  FailedMount       34m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy]: timed out waiting for the condition
  Warning  FailedMount       28m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls]: timed out waiting for the condition
  Warning  FailedMount       23m (x4 over 39m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy]: timed out waiting for the condition
  Warning  FailedMount       21m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets]: timed out waiting for the condition
  Warning  FailedMount       19m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume]: timed out waiting for the condition
  Warning  FailedMount       10m (x23 over 41m)  kubelet            MountVolume.SetUp failed for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" : rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
  Warning  FailedMount  5m35s              kubelet  Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle]: timed out waiting for the condition
  Warning  FailedMount  62s (x2 over 30m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb]: timed out waiting for the condition

k get pv|grep pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca   1Gi        RWO            Delete           Bound    openshift-monitoring/alertmanager-main-db-alertmanager-main-2                               openebs-lvmpv-ext4            5h30m

k get pv  pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: local.csi.openebs.io
  creationTimestamp: "2021-09-19T16:58:41Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
  resourceVersion: "1304749080"
  uid: c1f1c9dd-6440-459a-9e08-213e68657e45
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: alertmanager-main-db-alertmanager-main-2
    namespace: openshift-monitoring
    resourceVersion: "1304748971"
    uid: 630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
  csi:
    driver: local.csi.openebs.io
    fsType: ext4
    volumeAttributes:
      openebs.io/cas-type: localpv-lvm
      openebs.io/volgroup: datavg
      storage.kubernetes.io/csiProvisionerIdentity: 1631990632881-8081-local.csi.openebs.io
    volumeHandle: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: openebs.io/nodename
          operator: In
          values:
          - alt-ksx-g-c01oco03
  persistentVolumeReclaimPolicy: Delete
  storageClassName: openebs-lvmpv-ext4
  volumeMode: Filesystem
status:
  phase: Bound

oot@alt-ksx-g-c01oco03 ~]# lvm version
  LVM version:     2.02.187(2)-RHEL7 (2020-03-24)
  Library version: 1.02.170-RHEL7 (2020-03-24)
  Driver version:  4.37.1
  Configuration:   ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm --with-pool=internal --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --enable-python2-bindings --with-cluster=internal --with-clvmd=corosync --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --enable-lvmetad --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-dmfilemapd
[root@alt-ksx-g-c01oco03 ~]#

Environment:

LVM Driver version:
Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+9807387", GitCommit:"98073871f173baaa04dc2bafab50effd62c308a6", GitTreeState:"clean", BuildDate:"2021-08-06T14:06:43Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Kubernetes installer & version: openshift 4.8
Cloud provider or hardware configuration: n/a
OS (e.g. from /etc/os-release): RHEL 7.x

Sep 19 '21 22:09 davidkarlsen

see in the messages logs how it was fine before boot, then goes missing 40mins later when booting:

lertmanager-main-db") pod "550264f7-6dc2-4365-96f0-7d640a16b85f" (UID: "550264f7-6dc2-4365-96f0-7d640a16b85f"). InnerVolumeSpecName "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca". PluginName "kubernetes.io/csi", VolumeGidValue ""
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.195980    3082 reconciler.go:312] "operationExecutor.UnmountDevice started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") on node \"alt-ksx-g-c01oco03\" "
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.203350    3082 operation_generator.go:930] UnmountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" %!(EXTRA string=UnmountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" (UniqueName: "kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca") on node "alt-ksx-g-c01oco03" )
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.296581    3082 reconciler.go:319] "Volume detached for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") on node \"alt-ksx-g-c01oco03\" DevicePath \"\""
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.573075    3065 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") "
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.677388    3065 reconciler.go:269] "operationExecutor.MountVolume started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") "
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.685679    3065 operation_generator.go:616] MountVolume.MountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" (UniqueName: "kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca") pod "alertmanager-main-2" (UID: "2a1d2646-68b5-4916-99b3-18d3918d4831") device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/globalmount"
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: E0919 23:47:36.928568    3065 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca podName: nodeName:}" failed. No retries permitted until 2021-09-19 23:47:37.428526511 +0200 CEST m=+32.344727832 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") : rpc error: code = Internal desc = failed to format and mount the volume error: format of disk \"/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" failed: type:(\"ext4\") target:(\"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount\") options:(\"defaults\") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)\nThe file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.\n) "

I have seen this on several pvc's, so it's not a single isolated case.

helm -n openebs list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION openebs openebs 35 2021-09-16 23:51:51.718139269 +0000 UTC deployed openebs-2.12.7 2.12.1

Sep 19 '21 22:09 davidkarlsen

@davidkarlsen -- few questions:

What is the size of the volume group?
How many volumes are you creating on the volumes group and their total size, when the above issue starts to show up?
Can you run the lvm commands to see if the volumes really exist on the node and the capacity information of volumes and the corresponding vg.

Sep 20 '21 07:09 kmova

around 300G, plenty of space
a few 1-3
I've run pvscan --cache and lvscan, not to be found at all (!)

Sep 20 '21 07:09 davidkarlsen

this is the SC definition:

k get sc openebs-lvmpv-ext4  -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"ffec0c8cb91220cd2484eb20ea0dffa03a63bb68"},"labels":{"kustomize.toolkit.fluxcd.io/name":"flux-system","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"openebs-lvmpv-ext4"},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"provisioner":"local.csi.openebs.io","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
    kustomize.toolkit.fluxcd.io/checksum: ffec0c8cb91220cd2484eb20ea0dffa03a63bb68
  creationTimestamp: "2021-09-19T16:42:58Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: openebs-lvmpv-ext4
  resourceVersion: "1309101171"
  uid: ba5dd4a1-503b-45cd-8d2f-54aadf92753d
parameters:
  fsType: ext4
  shared: "yes"
  thinProvision: "no"
  volgroup: datavg
provisioner: local.csi.openebs.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Sep 20 '21 13:09 davidkarlsen

@davidkarlsen - I wonder if this may be related to https://access.redhat.com/solutions/2175491

Sep 20 '21 13:09 kmova

@davidkarlsen Currently we support ubuntu based systems only, if possible, try to change the host OS to ubuntu. Also, is the issue reproducible everytime you reboot the machine?

Sep 20 '21 13:09 pawanpraka1

@davidkarlsen Currently we support ubuntu based systems only, if possible, try to change the host OS to ubuntu. Also, is the issue reproducible everytime you reboot the machine?

not every single time, but very often. as this is openshift, we can't use ubuntu

Sep 20 '21 13:09 davidkarlsen

@davidkarlsen - I wonder if this may be related to access.redhat.com/solutions/2175491

Hmm, how can I figure out if multipath is in use? But I doubt it as we're on:

uname -a
Linux alt-ksx-g-c01oco01 3.10.0-1160.36.2.el7.x86_64 #1 SMP Thu Jul 8 02:53:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
-sh-4.2$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.9 (Maipo)

which should not be affected.

Sep 20 '21 13:09 davidkarlsen

How is the reboot of the machine triggered? Does it clear all the volumes? Can you try to capture the lvm config file after volumes are provisioned and post the reboot to check if this is actually a config sync issue?

Also we are trying to put together a troubleshooting guide for LVM metadata corruption cases and how to recover from them. Here is the draft version: https://docs.google.com/document/d/1qB7VjPHrCczLlSg9gX6X_3QWiuEOoObXINl9ovMKmjU/edit

(cc: @prateekpandey14 )

Sep 20 '21 14:09 kmova

How is the reboot of the machine triggered? Does it clear all the volumes? Can you try to capture the lvm config file after volumes are provisioned and post the reboot to check if this is actually a config sync issue?

No - it's not set to clear all volumes as reboot! :-) we cordon the node, then drain all but daemonsets, with options: --delete-emptydir-data --ignore-daemonsets

indeed I do not find any references to the volumes in the lvm config! (this is prior to any reboot):

lvscan 
  ACTIVE            '/dev/datavg/openebs' [5.00 GiB] inherit
  ACTIVE            '/dev/datavg/pvc-222d5156-3507-4f43-8323-8368a0b114d7' [1.00 GiB] inherit
  ACTIVE            '/dev/datavg/pvc-ef24242b-90ec-4f51-bc52-928e331b86d3' [100.00 GiB] inherit
  ACTIVE            '/dev/datavg/pvc-e997ee44-85f8-4b07-9785-ac9324acb1b5' [100.00 GiB] inherit
  ACTIVE            '/dev/rootvg/swap_lv' [2.00 GiB] inherit
  ACTIVE            '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
  ACTIVE            '/dev/rootvg/var_lv' [150.00 GiB] inherit
  ACTIVE            '/dev/rootvg/home_lv' [1000.00 MiB] inherit
  ACTIVE            '/dev/rootvg/root_lv' [<6.84 GiB] inherit

find /etc/lvm/ -type f|xargs fgrep -il 'pvc-222d5156-3507-4f43-8323-8368a0b114d7'
[root@alt-ksx-g-c01oco01 lvm]# find /etc/lvm/ -type f|xargs fgrep -il pvc-ef24242b-90ec-4f51-bc52-928e331b86d3
[root@alt-ksx-g-c01oco01 lvm]# find /etc/lvm/ -type f|xargs fgrep -il pvc-e997ee44-85f8-4b07-9785-ac9324acb1b5

Sep 20 '21 16:09 davidkarlsen

could this perhaps be a se-linux issue? I assume the lvm command are executed into the host-system from one of the openebs containers, and that might not have enough privilegies?

SCCs in openshift: https://docs.openshift.com/container-platform/4.8/authentication/managing-security-context-constraints.html#default-sccs_configuring-internal-oauth

k get rolebinding system:openshift:scc:privileged-openebs -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"ffec0c8cb91220cd2484eb20ea0dffa03a63bb68"},"labels":{"kustomize.toolkit.fluxcd.io/name":"flux-system","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"system:openshift:scc:privileged-openebs","namespace":"openebs"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"system:openshift:scc:privileged"},"subjects":[{"kind":"ServiceAccount","name":"openebs","namespace":"openebs"},{"kind":"ServiceAccount","name":"openebs-lvm-node-sa","namespace":"openebs"}]}
    kustomize.toolkit.fluxcd.io/checksum: ffec0c8cb91220cd2484eb20ea0dffa03a63bb68
  creationTimestamp: "2020-12-19T13:04:22Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: system:openshift:scc:privileged-openebs
  namespace: openebs
  resourceVersion: "1309101387"
  uid: f1e32386-120c-40a7-b624-5b5188539d34
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:privileged
subjects:
- kind: ServiceAccount
  name: openebs
  namespace: openebs
- kind: ServiceAccount
  name: openebs-lvm-node-sa
  namespace: openebs

Sep 20 '21 16:09 davidkarlsen

This should be of interest:

docker run --rm -ti --entrypoint /bin/sh k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown.

and from the pod:

    image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - rm -rf /registration/lvm-localpv /registration/lvm-localpv-reg.sock
    name: csi-node-driver-registrar

i.e. the lifecycle-hook does not run!

Sep 20 '21 17:09 davidkarlsen

@davidkarlsen -- the above information is really helpful! container permissions and location where the configuration is saved do sound like the possible causes for this issue.

Just spoke to @pawanpraka1 - and he wants to test out a few things. Will get back to this thread in a bit.

Sep 21 '21 11:09 kmova

i.e. the lifecycle-hook does not run!

That is a good find, but should not cause the issue. There will be stale socket file present on the system because of that.

@davidkarlsen can you execute pvscan --cache (and lvscan) command before doing node reboot. I am wondering whether node cache is not updated before reboot, if not, we might lose the volumes.

Sep 21 '21 11:09 pawanpraka1

@davidkarlsen can you execute pvscan --cache (and lvscan) command before doing node reboot. I am wondering whether node cache is not updated before reboot, if not, we might lose the volumes.

It seems to not persist:

oco02 ~]# pvscan --cache
[root@alt-ksx-g-c01oco02 ~]# lvscan 
  ACTIVE            '/dev/datavg/openebs' [5.00 GiB] inherit
  ACTIVE            '/dev/datavg/pvc-6e5a7f2f-fc50-443e-859d-d6f76de94064' [1.00 GiB] inherit
  ACTIVE            '/dev/rootvg/swap_lv' [2.00 GiB] inherit
  ACTIVE            '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
  ACTIVE            '/dev/rootvg/var_lv' [150.00 GiB] inherit
  ACTIVE            '/dev/rootvg/home_lv' [1000.00 MiB] inherit
  ACTIVE            '/dev/rootvg/root_lv' [<6.84 GiB] inherit
[root@alt-ksx-g-c01oco02 ~]# cd /etc/
[root@alt-ksx-g-c01oco02 etc]# cd lvm/
[root@alt-ksx-g-c01oco02 lvm]# find . -type f|xargs fgrep -il  vc-6e5a7f2f-fc50-443e-859d-d6f76
[root@alt-ksx-g-c01oco02 lvm]#

Sep 21 '21 14:09 davidkarlsen

Ok, so I've tried creating a bunch of volumes and rebooting machines. Two changes:

I noticed the previous openebs install was not 100%, because in some upgrade the storageCapacity flag went from false to true, so the helm install was not complete, as this flag is immutable. I deleted the csidriver and let the chart install it again which then succeeded.
I previously used XFS-based filesystems, these failed formatting by openebs due to https://github.com/openebs/lvm-localpv/issues/135 (which is really a kubernetes issue) - so these volumes were formatted by hand, now I use ext4 across the line.

I haven't seen the problem since, so I think we close this and I'll reopen if I can provoke it consistently.

Sep 29 '21 21:09 davidkarlsen

We occur this issue too!

After a reboot, we found a lv disappear while other lvs are normal. R6 M7ZPTIVO}V 86A)R2

The lost lv coundn't be found in /dev/mapper or /dev/lvmvg. There isn't any clue in the openebs pods logs or the lvm services

And found one wired thing is that there's no record of pvc-070f0023-d09e-4041-9cb9-dbd1a579e717 in /etc/lvm/archive. But the lv is surely created successfully and the pod worked well before restart. pvscan --cache return no error

root@master1:/etc/lvm/archive# lvm version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.41.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync

Could anyone take a look at this serious issue? Thanks in advance!

Aug 03 '22 02:08 npu21

@davidkarlsen Could you reopen this issue?

Aug 03 '22 02:08 npu21

@kmova @pawanpraka1 Could you please take a look?

Aug 03 '22 09:08 npu21

@davidkarlsen Could you reopen this issue?

reopened

Aug 03 '22 16:08 davidkarlsen

@npu21 what OS do you run on the nodes?

Aug 03 '22 16:08 davidkarlsen

@davidkarlsen Thanks! Linux version 5.4.61-050461-generic (kernel@kathleen) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #202008260931 SMP Wed Aug 26 09:34:29 UTC 2020

Aug 04 '22 02:08 npu21

one more clue is the lvm logs

Aug 04 '22 03:08 npu21

I am currently evaluating some local storage options and reading through issues here and elsewhere but thought I might chime in. This is sounding like a case of openebs-lvm-node not having an exclusive lock on the metadata of the volume group (ie lvm2-lvmmetad.service is interfering). I do not think it is a good idea having multiple services operating on the volume group at once. Could you try to run some tests with lvm2-lvmmetad.service stopped/disabled? It is not even technically required to be running for the pvcreate and vgcreate steps in the install instructions.

Aug 22 '22 23:08 gunzy83

@gunzy83 Agree! There seems need to add an exclusive lock on createing and delete logic volumes @wrtabhi611 Could you please take a look at this ? When going through the source code, I found there are places operating pvs,vgs,lvs supposing atomic operation, which are not in fact. Thanks in advance!

Aug 24 '22 08:08 npu21

@gunzy83 Agree! There seems need to add an exclusive lock on createing and delete logic volumes

I wonder if lvm2-lvmmetad.service operates with the same semantics (assuming it has exclusive use of the pvs/vgs/lvs)? If so adding logic into openebs would be the way to go. If not, providing some general guidance in the documentation that openebs-lvm-node needs exclusive use would be prudent.

Aug 24 '22 09:08 gunzy83

Why would this be needed? Doesn't the service run normally, and users can run LVM commands without LVM getting corrupt?

Aug 24 '22 09:08 davidkarlsen

Only a theory but users on the host would be running those commands from the same namespace I assume. Inside a container you would need to break namespace isolation for commands/services on the host to be able to coordinate with the commands being run from the container. Would be happy to be proven wrong.

For reference, Topolvm look to have solved the issue they had, they were running their lvmd on the host to avoid it until they implemented a solution using nsenter and can now run it in a daemonset. carina.io also use nsenter to escape isolation but I can't find any issues on Github to follow that show how they came to that.

Aug 24 '22 10:08 gunzy83

@davidkarlsen @gunzy83 In fact, we have reproduced this issue more than once. Procedure

These 9 pvcs have to occupy the vg full
1. Execute mkdir -p /root/lvmtest/pvc to create 9 pvc yaml files, execute kubectl apply -f /root/lvmtest/pvc/ to create 9 pvcs, 9 pvcs will be the vg where lvm is located full
2. Execute mkdir -p /root/lvmtest/pod to create 9 pod yaml files, execute kubectl apply -f /root/lvmtest/pod/ to create 9 pods and mount the pvc created in the previous step respectively
3. Execute kubectl get pvc to see that all 9 pvcs are Bound
4. Execute lvs to see if there are 9 corresponding lvs
5. Execute kubectl delete -f /root/lvmtest/pod/, kubectl delete -f /root/lvmtest/pvc/, delete pod and pvc, and then execute kubectl get pvc to see if there is any pvc
6. Execute kubectl apply -f /root/lvmtest/pvc/ to create 9 pvcs
7. Execute kubectl apply -f /root/lvmtest/pod/ to create 9 pods and mount the pvc created in the previous step respectively
8.1. Execute kubectl delete lvmvolume -n kube-system pvc-xxx, delete the lv created in the first round in order from small to large
8.2. Execute lvremove, delete the lv created in the first round in order from small to large
9. Execute kubectl get pvc to check whether the 9 pvcs are Bound
10. Execute lvs to see if there are 9 corresponding lvs, execute ls /dev/mapper, ls /dev/lvmvg to see if there are 9 corresponding lvm directories

After runing above steps circularly ,there are chances to reproduct the issue. The lvs will display <9 LVs , and the lost LV will disappear forever after reboot.

Aug 25 '22 03:08 npu21

@davidkarlsen Did you notice this locking_type in /etc/lvm/lvm.conf ? My test env didn't config this previously. I've configured this and retest now

Aug 25 '22 07:08 npu21

lvm-localpv lvm-localpv copied to clipboard

Volumes go missing.

lvm-localpv
lvm-localpv copied to clipboard