lvm-localpv
lvm-localpv copied to clipboard
Published
20 hours ago •
openebs
openebs
Volumes go missing.
What steps did you take and what happened: Created a pvc, which got formatted and mounted just fine. Then booted the machine, after bootup the volume is not to be found! This causes loss of data and is very severe.
What did you expect to happen: Not loosing volumes!
The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other Pastebin is fine.)
-
kubectl logs -f openebs-lvm-controller-0 -n kube-system -c openebs-lvm-plugin
I0919 21:07:44.486525 1 main.go:136] LVM Driver Version :- 0.8.0 - commit :- 929ae4439f2da71a2d6ee5bda6a33dd2f7d424fc
I0919 21:07:44.486615 1 main.go:137] DriverName: local.csi.openebs.io Plugin: controller EndPoint: unix:///var/lib/csi/sockets/pluginproxy/csi.sock NodeID: SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I0919 21:07:44.486646 1 driver.go:48] enabling volume access mode: SINGLE_NODE_WRITER
I0919 21:07:44.496097 1 controller.go:212] waiting for k8s & lvm node informer caches to be synced
I0919 21:07:44.596266 1 controller.go:216] synced k8s & lvm node informer caches
I0919 21:07:44.596300 1 controller.go:218] initializing csi provisioning leak protection controller
I0919 21:07:44.596895 1 leak_protection.go:129] "starting up csi pvc controller"
I0919 21:07:44.596916 1 shared_informer.go:240] Waiting for caches to sync for CSI Provisioner
I0919 21:07:44.597275 1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
I0919 21:07:44.697073 1 shared_informer.go:247] Caches are synced for CSI Provisioner
I0919 21:07:44.991733 1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:44.999709 1 grpc.go:81] GRPC response: {}
I0919 21:07:45.001288 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.001331 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.002064 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0919 21:07:45.002142 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0919 21:07:45.002973 1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.003009 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0919 21:07:45.325452 1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:45.325666 1 grpc.go:81] GRPC response: {}
I0919 21:07:45.326637 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.326779 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.327324 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0919 21:07:45.327354 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0919 21:07:45.328054 1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.328135 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0919 21:07:45.673459 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.673695 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.675001 1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:45.675122 1 grpc.go:81] GRPC response: {}
I0919 21:07:45.675962 1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.676000 1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
....
I0919 22:20:15.437512 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco06","openebs.io/nodename":"alt-ksx-g-c01oco06"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.437667 1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.438064 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-eos-g-c01oco03","openebs.io/nodename":"alt-eos-g-c01oco03"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.438195 1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.438548 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ebs-g-c01oco01","openebs.io/nodename":"alt-ebs-g-c01oco01"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.438674 1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.439082 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco09","openebs.io/nodename":"alt-ksx-g-c01oco09"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.439263 1 grpc.go:81] GRPC response: {"available_capacity":2171097579520}
I0919 22:20:15.439687 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-eos-g-c01oco02","openebs.io/nodename":"alt-eos-g-c01oco02"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.439817 1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.440257 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco06","openebs.io/nodename":"alt-ksx-g-c01oco06"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.440385 1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.440940 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco04","openebs.io/nodename":"alt-ksx-g-c01oco04"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.441111 1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.441526 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco07","openebs.io/nodename":"alt-ksx-g-c01oco07"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.441646 1 grpc.go:81] GRPC response: {"available_capacity":2171097579520}
I0919 22:20:15.441994 1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ebs-g-c01oco01","openebs.io/nodename":"alt-ebs-g-c01oco01"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.442129 1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
-
kubectl logs -f openebs-lvm-node-[xxxx] -n kube-system -c openebs-lvm-plugin -
kubectl get pods -n kube-system
k -n openebs get pods
NAME READY STATUS RESTARTS AGE
openebs-admission-server-894884484-gcc5f 1/1 Running 0 29h
openebs-apiserver-6494d57b96-xfltt 1/1 Running 0 77m
openebs-localpv-provisioner-5787758b59-tzs8l 1/1 Running 1 29h
openebs-lvm-localpv-controller-0 5/5 Running 0 76m
openebs-lvm-localpv-node-7x2kw 2/2 Running 6 5d4h
openebs-lvm-localpv-node-98kxr 2/2 Running 6 5d4h
openebs-lvm-localpv-node-9r2kd 2/2 Running 20 5d4h
openebs-lvm-localpv-node-h949v 2/2 Running 6 5d4h
openebs-lvm-localpv-node-jff5w 2/2 Running 6 5d4h
openebs-lvm-localpv-node-kccx5 2/2 Running 6 5d4h
openebs-lvm-localpv-node-ln7r4 2/2 Running 6 5d4h
openebs-lvm-localpv-node-pzxw4 2/2 Running 6 5d4h
openebs-lvm-localpv-node-qxwn9 2/2 Running 11 5d4h
openebs-lvm-localpv-node-qzxzb 2/2 Running 6 5d4h
openebs-lvm-localpv-node-trfvk 2/2 Running 6 5d4h
openebs-lvm-localpv-node-w25kv 2/2 Running 6 5d4h
openebs-lvm-localpv-node-wxv6w 2/2 Running 6 5d4h
openebs-lvm-localpv-node-xkmzj 2/2 Running 6 5d4h
openebs-ndm-5rq88 1/1 Running 5 10d
openebs-ndm-ftnwd 1/1 Running 3 10d
openebs-ndm-k9z5w 1/1 Running 3 10d
openebs-ndm-mxzss 1/1 Running 3 10d
openebs-ndm-operator-7bd7455b5b-khrmt 1/1 Running 0 29h
openebs-ndm-p79zz 1/1 Running 3 10d
openebs-ndm-p7f77 1/1 Running 3 10d
openebs-ndm-phfjw 1/1 Running 3 10d
openebs-ndm-zsqmp 1/1 Running 3 10d
openebs-provisioner-9f948db4-ljndj 1/1 Running 1 29h
openebs-snapshot-operator-68cf9946d6-vc5ls 2/2 Running 0 77m
I0919 21:47:33.032487 1 main.go:136] LVM Driver Version :- 0.8.0 - commit :- 929ae4439f2da71a2d6ee5bda6a33dd2f7d424fc
I0919 21:47:33.032679 1 main.go:137] DriverName: local.csi.openebs.io Plugin: agent EndPoint: unix:///plugin/csi.sock NodeID: alt-ksx-g-c01oco03 SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I0919 21:47:33.032725 1 driver.go:48] enabling volume access mode: SINGLE_NODE_WRITER
I0919 21:47:33.033388 1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//plugin/csi.sock", Net:"unix"}
I0919 21:47:33.036307 1 builder.go:107] Creating event broadcaster
I0919 21:47:33.036946 1 builder.go:107] Creating event broadcaster
I0919 21:47:33.037042 1 snapshot.go:165] Starting Snap controller
I0919 21:47:33.037049 1 snapshot.go:168] Waiting for informer caches to sync
I0919 21:47:33.036950 1 volume.go:265] Starting Vol controller
I0919 21:47:33.037584 1 volume.go:268] Waiting for informer caches to sync
I0919 21:47:33.055970 1 volume.go:209] Got add event for Vol pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
I0919 21:47:33.056056 1 volume.go:209] Got add event for Vol pvc-63ebce02-7077-48ea-88b3-301b1cf7909d
I0919 21:47:33.067860 1 builder.go:116] Creating event broadcaster
I0919 21:47:33.068028 1 lvmnode.go:191] Starting Node controller
I0919 21:47:33.068466 1 lvmnode.go:194] Waiting for informer caches to sync
I0919 21:47:33.071142 1 lvmnode.go:127] Got add event for lvm node openebs/alt-ksx-g-c01oco03
I0919 21:47:33.137317 1 snapshot.go:172] Starting Snap workers
I0919 21:47:33.137350 1 snapshot.go:179] Started Snap workers
I0919 21:47:33.137728 1 volume.go:272] Starting Vol workers
I0919 21:47:33.137738 1 volume.go:279] Started Vol workers
I0919 21:47:33.137830 1 volume.go:98] lvm volume already provisioned
I0919 21:47:33.137840 1 volume.go:337] Successfully synced 'openebs/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca'
I0919 21:47:33.137832 1 volume.go:98] lvm volume already provisioned
I0919 21:47:33.137904 1 volume.go:337] Successfully synced 'openebs/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d'
I0919 21:47:33.168649 1 lvmnode.go:199] Starting Node workers
I0919 21:47:33.168667 1 lvmnode.go:206] Started Node workers
I0919 21:47:33.489540 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:47:33.489636 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a060 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:47:33.497190 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:47:33.497209 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:47:33.754135 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:47:33.755194 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:47:35.327404 1 grpc.go:72] GRPC call: /csi.v1.Node/NodeGetInfo requests {}
I0919 21:47:35.334632 1 grpc.go:81] GRPC response: {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco03","openebs.io/nodename":"alt-ksx-g-c01oco03"}},"node_id":"alt-ksx-g-c01oco03"}
I0919 21:47:36.692782 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:36.762805 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:36.927823 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:47:36.927901 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:47:36.927997 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 21:47:37.493233 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:37.498794 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:37.499905 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:47:37.499975 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:47:37.500016 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 21:47:37.818527 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:47:37.823301 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:47:37.824462 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:47:37.824478 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:47:37.824495 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 21:47:38.411354 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:47:38.417422 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:47:38.418586 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:47:38.418630 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:47:38.418687 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 21:47:38.520481 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:47:38.525817 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:47:38.526895 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
) ...
I0919 21:48:33.198169 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:48:33.198235 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006224c9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:48:33.205156 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:48:33.205174 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:48:40.776987 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:48:40.793493 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:48:40.794577 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:48:40.794632 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:48:40.794757 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 21:48:41.690835 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:48:41.697249 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:48:41.698425 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:48:41.698442 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:48:41.698463 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 21:58:33.193143 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:58:33.193387 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026bce8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:58:33.200591 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:58:33.200709 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:59:33.197599 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 21:59:33.197854 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006320f9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 21:59:33.203510 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 21:59:33.203600 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 21:59:55.071045 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 21:59:55.077098 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 21:59:55.078420 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:59:55.078595 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 21:59:55.078709 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 21:59:56.077218 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 21:59:56.081863 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 21:59:56.083050 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:59:56.083154 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 21:59:56.083251 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:00:33.194333 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:00:33.194546 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000131069 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:00:33.200632 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:00:33.200795 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:01:33.192396 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:01:33.192458 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006a808a BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:01:33.199002 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:01:33.199019 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:01:57.130951 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:01:57.136043 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:01:57.137124 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:01:57.137250 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:01:57.137358 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:01:58.139894 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:01:58.145250 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:01:58.146369 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:01:58.146436 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:01:58.146499 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:02:33.198541 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:02:33.198810 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000632248 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:02:33.219623 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:02:33.219775 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:03:33.197462 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:03:33.197666 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006221bb BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:03:33.204957 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:03:33.205147 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:03:59.214157 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:03:59.221739 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:03:59.223076 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:03:59.223095 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:03:59.223124 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:04:00.225863 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:04:00.230920 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:04:00.232305 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:04:00.232440 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:04:00.232540 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:04:33.193373 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:04:33.193540 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000043278 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:04:33.199806 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:04:33.199958 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:05:33.201598 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:05:33.201968 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0003e00aa BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:05:33.209690 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:05:33.209786 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:06:01.309472 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:06:01.315687 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:06:01.316691 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:06:01.316707 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:06:01.316734 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:06:02.317247 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:06:02.321985 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:06:02.323009 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:06:02.323099 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:06:02.323181 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:06:33.192108 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:06:33.192174 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212090 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:06:33.203863 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:06:33.203885 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:07:33.197599 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:07:33.197883 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:07:33.204772 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:07:33.204789 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:08:03.336415 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:08:03.343673 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:08:03.344630 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:08:03.344645 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:08:03.344675 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:08:04.353996 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:08:04.364752 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:08:04.365823 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:08:04.365942 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:08:04.366035 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:08:33.195376 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:08:33.195672 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006ac7e8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:08:33.205158 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:08:33.205327 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:09:33.195347 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:09:33.195593 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0005901da BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:09:33.204502 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:09:33.204669 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:10:05.398785 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:10:05.405224 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:10:05.406422 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:10:05.406541 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:10:05.406634 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:10:06.407071 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:10:06.412408 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:10:06.413575 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:10:06.413705 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:10:06.413795 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:10:33.195388 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:10:33.195460 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000622978 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:10:33.209769 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:10:33.209910 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:11:33.195346 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:11:33.195445 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0000421f0 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:11:33.206672 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:11:33.206754 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:12:07.452479 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:12:07.457948 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:12:07.459041 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:12:07.459053 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:12:07.459073 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:12:08.468111 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:12:08.473579 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:12:08.474930 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:12:08.475057 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:12:08.475153 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:12:33.199283 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:12:33.199554 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001ab668 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:12:33.208417 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:12:33.208537 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:13:33.194398 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:13:33.194604 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001aa000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:13:33.202748 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:13:33.202825 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:14:09.561523 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:14:09.566213 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:14:09.567230 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:14:09.567328 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:14:09.567438 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:14:10.571841 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:14:10.577746 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:14:10.579041 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:14:10.579115 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:14:10.579204 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:14:33.194217 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:14:33.194488 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001300e9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:14:33.203751 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:14:33.203929 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:15:33.197368 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:<nil>} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:15:33.197592 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a0c9 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:15:33.204602 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:15:33.204716 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:15:33.204919 1 lvmnode.go:139] Got update event for lvm node openebs/alt-ksx-g-c01oco03
I0919 22:15:33.227390 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:15:33.227723 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a6f0 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:15:33.237650 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:15:33.237836 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:16:11.616127 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:16:11.621182 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:16:11.622289 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:16:11.622376 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:16:11.622450 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:16:12.629181 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:16:12.634759 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:16:12.638315 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:16:12.638330 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:16:12.638354 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:16:33.194321 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:16:33.194582 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001ab10d BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:16:33.200940 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:16:33.201043 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:17:33.195250 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:17:33.195449 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000800000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:17:33.203085 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:17:33.203146 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:18:13.705268 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:18:13.710264 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:18:13.711379 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:18:13.711399 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:18:13.711423 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:18:14.712131 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:18:14.717477 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:18:14.718574 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:18:14.718671 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:18:14.718739 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:18:33.197552 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:18:33.197616 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212598 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:18:33.203316 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:18:33.203334 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:19:33.198358 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:19:33.198425 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000130000 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:19:33.204133 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:19:33.204295 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:20:15.780801 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:20:15.785438 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:20:15.786634 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:20:15.786799 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:20:15.786892 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:20:16.798260 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:20:16.805115 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:20:16.806421 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:20:16.806537 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:20:16.806623 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:22:17.863627 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
E0919 22:22:17.863684 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
I0919 22:22:18.863589 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"}
I0919 22:22:18.869062 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d]
E0919 22:22:18.870243 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:22:18.870264 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
E0919 22:22:18.870295 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.
)
I0919 22:22:33.195255 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:22:33.195326 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006ac5ea BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:22:33.201762 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:22:33.201935 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:23:33.197380 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:<nil>} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]
I0919 22:23:33.197443 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0000421f8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:23:33.204502 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:23:33.204519 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
I0919 22:24:19.937883 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"}
I0919 22:24:19.942809 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca]
E0919 22:24:19.943870 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
...
I0919 22:25:33.199426 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000130079 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:<nil>} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]}
I0919 22:25:33.220225 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03
I0919 22:25:33.220325 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'
-
kubectl get lvmvol -A -o yaml
apiVersion: v1
items:
- apiVersion: local.openebs.io/v1alpha1
kind: LVMVolume
metadata:
creationTimestamp: "2021-09-19T16:58:40Z"
finalizers:
- lvm.openebs.io/finalizer
generation: 3
labels:
kubernetes.io/nodename: alt-ksx-g-c01oco03
name: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
namespace: openebs
resourceVersion: "1304748995"
uid: 303e29b0-39f7-413a-9a1a-cecd32757061
spec:
capacity: "1073741824"
ownerNodeID: alt-ksx-g-c01oco03
shared: "yes"
thinProvision: "no"
vgPattern: ^datavg$
volGroup: datavg
status:
state: Ready
- apiVersion: local.openebs.io/v1alpha1
kind: LVMVolume
metadata:
creationTimestamp: "2021-09-19T17:48:47Z"
finalizers:
- lvm.openebs.io/finalizer
generation: 3
labels:
kubernetes.io/nodename: alt-ksx-g-c01oco03
name: pvc-63ebce02-7077-48ea-88b3-301b1cf7909d
namespace: openebs
resourceVersion: "1304943732"
uid: 79c320fc-4a6f-4c0e-b02e-0d433d9cf74f
spec:
capacity: "1073741824"
ownerNodeID: alt-ksx-g-c01oco03
shared: "yes"
thinProvision: "no"
vgPattern: ^datavg$
volGroup: datavg
status:
state: Ready
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
pvscan --cache
[root@alt-ksx-g-c01oco03 ~]# lvscan
ACTIVE '/dev/datavg/openebs' [5.00 GiB] inherit
ACTIVE '/dev/rootvg/swap_lv' [2.00 GiB] inherit
ACTIVE '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
ACTIVE '/dev/rootvg/var_lv' [150.00 GiB] inherit
ACTIVE '/dev/rootvg/home_lv' [1000.00 MiB] inherit
ACTIVE '/dev/rootvg/root_lv' [<6.84 GiB] inherit
k get pvc alertmanager-main-db-alertmanager-main-2 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: local.csi.openebs.io
volume.kubernetes.io/selected-node: alt-ksx-g-c01oco03
creationTimestamp: "2021-09-19T16:58:40Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
alertmanager: main
app: alertmanager
app.kubernetes.io/instance: main
app.kubernetes.io/managed-by: prometheus-operator
app.kubernetes.io/name: alertmanager
app.kubernetes.io/version: 0.21.0
name: alertmanager-main-db-alertmanager-main-2
namespace: openshift-monitoring
resourceVersion: "1304749087"
uid: 630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: openebs-lvmpv-ext4
volumeMode: Filesystem
volumeName: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
phase: Bound
k describe pod alertmanager-main-2 |tail -10
Warning FailedMount 34m kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy]: timed out waiting for the condition
Warning FailedMount 28m kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls]: timed out waiting for the condition
Warning FailedMount 23m (x4 over 39m) kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy]: timed out waiting for the condition
Warning FailedMount 21m kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets]: timed out waiting for the condition
Warning FailedMount 19m kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume]: timed out waiting for the condition
Warning FailedMount 10m (x23 over 41m) kubelet MountVolume.SetUp failed for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" : rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
Warning FailedMount 5m35s kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle]: timed out waiting for the condition
Warning FailedMount 62s (x2 over 30m) kubelet Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb]: timed out waiting for the condition
k get pv|grep pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca 1Gi RWO Delete Bound openshift-monitoring/alertmanager-main-db-alertmanager-main-2 openebs-lvmpv-ext4 5h30m
k get pv pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: local.csi.openebs.io
creationTimestamp: "2021-09-19T16:58:41Z"
finalizers:
- kubernetes.io/pv-protection
name: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
resourceVersion: "1304749080"
uid: c1f1c9dd-6440-459a-9e08-213e68657e45
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: alertmanager-main-db-alertmanager-main-2
namespace: openshift-monitoring
resourceVersion: "1304748971"
uid: 630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
csi:
driver: local.csi.openebs.io
fsType: ext4
volumeAttributes:
openebs.io/cas-type: localpv-lvm
openebs.io/volgroup: datavg
storage.kubernetes.io/csiProvisionerIdentity: 1631990632881-8081-local.csi.openebs.io
volumeHandle: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: openebs.io/nodename
operator: In
values:
- alt-ksx-g-c01oco03
persistentVolumeReclaimPolicy: Delete
storageClassName: openebs-lvmpv-ext4
volumeMode: Filesystem
status:
phase: Bound
oot@alt-ksx-g-c01oco03 ~]# lvm version
LVM version: 2.02.187(2)-RHEL7 (2020-03-24)
Library version: 1.02.170-RHEL7 (2020-03-24)
Driver version: 4.37.1
Configuration: ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm --with-pool=internal --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --enable-python2-bindings --with-cluster=internal --with-clvmd=corosync --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --enable-lvmetad --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-dmfilemapd
[root@alt-ksx-g-c01oco03 ~]#
Environment:
- LVM Driver version:
- Kubernetes version (use
kubectl version): Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+9807387", GitCommit:"98073871f173baaa04dc2bafab50effd62c308a6", GitTreeState:"clean", BuildDate:"2021-08-06T14:06:43Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} - Kubernetes installer & version: openshift 4.8
- Cloud provider or hardware configuration: n/a
- OS (e.g. from
/etc/os-release): RHEL 7.x
see in the messages logs how it was fine before boot, then goes missing 40mins later when booting:
lertmanager-main-db") pod "550264f7-6dc2-4365-96f0-7d640a16b85f" (UID: "550264f7-6dc2-4365-96f0-7d640a16b85f"). InnerVolumeSpecName "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca". PluginName "kubernetes.io/csi", VolumeGidValue ""
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.195980 3082 reconciler.go:312] "operationExecutor.UnmountDevice started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") on node \"alt-ksx-g-c01oco03\" "
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.203350 3082 operation_generator.go:930] UnmountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" %!(EXTRA string=UnmountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" (UniqueName: "kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca") on node "alt-ksx-g-c01oco03" )
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.296581 3082 reconciler.go:319] "Volume detached for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") on node \"alt-ksx-g-c01oco03\" DevicePath \"\""
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.573075 3065 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") "
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.677388 3065 reconciler.go:269] "operationExecutor.MountVolume started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") "
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.685679 3065 operation_generator.go:616] MountVolume.MountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" (UniqueName: "kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca") pod "alertmanager-main-2" (UID: "2a1d2646-68b5-4916-99b3-18d3918d4831") device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/globalmount"
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: E0919 23:47:36.928568 3065 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca podName: nodeName:}" failed. No retries permitted until 2021-09-19 23:47:37.428526511 +0200 CEST m=+32.344727832 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") : rpc error: code = Internal desc = failed to format and mount the volume error: format of disk \"/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" failed: type:(\"ext4\") target:(\"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount\") options:(\"defaults\") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)\nThe file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.\n) "
I have seen this on several pvc's, so it's not a single isolated case.
helm -n openebs list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION openebs openebs 35 2021-09-16 23:51:51.718139269 +0000 UTC deployed openebs-2.12.7 2.12.1
@davidkarlsen -- few questions:
- What is the size of the volume group?
- How many volumes are you creating on the volumes group and their total size, when the above issue starts to show up?
- Can you run the lvm commands to see if the volumes really exist on the node and the capacity information of volumes and the corresponding vg.
- around 300G, plenty of space
- a few 1-3
- I've run pvscan --cache and lvscan, not to be found at all (!)
this is the SC definition:
k get sc openebs-lvmpv-ext4 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"ffec0c8cb91220cd2484eb20ea0dffa03a63bb68"},"labels":{"kustomize.toolkit.fluxcd.io/name":"flux-system","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"openebs-lvmpv-ext4"},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"provisioner":"local.csi.openebs.io","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
kustomize.toolkit.fluxcd.io/checksum: ffec0c8cb91220cd2484eb20ea0dffa03a63bb68
creationTimestamp: "2021-09-19T16:42:58Z"
labels:
kustomize.toolkit.fluxcd.io/name: flux-system
kustomize.toolkit.fluxcd.io/namespace: flux-system
name: openebs-lvmpv-ext4
resourceVersion: "1309101171"
uid: ba5dd4a1-503b-45cd-8d2f-54aadf92753d
parameters:
fsType: ext4
shared: "yes"
thinProvision: "no"
volgroup: datavg
provisioner: local.csi.openebs.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
@davidkarlsen - I wonder if this may be related to https://access.redhat.com/solutions/2175491
@davidkarlsen Currently we support ubuntu based systems only, if possible, try to change the host OS to ubuntu. Also, is the issue reproducible everytime you reboot the machine?
@davidkarlsen Currently we support ubuntu based systems only, if possible, try to change the host OS to ubuntu. Also, is the issue reproducible everytime you reboot the machine?
not every single time, but very often. as this is openshift, we can't use ubuntu
@davidkarlsen - I wonder if this may be related to access.redhat.com/solutions/2175491
Hmm, how can I figure out if multipath is in use? But I doubt it as we're on:
uname -a
Linux alt-ksx-g-c01oco01 3.10.0-1160.36.2.el7.x86_64 #1 SMP Thu Jul 8 02:53:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
-sh-4.2$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
which should not be affected.
How is the reboot of the machine triggered? Does it clear all the volumes? Can you try to capture the lvm config file after volumes are provisioned and post the reboot to check if this is actually a config sync issue?
Also we are trying to put together a troubleshooting guide for LVM metadata corruption cases and how to recover from them. Here is the draft version: https://docs.google.com/document/d/1qB7VjPHrCczLlSg9gX6X_3QWiuEOoObXINl9ovMKmjU/edit
(cc: @prateekpandey14 )
How is the reboot of the machine triggered? Does it clear all the volumes? Can you try to capture the lvm config file after volumes are provisioned and post the reboot to check if this is actually a config sync issue?
No - it's not set to clear all volumes as reboot! :-) we cordon the node, then drain all but daemonsets, with options: --delete-emptydir-data --ignore-daemonsets
indeed I do not find any references to the volumes in the lvm config! (this is prior to any reboot):
lvscan
ACTIVE '/dev/datavg/openebs' [5.00 GiB] inherit
ACTIVE '/dev/datavg/pvc-222d5156-3507-4f43-8323-8368a0b114d7' [1.00 GiB] inherit
ACTIVE '/dev/datavg/pvc-ef24242b-90ec-4f51-bc52-928e331b86d3' [100.00 GiB] inherit
ACTIVE '/dev/datavg/pvc-e997ee44-85f8-4b07-9785-ac9324acb1b5' [100.00 GiB] inherit
ACTIVE '/dev/rootvg/swap_lv' [2.00 GiB] inherit
ACTIVE '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
ACTIVE '/dev/rootvg/var_lv' [150.00 GiB] inherit
ACTIVE '/dev/rootvg/home_lv' [1000.00 MiB] inherit
ACTIVE '/dev/rootvg/root_lv' [<6.84 GiB] inherit
find /etc/lvm/ -type f|xargs fgrep -il 'pvc-222d5156-3507-4f43-8323-8368a0b114d7'
[root@alt-ksx-g-c01oco01 lvm]# find /etc/lvm/ -type f|xargs fgrep -il pvc-ef24242b-90ec-4f51-bc52-928e331b86d3
[root@alt-ksx-g-c01oco01 lvm]# find /etc/lvm/ -type f|xargs fgrep -il pvc-e997ee44-85f8-4b07-9785-ac9324acb1b5
could this perhaps be a se-linux issue? I assume the lvm command are executed into the host-system from one of the openebs containers, and that might not have enough privilegies?
SCCs in openshift: https://docs.openshift.com/container-platform/4.8/authentication/managing-security-context-constraints.html#default-sccs_configuring-internal-oauth
k get rolebinding system:openshift:scc:privileged-openebs -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"ffec0c8cb91220cd2484eb20ea0dffa03a63bb68"},"labels":{"kustomize.toolkit.fluxcd.io/name":"flux-system","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"system:openshift:scc:privileged-openebs","namespace":"openebs"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"system:openshift:scc:privileged"},"subjects":[{"kind":"ServiceAccount","name":"openebs","namespace":"openebs"},{"kind":"ServiceAccount","name":"openebs-lvm-node-sa","namespace":"openebs"}]}
kustomize.toolkit.fluxcd.io/checksum: ffec0c8cb91220cd2484eb20ea0dffa03a63bb68
creationTimestamp: "2020-12-19T13:04:22Z"
labels:
kustomize.toolkit.fluxcd.io/name: flux-system
kustomize.toolkit.fluxcd.io/namespace: flux-system
name: system:openshift:scc:privileged-openebs
namespace: openebs
resourceVersion: "1309101387"
uid: f1e32386-120c-40a7-b624-5b5188539d34
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:openshift:scc:privileged
subjects:
- kind: ServiceAccount
name: openebs
namespace: openebs
- kind: ServiceAccount
name: openebs-lvm-node-sa
namespace: openebs
This should be of interest:
docker run --rm -ti --entrypoint /bin/sh k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown.
and from the pod:
image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- rm -rf /registration/lvm-localpv /registration/lvm-localpv-reg.sock
name: csi-node-driver-registrar
i.e. the lifecycle-hook does not run!
@davidkarlsen -- the above information is really helpful! container permissions and location where the configuration is saved do sound like the possible causes for this issue.
Just spoke to @pawanpraka1 - and he wants to test out a few things. Will get back to this thread in a bit.
i.e. the lifecycle-hook does not run!
That is a good find, but should not cause the issue. There will be stale socket file present on the system because of that.
@davidkarlsen can you execute pvscan --cache (and lvscan) command before doing node reboot. I am wondering whether node cache is not updated before reboot, if not, we might lose the volumes.
@davidkarlsen can you execute
pvscan --cache(and lvscan) command before doing node reboot. I am wondering whether node cache is not updated before reboot, if not, we might lose the volumes.
It seems to not persist:
oco02 ~]# pvscan --cache
[root@alt-ksx-g-c01oco02 ~]# lvscan
ACTIVE '/dev/datavg/openebs' [5.00 GiB] inherit
ACTIVE '/dev/datavg/pvc-6e5a7f2f-fc50-443e-859d-d6f76de94064' [1.00 GiB] inherit
ACTIVE '/dev/rootvg/swap_lv' [2.00 GiB] inherit
ACTIVE '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
ACTIVE '/dev/rootvg/var_lv' [150.00 GiB] inherit
ACTIVE '/dev/rootvg/home_lv' [1000.00 MiB] inherit
ACTIVE '/dev/rootvg/root_lv' [<6.84 GiB] inherit
[root@alt-ksx-g-c01oco02 ~]# cd /etc/
[root@alt-ksx-g-c01oco02 etc]# cd lvm/
[root@alt-ksx-g-c01oco02 lvm]# find . -type f|xargs fgrep -il vc-6e5a7f2f-fc50-443e-859d-d6f76
[root@alt-ksx-g-c01oco02 lvm]#
Ok, so I've tried creating a bunch of volumes and rebooting machines. Two changes:
- I noticed the previous openebs install was not 100%, because in some upgrade the storageCapacity flag went from false to true, so the helm install was not complete, as this flag is immutable. I deleted the csidriver and let the chart install it again which then succeeded.
- I previously used XFS-based filesystems, these failed formatting by openebs due to https://github.com/openebs/lvm-localpv/issues/135 (which is really a kubernetes issue) - so these volumes were formatted by hand, now I use ext4 across the line.
I haven't seen the problem since, so I think we close this and I'll reopen if I can provoke it consistently.
We occur this issue too!
After a reboot, we found a lv disappear while other lvs are normal.
The lost lv coundn't be found in /dev/mapper or /dev/lvmvg. There isn't any clue in the openebs pods logs or the lvm services
And found one wired thing is that there's no record of pvc-070f0023-d09e-4041-9cb9-dbd1a579e717 in /etc/lvm/archive. But the lv is surely created successfully and the pod worked well before restart. pvscan --cache return no error
root@master1:/etc/lvm/archive# lvm version
LVM version: 2.02.176(2) (2017-11-03)
Library version: 1.02.145 (2017-11-03)
Driver version: 4.41.0
Configuration: ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync
Could anyone take a look at this serious issue? Thanks in advance!
@davidkarlsen Thanks! Linux version 5.4.61-050461-generic (kernel@kathleen) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #202008260931 SMP Wed Aug 26 09:34:29 UTC 2020
I am currently evaluating some local storage options and reading through issues here and elsewhere but thought I might chime in. This is sounding like a case of openebs-lvm-node not having an exclusive lock on the metadata of the volume group (ie lvm2-lvmmetad.service is interfering). I do not think it is a good idea having multiple services operating on the volume group at once. Could you try to run some tests with lvm2-lvmmetad.service stopped/disabled? It is not even technically required to be running for the pvcreate and vgcreate steps in the install instructions.
@gunzy83 Agree! There seems need to add an exclusive lock on createing and delete logic volumes @wrtabhi611 Could you please take a look at this ? When going through the source code, I found there are places operating pvs,vgs,lvs supposing atomic operation, which are not in fact. Thanks in advance!
@gunzy83 Agree! There seems need to add an exclusive lock on createing and delete logic volumes
I wonder if lvm2-lvmmetad.service operates with the same semantics (assuming it has exclusive use of the pvs/vgs/lvs)? If so adding logic into openebs would be the way to go. If not, providing some general guidance in the documentation that openebs-lvm-node needs exclusive use would be prudent.
Why would this be needed? Doesn't the service run normally, and users can run LVM commands without LVM getting corrupt?
Only a theory but users on the host would be running those commands from the same namespace I assume. Inside a container you would need to break namespace isolation for commands/services on the host to be able to coordinate with the commands being run from the container. Would be happy to be proven wrong.
For reference, Topolvm look to have solved the issue they had, they were running their lvmd on the host to avoid it until they implemented a solution using nsenter and can now run it in a daemonset. carina.io also use nsenter to escape isolation but I can't find any issues on Github to follow that show how they came to that.
@davidkarlsen @gunzy83 In fact, we have reproduced this issue more than once. Procedure
These 9 pvcs have to occupy the vg full
1. Execute mkdir -p /root/lvmtest/pvc to create 9 pvc yaml files, execute kubectl apply -f /root/lvmtest/pvc/ to create 9 pvcs, 9 pvcs will be the vg where lvm is located full
2. Execute mkdir -p /root/lvmtest/pod to create 9 pod yaml files, execute kubectl apply -f /root/lvmtest/pod/ to create 9 pods and mount the pvc created in the previous step respectively
3. Execute kubectl get pvc to see that all 9 pvcs are Bound
4. Execute lvs to see if there are 9 corresponding lvs
5. Execute kubectl delete -f /root/lvmtest/pod/, kubectl delete -f /root/lvmtest/pvc/, delete pod and pvc, and then execute kubectl get pvc to see if there is any pvc
6. Execute kubectl apply -f /root/lvmtest/pvc/ to create 9 pvcs
7. Execute kubectl apply -f /root/lvmtest/pod/ to create 9 pods and mount the pvc created in the previous step respectively
8.1. Execute kubectl delete lvmvolume -n kube-system pvc-xxx, delete the lv created in the first round in order from small to large
8.2. Execute lvremove, delete the lv created in the first round in order from small to large
9. Execute kubectl get pvc to check whether the 9 pvcs are Bound
10. Execute lvs to see if there are 9 corresponding lvs, execute ls /dev/mapper, ls /dev/lvmvg to see if there are 9 corresponding lvm directories
After runing above steps circularly ,there are chances to reproduct the issue. The lvs will display <9 LVs , and the lost LV will disappear forever after reboot.
@davidkarlsen Did you notice this locking_type in /etc/lvm/lvm.conf ? My test env didn't config this previously. I've configured this and retest now
