piraeus-operator
piraeus-operator copied to clipboard
Published
20 hours ago •
piraeusdatastore
piraeusdatastore
Pod no longer able to mount persistent volume
We are running a Strimzi Kafka cluster using Pireaus as the storage layer. Since a fews days (not sure how long), one of the kafka pods is no longer able to mount its persistent volume. The operator is now stuck in a loop recreating the pod every 10 min.
Events from the pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m30s default-scheduler Successfully assigned kafka/main-kafka-1 to eu-central-68d6
Warning FailedMount 4m28s kubelet Unable to attach or mount volumes: unmounted volumes=[data-0], unattached volumes=[cluster-ca broker-certs linkerd-identity-end-entity linkerd-identity-token client-ca-cert ready-files kube-api-access-d4vhg data-0 strimzi-tmp kafka-metrics-and-logging linkerd-proxy-init-xtables-lock]: timed out waiting for the condition
Warning FailedMount 2m10s kubelet Unable to attach or mount volumes: unmounted volumes=[data-0], unattached volumes=[kafka-metrics-and-logging linkerd-identity-end-entity data-0 linkerd-identity-token strimzi-tmp ready-files client-ca-cert kube-api-access-d4vhg cluster-ca broker-certs linkerd-proxy-init-xtables-lock]: timed out waiting for the condition
Warning FailedMount 17s (x11 over 6m29s) kubelet MountVolume.SetUp failed for volume "pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9" : rpc error: code = Internal desc = NodePublishVolume failed for pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: failed to set source device readwrite: exit status 1
# linstor volume list
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State |
|===========================================================================================================================================================|
| eu-central-1771 | pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | local-disk | 0 | 1008 | /dev/drbd1008 | 40.78 GiB | InUse | UpToDate |
| eu-central-42f5 | pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | local-disk | 0 | 1008 | /dev/drbd1008 | 40.78 GiB | Unused | UpToDate |
| eu-central-68d6 | pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | local-disk | 0 | 1008 | /dev/drbd1008 | 40.78 GiB | Unused | UpToDate |
# linstor-csi-node
time="2023-04-17T11:16:20Z" level=info msg="mounting volume" blockAccessMode=false filesystem=ext4 linstorCSIComponent=client mountOpts="[_netdev]" source=/dev/drbd1008 target="/var/lib/kubelet/pods/c45e681a-598e-47d2-8548-9dc33c84408b/volumes/kubernetes.io~csi/pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/mount"
time="2023-04-17T11:16:20Z" level=error msg="method failed" error="rpc error: code = Internal desc = NodePublishVolume failed for pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: failed to set source device readwrite: exit status 1" linstorCSIComponent=driver method=/csi.v1.Node/NodePublishVolume nodeID=eu-central-68d6 provisioner=linstor.csi.linbit.com req="volume_id:\"pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9\" publish_context:<key:\"Aux/csi-publish-readonly\" value:\"false\" > target_path:\"/var/lib/kubelet/pods/c45e681a-598e-47d2-8548-9dc33c84408b/volumes/kubernetes.io~csi/pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/mount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_context:<key:\"csi.storage.k8s.io/ephemeral\" value:\"false\" > volume_context:<key:\"csi.storage.k8s.io/pod.name\" value:\"main-kafka-1\" > volume_context:<key:\"csi.storage.k8s.io/pod.namespace\" value:\"kafka\" > volume_context:<key:\"csi.storage.k8s.io/pod.uid\" value:\"c45e681a-598e-47d2-8548-9dc33c84408b\" > volume_context:<key:\"csi.storage.k8s.io/serviceAccount.name\" value:\"main-kafka\" > volume_context:<key:\"linstor.csi.linbit.com/mount-options\" value:\"\" > volume_context:<key:\"linstor.csi.linbit.com/post-mount-xfs-opts\" value:\"\" > volume_context:<key:\"linstor.csi.linbit.com/remote-access-policy\" value:\"true\" > volume_context:<key:\"linstor.csi.linbit.com/uses-volume-context\" value:\"true\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1679407629478-8081-linstor.csi.linbit.com\" > " resp="<nil>" version=v1.0.0-3fbda5a7e31517786796ae4834c159caa9ad8ac7
Not sure how to move forward here. I've found a similar error message in https://github.com/piraeusdatastore/piraeus-operator/issues/424, but don't believe it's the same issue.
I've generated an SOS report in the current state: sos-report.tar.gz
Something has kept the resource open on eu-central-1771, as you can see in the linstor volume list. This is not allowed by DRBD, as only one node can access the device at any point in time. I can't find anything related to pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 in the kernel logs, so it probably got promoted some time ago.
You may try kubectl exec eu-central-1771 -- drbdadm secondary pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9, which should produce an error message, telling you what kind of process kept the device open. I would guess it's some kind of mount process. You can search for where the device remains mounted using mount | grep drbd1008
Thanks for the quick reply @WanzenBug! I ran the command you suggested. It returned no errors and caused the volume on node eu-central-68d6 to be promoted to primary automatically. See:
~ linstor r l -r pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9
+---------------------------------------------------------------------------------------------------------------------+
| ResourceName | Node | Port | Usage | Conns | State | CreatedOn |
|=====================================================================================================================|
| pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | eu-central-1771 | 7008 | Unused | Ok | UpToDate | 2023-03-29 18:00:35 |
| pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | eu-central-42f5 | 7008 | Unused | Ok | UpToDate | 2023-03-29 18:00:33 |
| pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 | eu-central-68d6 | 7008 | InUse | Ok | UpToDate | 2023-03-29 18:00:35 |
+---------------------------------------------------------------------------------------------------------------------+
If this should ever happen again I now know how to solve it, but this should have been resolved by the controller right? Are there any other logs which I could check to see what might have caused this?
That kubectl exec eu-central-1771 -- drbdadm secondary pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 suceeded would suggest that something or someone ran drbdadm primary pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9, which disables automatic demotion on removing the last writer.
So I suspect some manual inspection took place. Only other possibilty I could think of is LINSTOR itself timed out during resource creation, but somehow the resource was still usable.
I'd take a look at the kernel logs for 1771, grep for pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9. Normally there is some log message indicating on why the resource was promoted to Primary.
I ran initially ran the kubectl exec eu-central-1771 -- drbdadm secondary pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 command twice as I thought I made mistake the first time, so that might show up in the logs.
Here are the results for that pvc. All these logs are from the moment I ran the command it seems.
root@eu-central-1771:/# cat /var/log/kern.log | grep pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9
Apr 17 11:42:48 eu-central-1771 kernel: [1720020.219727] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 853234971 (1->-1 3/2)
Apr 17 11:42:48 eu-central-1771 kernel: [1720020.244095] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 853234971: primary_nodes=0, weak_nodes=0
Apr 17 11:42:48 eu-central-1771 kernel: [1720020.244103] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 853234971 (24ms)
Apr 17 11:42:48 eu-central-1771 kernel: [1720020.244263] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: role( Primary -> Secondary )
Apr 17 11:44:09 eu-central-1771 kernel: [1720101.132850] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 115516091
Apr 17 11:44:09 eu-central-1771 kernel: [1720101.160600] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 115516091 (primary_nodes=4)
Apr 17 11:44:09 eu-central-1771 kernel: [1720101.160616] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: peer( Secondary -> Primary )
Apr 17 11:44:11 eu-central-1771 kernel: [1720103.045400] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 2489766596
Apr 17 11:44:11 eu-central-1771 kernel: [1720103.069434] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 2489766596 (primary_nodes=0)
Apr 17 11:44:11 eu-central-1771 kernel: [1720103.069491] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: peer( Primary -> Secondary )
Apr 17 11:44:12 eu-central-1771 kernel: [1720104.623706] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 1849616096
Apr 17 11:44:12 eu-central-1771 kernel: [1720104.649163] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 1849616096 (primary_nodes=4)
Apr 17 11:44:12 eu-central-1771 kernel: [1720104.649183] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: peer( Secondary -> Primary )
You might want to check older logs (kern.log.1, kern.log.2, etc..) to find the original role( Secondary -> Primary ) message.
Ah you're right. Found the relevant logs in kern.log.3
Mar 29 18:00:35 eu-central-1771 kernel: [101113.161400] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Starting worker thread (from drbdsetup [593999])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.163721] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbdsetup [594005])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.165024] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Starting sender thread (from drbdsetup [594008])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.243859] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: meta-data IO uses: blk-bio
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244043] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: susp-io( no -> quorum )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244046] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Diskless -> Attaching )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244057] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: Maximum number of peer devices = 7
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244312] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Method to ensure write ordering: barrier
Mar 29 18:00:35 eu-central-1771 kernel: [101113.244321] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: drbd_bm_resize called with capacity == 104857600
Mar 29 18:00:35 eu-central-1771 kernel: [101113.253005] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: resync bitmap: bits=13107200 words=1433600 pages=2800
Mar 29 18:00:35 eu-central-1771 kernel: [101113.253016] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: size = 50 GB (52428800 KB)
Mar 29 18:00:35 eu-central-1771 kernel: [101113.269051] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: recounting of set bits took additional 8ms
Mar 29 18:00:35 eu-central-1771 kernel: [101113.269072] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Attaching -> UpToDate )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.269074] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: attached to current UUID: C8594E29160A6996
Mar 29 18:00:35 eu-central-1771 kernel: [101113.269167] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: size = 50 GB (52428800 KB)
Mar 29 18:00:35 eu-central-1771 kernel: [101113.272023] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( StandAlone -> Unconnected )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.272706] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: conn( StandAlone -> Unconnected )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.272962] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting receiver thread (from drbd_w_pvc-22af [594000])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.273095] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Starting receiver thread (from drbd_w_pvc-22af [594000])
Mar 29 18:00:35 eu-central-1771 kernel: [101113.273104] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.273210] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: conn( Unconnected -> Connecting )
Mar 29 18:00:35 eu-central-1771 kernel: [101113.873810] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 29 18:00:35 eu-central-1771 kernel: [101113.873816] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 29 18:00:35 eu-central-1771 kernel: [101113.897579] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 29 18:00:35 eu-central-1771 kernel: [101113.936240] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Handshake to peer 2 successful: Agreed network protocol version 121
Mar 29 18:00:35 eu-central-1771 kernel: [101113.936246] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 29 18:00:35 eu-central-1771 kernel: [101113.936785] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Peer authenticated using 20 bytes HMAC
Mar 29 18:00:35 eu-central-1771 kernel: [101114.051797] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 2138728852 (1->2 499/146)
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075808] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: drbd_sync_handshake:
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075810] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: self C8594E29160A6996:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075813] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: peer C8594E29160A6996:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075816] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: uuid_compare()=no-sync by rule=both-off
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075827] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 2138728852: primary_nodes=0, weak_nodes=0
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075829] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 2138728852 (24ms)
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075871] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: susp-io( quorum -> no )
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075873] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075874] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: quorum( no -> yes )
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075876] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( DUnknown -> UpToDate ) repl( Off -> Established )
Mar 29 18:00:35 eu-central-1771 kernel: [101114.075977] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: cleared bm UUID and bitmap C8594E29160A6996:0000000000000000:0000000000000000:0000000000000000
Mar 29 18:00:36 eu-central-1771 kernel: [101114.101733] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 1179600745
Mar 29 18:00:36 eu-central-1771 kernel: [101114.115820] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 29 18:00:36 eu-central-1771 kernel: [101114.115825] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self C8594E29160A6996:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Mar 29 18:00:36 eu-central-1771 kernel: [101114.115829] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:C8594E29160A6996:0000000000000000:0000000000000000 bits:66716 flags:1020
Mar 29 18:00:36 eu-central-1771 kernel: [101114.115833] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=target-use-bitmap by rule=bitmap-peer
Mar 29 18:00:36 eu-central-1771 kernel: [101114.154248] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 1179600745 (primary_nodes=0)
Mar 29 18:00:36 eu-central-1771 kernel: [101114.154270] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.154273] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( UpToDate -> Outdated )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.154275] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.157310] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 183(1), total 183; compression: 100.0%
Mar 29 18:00:36 eu-central-1771 kernel: [101114.160904] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 183(1), total 183; compression: 100.0%
Mar 29 18:00:36 eu-central-1771 kernel: [101114.160928] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Outdated -> Inconsistent )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.160932] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: resync-susp( no -> connection dependency )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.160935] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: repl( WFBitMapT -> SyncTarget )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.161158] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Began resync as SyncTarget (will sync 266864 KB [66716 bits set]).
Mar 29 18:00:36 eu-central-1771 kernel: [101114.169779] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 655708152
Mar 29 18:00:36 eu-central-1771 kernel: [101114.213902] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 655708152 (primary_nodes=0)
Mar 29 18:00:36 eu-central-1771 kernel: [101114.214456] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( UpToDate -> Outdated )
Mar 29 18:00:36 eu-central-1771 kernel: [101114.220718] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Outdated -> Inconsistent )
Mar 29 18:00:38 eu-central-1771 kernel: [101116.629729] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Inconsistent -> UpToDate )
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735489] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Resync done (total 2 sec; paused 0 sec; 133432 K/sec)
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735500] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: updated UUIDs 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735510] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Inconsistent -> UpToDate )
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735512] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: resync-susp( connection dependency -> no )
Mar 29 18:00:38 eu-central-1771 kernel: [101116.735514] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: repl( SyncTarget -> Established )
Mar 29 18:00:40 eu-central-1771 kernel: [101118.974835] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 475334800
Mar 29 18:00:40 eu-central-1771 kernel: [101118.999881] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 475334800 (primary_nodes=1)
Mar 29 18:00:40 eu-central-1771 kernel: [101118.999898] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: peer( Secondary -> Primary )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.601953] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: meta connection shut down by peer.
Mar 31 20:11:19 eu-central-1771 kernel: [281754.601963] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.601965] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( UpToDate -> Consistent )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.601967] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.602000] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 tcp:eu-central-42f5: Not enough data received; missing 3716 bytes
Mar 31 20:11:19 eu-central-1771 kernel: [281754.602007] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: error receiving P_DATA, e: -5 l: 4096!
Mar 31 20:11:19 eu-central-1771 kernel: [281754.605439] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:19 eu-central-1771 kernel: [281754.605465] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653704] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653711] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( NetworkFailure -> Unconnected )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653717] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653721] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:19 eu-central-1771 kernel: [281754.653829] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 364178062 (1->-1 0/0)
Mar 31 20:11:20 eu-central-1771 kernel: [281755.158765] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( UpToDate -> Consistent )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.218375] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 364178062: primary_nodes=0, weak_nodes=0
Mar 31 20:11:20 eu-central-1771 kernel: [281755.218379] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 364178062 (564ms)
Mar 31 20:11:20 eu-central-1771 kernel: [281755.218387] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Consistent -> UpToDate )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.219530] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 55078831
Mar 31 20:11:20 eu-central-1771 kernel: [281755.219551] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Aborting remote state change 55078831
Mar 31 20:11:20 eu-central-1771 kernel: [281755.220102] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: repl( Established -> WFBitMapS )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.220110] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Reconciliation resync because 'eu-central-42f5' disappeared. (o=32)
Mar 31 20:11:20 eu-central-1771 kernel: [281755.220116] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Consistent -> Outdated )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.223345] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 53(1), total 53; compression: 100.0%
Mar 31 20:11:20 eu-central-1771 kernel: [281755.226284] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 53(1), total 53; compression: 100.0%
Mar 31 20:11:20 eu-central-1771 kernel: [281755.226297] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Outdated -> Inconsistent ) repl( WFBitMapS -> SyncSource )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.226499] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Began resync as SyncSource (will sync 1856 KB [464 bits set]).
Mar 31 20:11:20 eu-central-1771 kernel: [281755.316606] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Resync done (total 1 sec; paused 0 sec; 1856 K/sec)
Mar 31 20:11:20 eu-central-1771 kernel: [281755.316620] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established )
Mar 31 20:11:20 eu-central-1771 kernel: [281755.338686] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 234327999
Mar 31 20:11:20 eu-central-1771 kernel: [281755.340513] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 234327999 (primary_nodes=0)
Mar 31 20:11:25 eu-central-1771 kernel: [281760.150346] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 2293158436
Mar 31 20:11:25 eu-central-1771 kernel: [281760.168279] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: sock_recvmsg returned -11
Mar 31 20:11:25 eu-central-1771 kernel: [281760.168292] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> BrokenPipe )
Mar 31 20:11:25 eu-central-1771 kernel: [281760.168304] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: short read (expected size 8)
Mar 31 20:11:25 eu-central-1771 kernel: [281760.168306] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: timeout while waiting for feature packet
Mar 31 20:11:25 eu-central-1771 kernel: [281760.171705] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:25 eu-central-1771 kernel: [281760.171735] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:25 eu-central-1771 kernel: [281760.221661] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:25 eu-central-1771 kernel: [281760.221675] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( BrokenPipe -> Unconnected )
Mar 31 20:11:26 eu-central-1771 kernel: [281761.255906] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:26 eu-central-1771 kernel: [281761.852592] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 tcp:eu-central-42f5: initial packet M crossed
Mar 31 20:11:31 eu-central-1771 kernel: [281766.467899] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:31 eu-central-1771 kernel: [281766.467903] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:31 eu-central-1771 kernel: [281766.693437] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:31 eu-central-1771 kernel: [281766.693544] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> NetworkFailure )
Mar 31 20:11:31 eu-central-1771 kernel: [281766.693608] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:31 eu-central-1771 kernel: [281766.693630] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:31 eu-central-1771 kernel: [281766.737349] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:31 eu-central-1771 kernel: [281766.737360] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( NetworkFailure -> Unconnected )
Mar 31 20:11:31 eu-central-1771 kernel: [281766.737366] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:31 eu-central-1771 kernel: [281766.737371] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:35 eu-central-1771 kernel: [281770.453773] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:35 eu-central-1771 kernel: [281770.453776] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:36 eu-central-1771 kernel: [281771.851035] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:36 eu-central-1771 kernel: [281771.973852] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Rejecting concurrent remote state change 2027767180 because of state change 2293158436
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987815] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987823] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987828] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:482 flags:1520
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987832] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Initial UUIDs and state not sent yet. Not verifying
Mar 31 20:11:36 eu-central-1771 kernel: [281771.987834] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:11:39 eu-central-1771 kernel: [281774.249929] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 2908187164 because of state change 2293158436
Mar 31 20:11:39 eu-central-1771 kernel: [281774.809699] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Rejecting concurrent remote state change 2622719287 because of state change 2293158436
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831729] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831734] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831738] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:482 flags:1520
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831741] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Initial UUIDs and state not sent yet. Not verifying
Mar 31 20:11:39 eu-central-1771 kernel: [281774.831743] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:11:40 eu-central-1771 kernel: [281775.361450] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: sock was shut down by peer
Mar 31 20:11:40 eu-central-1771 kernel: [281775.361465] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> BrokenPipe )
Mar 31 20:11:40 eu-central-1771 kernel: [281775.361561] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:40 eu-central-1771 kernel: [281775.361602] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:40 eu-central-1771 kernel: [281775.412825] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:40 eu-central-1771 kernel: [281775.412833] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( BrokenPipe -> Unconnected )
Mar 31 20:11:40 eu-central-1771 kernel: [281775.412839] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:40 eu-central-1771 kernel: [281775.412843] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:43 eu-central-1771 kernel: [281778.339773] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:43 eu-central-1771 kernel: [281778.342297] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:44 eu-central-1771 kernel: [281779.946455] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:46 eu-central-1771 kernel: [281781.541579] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Rejecting concurrent remote state change 3516829557 because of state change 2293158436
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563614] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563617] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563622] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:482 flags:1520
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563626] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Initial UUIDs and state not sent yet. Not verifying
Mar 31 20:11:46 eu-central-1771 kernel: [281781.563628] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:11:48 eu-central-1771 kernel: [281783.797991] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3330022623 because of state change 2293158436
Mar 31 20:11:49 eu-central-1771 kernel: [281784.273554] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Rejecting concurrent remote state change 3367399829 because of state change 2293158436
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287591] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287597] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287600] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:482 flags:1520
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287603] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Initial UUIDs and state not sent yet. Not verifying
Mar 31 20:11:49 eu-central-1771 kernel: [281784.287605] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:11:49 eu-central-1771 kernel: [281784.805598] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: sock was shut down by peer
Mar 31 20:11:49 eu-central-1771 kernel: [281784.805612] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> BrokenPipe )
Mar 31 20:11:49 eu-central-1771 kernel: [281784.805690] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:49 eu-central-1771 kernel: [281784.805708] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:49 eu-central-1771 kernel: [281784.806150] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 1677594074 because of state change 2293158436
Mar 31 20:11:49 eu-central-1771 kernel: [281784.840895] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:49 eu-central-1771 kernel: [281784.840907] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( BrokenPipe -> Unconnected )
Mar 31 20:11:49 eu-central-1771 kernel: [281784.840917] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:49 eu-central-1771 kernel: [281784.840924] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:50 eu-central-1771 kernel: [281785.314158] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3281136363 because of state change 2293158436
Mar 31 20:11:50 eu-central-1771 kernel: [281785.614903] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3396001944 because of state change 2293158436
Mar 31 20:11:50 eu-central-1771 kernel: [281785.897856] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3100672356 because of state change 2293158436
Mar 31 20:11:54 eu-central-1771 kernel: [281789.251437] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: sock_recvmsg returned -11
Mar 31 20:11:54 eu-central-1771 kernel: [281789.251453] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> BrokenPipe )
Mar 31 20:11:54 eu-central-1771 kernel: [281789.251465] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: short read (expected size 8)
Mar 31 20:11:54 eu-central-1771 kernel: [281789.251467] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: timeout while waiting for feature packet
Mar 31 20:11:54 eu-central-1771 kernel: [281789.254285] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:54 eu-central-1771 kernel: [281789.254313] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:54 eu-central-1771 kernel: [281789.288483] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:54 eu-central-1771 kernel: [281789.288493] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( BrokenPipe -> Unconnected )
Mar 31 20:11:55 eu-central-1771 kernel: [281790.307428] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:55 eu-central-1771 kernel: [281790.781696] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 1128590826 because of state change 2293158436
Mar 31 20:11:55 eu-central-1771 kernel: [281790.881570] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 547725012 because of state change 2293158436
Mar 31 20:11:55 eu-central-1771 kernel: [281790.981877] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 4294843850 because of state change 2293158436
Mar 31 20:11:55 eu-central-1771 kernel: [281791.075624] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 2939194751 because of state change 2293158436
Mar 31 20:11:56 eu-central-1771 kernel: [281791.143391] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Two-phase commit 2293158436 timeout
Mar 31 20:11:56 eu-central-1771 kernel: [281791.185651] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 1137476348
Mar 31 20:11:56 eu-central-1771 kernel: [281791.225790] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 1137476348 (primary_nodes=1)
Mar 31 20:11:56 eu-central-1771 kernel: [281791.225805] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( UpToDate -> Outdated )
Mar 31 20:11:56 eu-central-1771 kernel: [281791.541222] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:56 eu-central-1771 kernel: [281791.541227] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:56 eu-central-1771 kernel: [281791.564915] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:56 eu-central-1771 kernel: [281791.565053] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> NetworkFailure )
Mar 31 20:11:56 eu-central-1771 kernel: [281791.565140] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:11:56 eu-central-1771 kernel: [281791.565160] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:11:56 eu-central-1771 kernel: [281791.608948] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:11:56 eu-central-1771 kernel: [281791.608961] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( NetworkFailure -> Unconnected )
Mar 31 20:11:56 eu-central-1771 kernel: [281791.608971] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:11:56 eu-central-1771 kernel: [281791.608975] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:11:57 eu-central-1771 kernel: [281792.249264] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:11:57 eu-central-1771 kernel: [281792.249270] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:11:57 eu-central-1771 kernel: [281792.273305] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:11:57 eu-central-1771 kernel: [281792.306605] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 2678655836 (1->-1 3/1)
Mar 31 20:11:57 eu-central-1771 kernel: [281792.330744] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Aborting cluster-wide state change 2678655836 (24ms) rv = -10
Mar 31 20:11:57 eu-central-1771 kernel: [281792.330801] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Declined by peer eu-central-68d6 (id: 2), see the kernel log there
Mar 31 20:11:57 eu-central-1771 kernel: [281792.355011] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 3878372240
Mar 31 20:12:28 eu-central-1771 kernel: [281823.669385] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3169814138 because of state change 3878372240
Mar 31 20:13:01 eu-central-1771 kernel: [281856.404878] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 3856063766 because of state change 3878372240
Mar 31 20:13:34 eu-central-1771 kernel: [281889.172273] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 4201528038 because of state change 3878372240
Mar 31 20:14:06 eu-central-1771 kernel: [281921.939661] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 302133202 because of state change 3878372240
Mar 31 20:14:39 eu-central-1771 kernel: [281954.707140] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Rejecting concurrent remote state change 4284797115 because of state change 3878372240
Mar 31 20:33:57 eu-central-1771 kernel: [283112.643635] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> NetworkFailure )
Mar 31 20:33:57 eu-central-1771 kernel: [283112.643781] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change failed: In transient state, retry after next state change
Mar 31 20:33:57 eu-central-1771 kernel: [283112.649571] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Two-phase commit 3878372240 timeout
Mar 31 20:33:57 eu-central-1771 kernel: [283112.652532] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Terminating sender thread
Mar 31 20:33:57 eu-central-1771 kernel: [283112.657138] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Starting sender thread (from drbd_r_pvc-22af [594026])
Mar 31 20:33:57 eu-central-1771 kernel: [283112.697768] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 2828581286 (1->-1 3/1)
Mar 31 20:33:57 eu-central-1771 kernel: [283112.698884] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Connection closed
Mar 31 20:33:57 eu-central-1771 kernel: [283112.698890] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( NetworkFailure -> Unconnected )
Mar 31 20:33:57 eu-central-1771 kernel: [283112.698904] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Restarting receiver thread
Mar 31 20:33:57 eu-central-1771 kernel: [283112.698907] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Unconnected -> Connecting )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.217978] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Aborting cluster-wide state change 2828581286 (520ms) rv = -19
Mar 31 20:33:58 eu-central-1771 kernel: [283113.218194] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( UpToDate -> Consistent )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219488] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: drbd_sync_handshake:
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219491] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: self 39E73F8692AA2282:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:20
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219494] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: peer 9ECC182ED68E217C:39E73F8692AA2282:0000000000000000:0000000000000000 bits:322 flags:1060
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219497] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: uuid_compare()=target-use-bitmap by rule=bitmap-peer
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219499] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Becoming WFBitMapT after unstable
Mar 31 20:33:58 eu-central-1771 kernel: [283113.219505] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: repl( Established -> WFBitMapT )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.283315] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 2500614562
Mar 31 20:33:58 eu-central-1771 kernel: [283113.283755] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 2500614562 (primary_nodes=0)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.284542] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: pdsk( Consistent -> UpToDate )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.287085] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 102(1), total 102; compression: 100.0%
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289502] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 102(1), total 102; compression: 100.0%
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289520] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Outdated -> Inconsistent )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289522] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: repl( WFBitMapT -> SyncTarget )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289524] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: resync-susp( no -> connection dependency )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.289801] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Began resync as SyncTarget (will sync 1288 KB [322 bits set]).
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327865] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: Resync done (total 1 sec; paused 0 sec; 1288 K/sec)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327874] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: updated UUIDs 9ECC182ED68E217C:0000000000000000:0000000000000000:0000000000000000
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327884] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: disk( Inconsistent -> UpToDate )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327886] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-68d6: repl( SyncTarget -> Established )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.327888] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: resync-susp( connection dependency -> no )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.426064] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 205524943 (1->-1 3/1)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.426621] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 205524943: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF9
Mar 31 20:33:58 eu-central-1771 kernel: [283113.426624] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 205524943 (0ms)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.426629] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: role( Secondary -> Primary )
Mar 31 20:33:58 eu-central-1771 kernel: [283113.428703] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: new current UUID: 02F7C57DF6F0892D weak: FFFFFFFFFFFFFFF9
Mar 31 20:33:58 eu-central-1771 kernel: [283113.434186] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 3382725511 (1->-1 3/2)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.434618] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 3382725511: primary_nodes=0, weak_nodes=0
Mar 31 20:33:58 eu-central-1771 kernel: [283113.434625] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 3382725511 (0ms)
Mar 31 20:33:58 eu-central-1771 kernel: [283113.434670] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: role( Primary -> Secondary )
Mar 31 20:33:59 eu-central-1771 kernel: [283114.413647] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Handshake to peer 0 successful: Agreed network protocol version 121
Mar 31 20:33:59 eu-central-1771 kernel: [283114.417354] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Feature flags enabled on protocol level: 0x3f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Mar 31 20:34:00 eu-central-1771 kernel: [283115.627073] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Peer authenticated using 20 bytes HMAC
Mar 31 20:34:00 eu-central-1771 kernel: [283115.803468] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 4290203759
Mar 31 20:34:00 eu-central-1771 kernel: [283115.837602] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:34:00 eu-central-1771 kernel: [283115.837608] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 02F7C57DF6F0892C:9ECC182ED68E217C:0000000000000000:0000000000000000 bits:322 flags:120
Mar 31 20:34:00 eu-central-1771 kernel: [283115.837612] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 9ECC182ED68E217D:39E73F8692AA2282:0000000000000000:0000000000000000 bits:803 flags:1520
Mar 31 20:34:00 eu-central-1771 kernel: [283115.837615] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=source-use-bitmap by rule=bitmap-self
Mar 31 20:34:02 eu-central-1771 kernel: [283117.645818] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Aborting remote state change 4290203759
Mar 31 20:34:02 eu-central-1771 kernel: [283117.671936] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 3016942592
Mar 31 20:34:02 eu-central-1771 kernel: [283118.053903] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 3016942592 (primary_nodes=0)
Mar 31 20:34:03 eu-central-1771 kernel: [283118.655561] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 3964007541
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669535] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669538] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 02F7C57DF6F0892C:9ECC182ED68E217C:0000000000000000:0000000000000000 bits:322 flags:120
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669543] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 02F7C57DF6F0892C:39E73F8692AA2282:0000000000000000:0000000000000000 bits:803 flags:1924
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669546] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=no-sync by rule=reconnected
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669548] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: strategy = source-set-bitmap due to disk states. (UpToDate/Inconsistent)
Mar 31 20:34:03 eu-central-1771 kernel: [283118.669551] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Setting and writing one bitmap slot, after drbd_sync_handshake
Mar 31 20:34:03 eu-central-1771 kernel: [283118.687235] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008: bitmap WRITE of 2800 pages took 16 ms
Mar 31 20:34:04 eu-central-1771 kernel: [283119.167606] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Aborting remote state change 3964007541
Mar 31 20:34:04 eu-central-1771 kernel: [283119.203553] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Preparing remote state change 750722015
Mar 31 20:34:04 eu-central-1771 kernel: [283119.204220] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-68d6: Committing remote state change 750722015 (primary_nodes=0)
Mar 31 20:34:04 eu-central-1771 kernel: [283119.219501] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 3957309307
Mar 31 20:34:04 eu-central-1771 kernel: [283119.233562] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: drbd_sync_handshake:
Mar 31 20:34:04 eu-central-1771 kernel: [283119.233568] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: self 02F7C57DF6F0892C:9ECC182ED68E217C:0000000000000000:0000000000000000 bits:13107200 flags:120
Mar 31 20:34:04 eu-central-1771 kernel: [283119.233573] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: peer 9ECC182ED68E217D:39E73F8692AA2282:0000000000000000:0000000000000000 bits:13107200 flags:1124
Mar 31 20:34:04 eu-central-1771 kernel: [283119.233577] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: uuid_compare()=source-use-bitmap by rule=bitmap-self
Mar 31 20:34:04 eu-central-1771 kernel: [283119.437449] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 3957309307 (primary_nodes=0)
Mar 31 20:34:04 eu-central-1771 kernel: [283119.437463] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Mar 31 20:34:04 eu-central-1771 kernel: [283119.437466] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: pdsk( DUnknown -> Inconsistent ) repl( Off -> WFBitMapS )
Mar 31 20:34:04 eu-central-1771 kernel: [283119.442364] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
Mar 31 20:34:05 eu-central-1771 kernel: [283120.294671] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
Mar 31 20:34:05 eu-central-1771 kernel: [283120.294691] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: repl( WFBitMapS -> SyncSource )
Mar 31 20:34:05 eu-central-1771 kernel: [283120.294963] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Began resync as SyncSource (will sync 52428800 KB [13107200 bits set]).
Mar 31 20:34:10 eu-central-1771 kernel: [283125.405033] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Preparing remote state change 2838873287
Mar 31 20:34:10 eu-central-1771 kernel: [283125.856391] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9 eu-central-42f5: Committing remote state change 2838873287 (primary_nodes=0)
Mar 31 20:34:40 eu-central-1771 kernel: [283155.557219] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Preparing cluster-wide state change 4115924573 (1->-1 3/1)
Mar 31 20:34:48 eu-central-1771 kernel: [283163.286904] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: State change 4115924573: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF8
Mar 31 20:34:48 eu-central-1771 kernel: [283163.286911] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: Committing cluster-wide state change 4115924573 (7728ms)
Mar 31 20:34:48 eu-central-1771 kernel: [283163.286921] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9: role( Secondary -> Primary )
Mar 31 20:49:23 eu-central-1771 kernel: [284039.044227] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: updated UUIDs 02F7C57DF6F0892C:0000000000000000:9ECC182ED68E217C:0000000000000000
Mar 31 20:49:24 eu-central-1771 kernel: [284039.076002] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: Resync done (total 918 sec; paused 0 sec; 57108 K/sec)
Mar 31 20:49:24 eu-central-1771 kernel: [284039.076016] drbd pvc-22af8af6-3ecb-4216-b4d3-f1b5410266c9/0 drbd1008 eu-central-42f5: pdsk( Inconsistent -> UpToDate ) repl( SyncSource -> Established )