etcd

etcd copied to clipboard

2022-01-12 08:11:44.420767 W | etcdserver: read-only range request "key:\"/registry/pods/mss-backend-id-staging/mis-server-bbd655577-ggrrz\" " with result "range_response_count:1 size:1925" took too long (1.580800472s) to execute
2022-01-12 08:11:44.420876 W | etcdserver: read-only range request "key:\"/registry/pods/apc-dp-gateway-id-live/transparent-gateway-5f547759b5-gbsgk\" " with result "range_response_count:1 size:2122" took too long (6.618746881s) to execute
2022-01-12 08:11:44.421002 W | etcdserver: read-only range request "key:\"/registry/events/apc-message-id-uat/\" range_end:\"/registry/events/apc-message-id-uat0\" " with result "range_response_count:45 size:25601" took too long (4.480516042s) to execute
2022-01-12 08:11:44.421112 W | etcdserver: read-only range request "key:\"/registry/namespaces/\" range_end:\"/registry/namespaces0\" " with result "range_response_count:68 size:21584" took too long (1.70726542s) to execute
2022-01-12 08:11:44.421265 W | etcdserver: read-only range request "key:\"/registry/namespaces/\" range_end:\"/registry/namespaces0\" " with result "range_response_count:68 size:21584" took too long (1.429287931s) to execute
2022-01-12 08:12:45.084454 I | mvcc: store.index: compact 267837780
2022-01-12 08:12:46.769358 I | mvcc: finished scheduled compaction at 267837780 (took 1.589393042s)
2022-01-12 08:17:45.186465 I | mvcc: store.index: compact 267841308
2022-01-12 08:17:45.286417 W | etcdserver: request "header:<ID:14966999623285420954 username:\"kube-apiserver-etcd-client\" auth_revision:1 > compaction:<revision:267841308 > " with result "size:7" took too long (100.184669ms) to execute
2022-01-12 08:17:46.867400 I | mvcc: finished scheduled compaction at 267841308 (took 1.580968039s)
2022-01-12 08:20:12.571244 I | raft: f852013d0a603fa4 [logterm: 10967, index: 335494111, vote: f852013d0a603fa4] ignored MsgVote from d1460a24dccdab22 [logterm: 10967, index: 335494109] at term 10967: lease is not expired (remaining ticks: 2)
2022-01-12 08:20:12.573443 I | raft: f852013d0a603fa4 [term: 10967] received a MsgApp message with higher term from d1460a24dccdab22 [term: 10968]
2022-01-12 08:20:12.573458 I | raft: f852013d0a603fa4 became follower at term 10968
2022-01-12 08:20:12.573465 I | raft: found conflict at index 335494110 [existing term: 10967, conflicting term: 10968]
2022-01-12 08:20:12.573486 I | raft: truncate the unstable entries before index 335494110
2022-01-12 08:20:12.573495 I | raft: raft.node: f852013d0a603fa4 changed leader from f852013d0a603fa4 to d1460a24dccdab22 at term 10968
2022-01-12 08:20:14.169194 W | wal: sync duration of 2.88067165s, expected less than 1s
2022-01-12 08:20:14.171076 W | etcdserver: read-only range request "key:\"/registry/pods/foms-backend-id-uat/unbind-shop-project-cancel-tickets-1641975600-p8kst\" " with result "error:etcdserver: leader changed" took too long (2.827869503s) to execute
2022-01-12 08:20:14.187493 W | etcdserver: read-only range request "key:\"/registry/pods/apc-commission-id-live/apc-commission-job-6585d8d8f-qddqq\" " with result "range_response_count:1 size:1921" took too long (1.458233594s) to execute
2022-01-12 08:20:14.187663 W | etcdserver: read-only range request "key:\"/registry/pods/apc-account-id-uat/apc-pay-autofix-server-release-7cdc54bfc5-cqt7k\" " with result "range_response_count:1 size:2097" took too long (1.92348302s) to execute
2022-01-12 08:20:14.187685 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-live/config-center-74c8c7c5d-cx67h\" " with result "range_response_count:1 size:1979" took too long (2.137652323s) to execute
2022-01-12 08:20:14.187696 W | etcdserver: read-only range request "key:\"/registry/pods/apc-dp-id-uat/order-server-5bd99dd988-tv9c8\" " with result "range_response_count:1 size:1996" took too long (1.152485845s) to execute
2022-01-12 08:20:14.187735 W | etcdserver: read-only range request "key:\"/registry/crd.projectcalico.org/clusterinformations/default\" " with result "range_response_count:1 size:565" took too long (1.529138491s) to execute
2022-01-12 08:20:14.187749 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-live/image-server-c7db64756-vkrlv\" " with result "range_response_count:1 size:1966" took too long (2.17057573s) to execute
2022-01-12 08:20:14.187798 W | etcdserver: read-only range request "key:\"/registry/pods/apc-base-id-live/config-gateway-5585cc9f98-85ncw\" " with result "range_response_count:1 size:1949" took too long (1.810814158s) to execute
2022-01-12 08:20:14.187828 W | etcdserver: read-only range request "key:\"/registry/pods/foms-backend-id-uat/merchant-foms-marketing-approval-6674458756-l22dr\" " with result "range_response_count:1 size:2333" took too long (1.804069654s) to execute
2022-01-12 08:20:14.187851 W | etcdserver: read-only range request "key:\"/registry/pods/apc-frontend-service-id-uat/fe-config-client-service-dd94fd67-w9d7f\" " with result "range_response_count:1 size:2691" took too long (1.300726524s) to execute
2022-01-12 08:20:14.187862 W | etcdserver: read-only range request "key:\"/registry/pods/mss-backend-id-staging/mis-server-bbd655577-ggrrz\" " with result "range_response_count:1 size:1925" took too long (1.369661047s) to execute
2022-01-12 08:20:14.187871 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-live/apc-push-server-8f7d6fc77-nj52t\" " with result "range_response_count:1 size:2006" took too long (1.352170711s) to execute
2022-01-12 08:20:14.187890 W | etcdserver: read-only range request "key:\"/registry/pods/mss-frontend-gateway-id-live/api-partner-admin-gw-d555b679f-gsmhf\" " with result "range_response_count:1 size:2695" took too long (1.673834382s) to execute
2022-01-12 08:20:14.187904 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-live/log-collector-v2-d685cd4c-md6n9\" " with result "range_response_count:1 size:2305" took too long (1.259197019s) to execute
2022-01-12 08:20:14.187911 W | etcdserver: read-only range request "key:\"/registry/pods/mss-frontend-website-id-live/micro-template-server-56475bb9c7-l7l68\" " with result "range_response_count:1 size:2551" took too long (1.712978751s) to execute
2022-01-12 08:20:14.187927 W | etcdserver: read-only range request "key:\"/registry/pods/apc-message-id-live/apc-message-consumer-server-98db65b86-9h65v\" " with result "range_response_count:1 size:2162" took too long (768.380508ms) to execute
2022-01-12 08:20:14.187934 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-live/apc-account-server-6d649b7bc-9b2rn\" " with result "range_response_count:1 size:2062" took too long (1.668666471s) to execute
2022-01-12 08:20:14.187949 W | etcdserver: read-only range request "key:\"/registry/pods/apc-base-id-live/config-gateway-5585cc9f98-g8zm8\" " with result "range_response_count:1 size:1956" took too long (1.711931213s) to execute
2022-01-12 08:20:14.187958 W | etcdserver: read-only range request "key:\"/registry/pods/apc-promotion-id-live/apc-promotion-app-server-69c7677c98-fjr8b\" " with result "range_response_count:1 size:2133" took too long (780.273207ms) to execute
2022-01-12 08:20:14.187970 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/fulfill-server-bdf49d6cc-fdk4l\" " with result "range_response_count:1 size:1934" took too long (682.533589ms) to execute
2022-01-12 08:20:14.187986 W | etcdserver: read-only range request "key:\"/registry/pods/mss-backend-id-live/mis-uploader-5f84449c6-jpm6h\" " with result "range_response_count:1 size:1969" took too long (1.821169473s) to execute
2022-01-12 08:20:14.187996 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/apc-commission-job-595d744cbb-9gbw9\" " with result "range_response_count:1 size:1889" took too long (737.643196ms) to execute
2022-01-12 08:20:14.188034 W | etcdserver: read-only range request "key:\"/registry/pods/apc-wallet-id-uat/wallet-sapi-5697b5c7b4-zzmbq\" " with result "range_response_count:1 size:2020" took too long (1.544605054s) to execute
2022-01-12 08:20:14.188064 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/s3-uploader-6d44788849-wzp2s\" " with result "range_response_count:1 size:1918" took too long (2.202872417s) to execute
2022-01-12 08:20:14.188075 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-live/notify-system-ff58fbf54-p2shf\" " with result "range_response_count:1 size:1896" took too long (2.179303725s) to execute
2022-01-12 08:20:14.188086 W | etcdserver: read-only range request "key:\"/registry/pods/apc-frontend-service-id-live/fe-translation-translation-75f5c7b88-njtjn\" " with result "range_response_count:1 size:2840" took too long (481.432044ms) to execute
2022-01-12 08:20:14.188099 W | etcdserver: read-only range request "key:\"/registry/pods/apc-frontend-gateway-id-live/fe-admin-gateway-5d7766476d-bdn5g\" " with result "range_response_count:1 size:2684" took too long (2.03401502s) to execute
2022-01-12 08:20:14.188105 W | etcdserver: read-only range request "key:\"/registry/namespaces/kube-system\" " with result "range_response_count:1 size:180" took too long (2.016328082s) to execute
2022-01-12 08:20:14.188116 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:469" took too long (2.27934124s) to execute
2022-01-12 08:20:14.188128 W | etcdserver: read-only range request "key:\"/registry/pods/apc-dp-id-live/api-server-6bd46f6c9f-4f7c8\" " with result "range_response_count:1 size:1965" took too long (2.250441928s) to execute
2022-01-12 08:20:14.188144 W | etcdserver: read-only range request "key:\"/registry/pods/apc-message-id-uat/apc-message-server-cf6ffcdf9-q8vxx\" " with result "range_response_count:1 size:2036" took too long (299.321598ms) to execute
2022-01-12 08:20:14.188153 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/api-server-cd645df58-twczm\" " with result "range_response_count:1 size:1939" took too long (328.766404ms) to execute
2022-01-12 08:20:14.188171 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (996.058365ms) to execute
2022-01-12 08:20:14.188179 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/apc-message-schedule-server-565d56d7-b5mxh\" " with result "range_response_count:1 size:2144" took too long (2.810424272s) to execute
2022-01-12 08:20:14.188186 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (956.159584ms) to execute
2022-01-12 08:20:14.188196 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-live/apc-monitor-server-6cc487c557-66tnf\" " with result "range_response_count:1 size:2002" took too long (319.323338ms) to execute
2022-01-12 08:20:14.188202 W | etcdserver: read-only range request "key:\"/registry/pods/apc-message-id-live/apc-message-schedule-server-79b745578-8bsbt\" " with result "range_response_count:1 size:2170" took too long (213.610899ms) to execute
2022-01-12 08:20:14.188214 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/apc-report-admin-server-7989c7b8fc-jc2mv\" " with result "range_response_count:1 size:2064" took too long (554.139983ms) to execute
2022-01-12 08:20:14.188222 W | etcdserver: read-only range request "key:\"/registry/crd.projectcalico.org/clusterinformations/default\" " with result "range_response_count:1 size:565" took too long (2.530284737s) to execute
2022-01-12 08:20:14.188233 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-live/transparent-gateway-77949b7c75-6sl2v\" " with result "range_response_count:1 size:2115" took too long (2.528818762s) to execute
2022-01-12 08:20:14.188242 W | etcdserver: read-only range request "key:\"/registry/crd.projectcalico.org/networkpolicies\" range_end:\"/registry/crd.projectcalico.org/networkpoliciet\" count_only:true " with result "range_response_count:0 size:7" took too long (2.829242069s) to execute
2022-01-12 08:20:14.188251 W | etcdserver: read-only range request "key:\"/registry/pods/apc-dp-id-uat/fulfill-server-69644866bb-7hsf6\" " with result "range_response_count:1 size:1922" took too long (1.186969992s) to execute
2022-01-12 08:20:14.188289 W | etcdserver: read-only range request "key:\"/registry/pods/apc-frontend-service-id-live/frontend-publish-service-6bcf98547b-w2cwf\" " with result "range_response_count:1 size:2744" took too long (801.53786ms) to execute
2022-01-12 08:20:14.188321 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/apc-config-server-84f5dd7cc9-s4kkl\" " with result "range_response_count:1 size:2032" took too long (1.896892247s) to execute
2022-01-12 08:20:14.188362 W | etcdserver: read-only range request "key:\"/registry/pods/apc-dp-gateway-id-live/transparent-gateway-5f547759b5-pw2bm\" " with result "range_response_count:1 size:2122" took too long (654.225343ms) to execute
2022-01-12 08:20:14.188389 W | etcdserver: read-only range request "key:\"/registry/pods/apc-frontend-service-id-uat/rn-transaction-service-7949794974-jlb2j\" " with result "range_response_count:1 size:2872" took too long (1.778844526s) to execute
2022-01-12 08:20:14.188441 W | etcdserver: read-only range request "key:\"/registry/ingress/apc-account-id-live/\" range_end:\"/registry/ingress/apc-account-id-live0\" " with result "range_response_count:8 size:3962" took too long (1.205609373s) to execute
2022-01-12 08:20:14.188473 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/config-gateway-5594dd65d-s68b7\" " with result "range_response_count:1 size:1986" took too long (1.046593154s) to execute
2022-01-12 08:20:14.188481 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-live/apc-pay-autofix-server-849b5668b6-dfq28\" " with result "range_response_count:1 size:2119" took too long (2.720672711s) to execute
2022-01-12 08:20:14.188488 W | etcdserver: read-only range request "key:\"/registry/pods/apc-message-id-live/apc-admin-message-server-5d7d6d79db-gpnsc\" " with result "range_response_count:1 size:2108" took too long (652.950364ms) to execute
2022-01-12 08:20:14.188494 W | etcdserver: read-only range request "key:\"/registry/pods/apc-account-id-uat/apc-pay-server-release-6df5f4f4bb-2hrc4\" " with result "range_response_count:1 size:2030" took too long (2.721109319s) to execute
2022-01-12 08:20:14.188503 W | etcdserver: read-only range request "key:\"/registry/pods/mss-frontend-website-id-live/platform-partner-admin-6465bdb4df-8dmvh\" " with result "range_response_count:1 size:2628" took too long (552.006606ms) to execute
2022-01-12 08:20:14.188590 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/kube-scheduler\" " with result "range_response_count:1 size:451" took too long (1.901739828s) to execute
2022-01-12 08:20:14.188611 W | etcdserver: failed to revoke 3fa476dc16d69e4f ("lease not found")
2022-01-12 08:20:14.188630 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:7" took too long (526.589957ms) to execute
2022-01-12 08:20:14.188681 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/wallet-sapi-5f5849fb8b-cv5x2\" " with result "range_response_count:1 size:2030" took too long (587.075781ms) to execute
2022-01-12 08:20:14.188718 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/config-center-8cffff96d-jsr4z\" " with result "range_response_count:1 size:1972" took too long (142.486828ms) to execute
2022-01-12 08:20:14.188754 W | etcdserver: read-only range request "key:\"/registry/pods/mss-frontend-service-id-uat/partner-common-service-7b6cc55b5d-pfsr7\" " with result "range_response_count:1 size:2556" took too long (2.279323071s) to execute
2022-01-12 08:20:14.188790 W | etcdserver: read-only range request "key:\"/registry/pods/mss-backend-id-live/mis-admin-789d66cb85-wbxz6\" " with result "range_response_count:1 size:1937" took too long (2.414559336s) to execute
2022-01-12 08:20:14.188852 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/apc-approval-server-84cdbff84d-l2wqt\" " with result "range_response_count:1 size:2147" took too long (147.77209ms) to execute
2022-01-12 08:20:14.188894 W | etcdserver: read-only range request "key:\"/registry/pods/apc-admin-id-live/webpublish-admin-website-6d9f9d559f-hp5wf\" " with result "range_response_count:1 size:2747" took too long (613.311209ms) to execute
2022-01-12 08:20:14.188932 W | etcdserver: read-only range request "key:\"/registry/pods/apc-message-id-live/apc-message-server-c579799fb-pznmp\" " with result "range_response_count:1 size:2033" took too long (2.329582279s) to execute
2022-01-12 08:20:14.188971 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-live/s3-uploader-5b955f466-qpcrp\" " with result "range_response_count:1 size:2036" took too long (1.147124717s) to execute
2022-01-12 08:20:14.189016 W | etcdserver: read-only range request "key:\"/registry/pods/apc-account-id-live/apc-promotion-server-6f696c5d96-wm956\" " with result "range_response_count:1 size:2079" took too long (1.773830981s) to execute
2022-01-12 08:20:14.189068 W | etcdserver: read-only range request "key:\"/registry/pods/apc-account-id-uat/apc-pay-server-58bbf58c97-sjhd4\" " with result "range_response_count:1 size:1997" took too long (218.769178ms) to execute
2022-01-12 08:20:14.189102 W | etcdserver: read-only range request "key:\"/registry/pods/apc-commission-id-uat/apc-commission-job-7dd6dfd67d-kt775\" " with result "range_response_count:1 size:1901" took too long (1.554087944s) to execute
2022-01-12 08:20:14.189170 W | etcdserver: read-only range request "key:\"/registry/pods/apc-account-id-live/apc-approval-server-59c95b44f4-v8bhl\" " with result "range_response_count:1 size:2129" took too long (2.10809222s) to execute
2022-01-12 08:20:14.189212 W | etcdserver: read-only range request "key:\"/registry/pods/apc-aggregator-id-live/aggregator-app-server-66fc7698f6-pqzql\" " with result "range_response_count:1 size:2028" took too long (2.246379384s) to execute
2022-01-12 08:20:14.189268 W | etcdserver: read-only range request "key:\"/registry/pods/mss-backend-id-uat/mis-server-5d5df6bccf-wbfdz\" " with result "range_response_count:1 size:1911" took too long (2.455160044s) to execute
2022-01-12 08:20:14.189304 W | etcdserver: read-only range request "key:\"/registry/pods/mss-frontend-website-id-live/micro-template-server-56475bb9c7-8nrzz\" " with result "range_response_count:1 size:2554" took too long (2.369018714s) to execute
2022-01-12 08:20:14.189335 W | etcdserver: failed to revoke 3fa476dc16d69e4f ("lease not found")
2022-01-12 08:20:14.189351 W | etcdserver: read-only range request "key:\"/registry/crd.projectcalico.org/ippools/\" range_end:\"/registry/crd.projectcalico.org/ippools0\" " with result "range_response_count:1 size:561" took too long (989.392916ms) to execute
2022-01-12 08:20:14.189421 W | etcdserver: read-only range request "key:\"/registry/namespaces/\" range_end:\"/registry/namespaces0\" " with result "range_response_count:68 size:21584" took too long (945.059085ms) to execute
2022-01-12 08:20:14.189508 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gateway-id-live/transparent-gateway-67c6d9cd65-wkc4l\" " with result "range_response_count:1 size:2116" took too long (454.724476ms) to execute
2022-01-12 08:20:47.930544 I | etcdserver: start to snapshot (applied: 335494642, lastsnap: 335484641)
2022-01-12 08:20:47.931519 I | etcdserver: saved snapshot at index 335494642
2022-01-12 08:20:47.932198 I | etcdserver: compacted raft log at 335489642
2022-01-12 08:20:50.262288 I | pkg/fileutil: purged file /var/lib/etcd/member/snap/0000000000002ad2-0000000013fe7a9c.snap successfully
2022-01-12 08:22:45.271013 I | mvcc: store.index: compact 267844819
2022-01-12 08:22:45.378675 W | etcdserver: request "header:<ID:14966999623285423737 username:\"kube-apiserver-etcd-client\" auth_revision:1 > compaction:<revision:267844819 > " with result "size:7" took too long (107.793659ms) to execute
2022-01-12 08:22:45.378794 W | etcdserver: read-only range request "key:\"/registry/poddisruptionbudgets\" range_end:\"/registry/poddisruptionbudgett\" count_only:true " with result "range_response_count:0 size:7" took too long (102.634412ms) to execute
2022-01-12 08:22:46.956920 I | mvcc: finished scheduled compaction at 267844819 (took 1.578184708s)
2022-01-12 08:25:07.682391 W | etcdserver: read-only range request "key:\"/registry/pods/apc-promotion-id-live/apc-promotion-app-server-69c7677c98-96hct\" " with result "range_response_count:1 size:2134" took too long (135.058785ms) to execute
2022-01-12 08:25:07.707923 W | etcdserver: read-only range request "key:\"/registry/pods\" range_end:\"/registry/podt\" count_only:true " with result "range_response_count:0 size:10" took too long (129.132977ms) to execute
2022-01-12 08:27:45.361263 I | mvcc: store.index: compact 267848390
2022-01-12 08:27:45.471960 W | etcdserver: request "header:<ID:14966999623285426561 username:\"kube-apiserver-etcd-client\" auth_revision:1 > compaction:<revision:267848390 > " with result "size:7" took too long (110.833583ms) to execute
2022-01-12 08:27:47.056966 I | mvcc: finished scheduled compaction at 267848390 (took 1.584961344s)
2022-01-12 08:32:45.472373 I | mvcc: store.index: compact 267851884
2022-01-12 08:32:45.580999 W | etcdserver: request "header:<ID:14966999623285429384 username:\"kube-apiserver-etcd-client\" auth_revision:1 > compaction:<revision:267851884 > " with result "size:7" took too long (108.809527ms) to execute
2022-01-12 08:32:47.164715 I | mvcc: finished scheduled compaction at 267851884 (took 1.583634169s)
2022-01-12 08:33:59.589711 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-live/s3-uploader-5b955f466-h2xcq\" " with result "range_response_count:1 size:1927" took too long (155.498599ms) to execute
2022-01-12 08:33:59.590286 W | etcdserver: read-only range request "key:\"/registry/pods/apc-message-id-uat/apc-message-consumer-server-6694c848b5-b46gf\" " with result "range_response_count:1 size:2147" took too long (121.460095ms) to execute
2022-01-12 08:34:07.307656 I | etcdserver: start to snapshot (applied: 335504643, lastsnap: 335494642)
2022-01-12 08:34:07.308626 I | etcdserver: saved snapshot at index 335504643
2022-01-12 08:34:07.308958 I | etcdserver: compacted raft log at 335499643
2022-01-12 08:34:20.269827 I | pkg/fileutil: purged file /var/lib/etcd/member/snap/0000000000002ad3-0000000013fea1ae.snap successfully
2022-01-12 08:35:31.387710 I | raft: f852013d0a603fa4 [term: 10968] received a MsgVote message with higher term from 8294be07352acfb5 [term: 10969]
2022-01-12 08:35:31.387756 I | raft: f852013d0a603fa4 became follower at term 10969
2022-01-12 08:35:31.387767 I | raft: f852013d0a603fa4 [logterm: 10968, index: 335505871, vote: 0] cast MsgVote for 8294be07352acfb5 [logterm: 10968, index: 335505871] at term 10969
2022-01-12 08:35:31.387774 I | raft: raft.node: f852013d0a603fa4 lost leader d1460a24dccdab22 at term 10969
2022-01-12 08:35:31.390313 I | raft: raft.node: f852013d0a603fa4 elected leader 8294be07352acfb5 at term 10969
2022-01-12 08:35:31.391649 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/apc-message-consumer-server-6b55b9b746-t8vj4\" " with result "error:etcdserver: leader changed" took too long (1.039229803s) to execute
2022-01-12 08:35:31.392434 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-live/config-center-74c8c7c5d-6bvbn\" " with result "range_response_count:1 size:1979" took too long (1.034765677s) to execute
2022-01-12 08:35:31.392516 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/dp-transparent-gateway-6c768b67db-lpk7j\" " with result "range_response_count:1 size:2147" took too long (719.974975ms) to execute
2022-01-12 08:35:31.392568 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-uat/apc-monitor-server-5c8b9dd5f9-hr4w8\" " with result "range_response_count:1 size:2021" took too long (404.741938ms) to execute
2022-01-12 08:35:31.392613 W | etcdserver: read-only range request "key:\"/registry/pods/apc-account-id-live/apc-report-admin-server-64c7b5785f-vxbng\" " with result "range_response_count:1 size:2104" took too long (662.154978ms) to execute
2022-01-12 08:35:31.392628 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gateway-id-uat/transparent-gateway-59cf8597cc-mdgdn\" " with result "range_response_count:1 size:2161" took too long (492.431794ms) to execute
2022-01-12 08:35:31.392661 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-live/apc-admin-edge-server-59966c6b98-9sld7\" " with result "range_response_count:1 size:2104" took too long (650.396267ms) to execute
2022-01-12 08:35:31.392672 W | etcdserver: read-only range request "key:\"/registry/pods/apc-wallet-id-live/wallet-worker-5748b8f55f-ktxhs\" " with result "range_response_count:1 size:2059" took too long (535.13249ms) to execute
2022-01-12 08:35:31.392687 W | etcdserver: read-only range request "key:\"/registry/pods/mss-frontend-website-id-live/one-credential-server-78998b8d8b-q9jxt\" " with result "range_response_count:1 size:2540" took too long (258.053176ms) to execute
2022-01-12 08:35:31.392702 W | etcdserver: read-only range request "key:\"/registry/pods/apc-transaction-id-uat/apc-archive-server-5995687985-r77xr\" " with result "range_response_count:1 size:2053" took too long (972.988394ms) to execute
2022-01-12 08:35:31.392716 W | etcdserver: read-only range request "key:\"/registry/pods/apc-commission-id-live/apc-commission-server-5bb566df66-wnvp5\" " with result "range_response_count:1 size:2135" took too long (570.520297ms) to execute
2022-01-12 08:35:31.392730 W | etcdserver: read-only range request "key:\"/registry/controllers\" range_end:\"/registry/controllert\" count_only:true " with result "range_response_count:0 size:7" took too long (650.778042ms) to execute
2022-01-12 08:35:31.392739 W | etcdserver: read-only range request "key:\"/registry/pods/mss-frontend-website-id-live/partner-server-749989b464-mb4nm\" " with result "range_response_count:1 size:2445" took too long (355.239883ms) to execute
2022-01-12 08:35:31.392751 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-live/apc-promotion-server-6ff55b6f5-4jvlx\" " with result "range_response_count:1 size:2079" took too long (642.148759ms) to execute
2022-01-12 08:35:31.392763 W | etcdserver: read-only range request "key:\"/registry/pods/mss-backend-id-live/mis-partner-d869687dd-qjjgb\" " with result "range_response_count:1 size:1960" took too long (789.465738ms) to execute
2022-01-12 08:35:31.392772 W | etcdserver: read-only range request "key:\"/registry/pods/apc-gray-id-uat/apc-admin-message-server-7bdc5b54bc-xf9qz\" " with result "range_response_count:1 size:2121" took too long (971.024697ms) to execute
2022-01-12 08:35:31.392785 W | etcdserver: read-only range request "key:\"/registry/pods/apc-message-id-uat/apc-admin-message-server-679f8f986c-2nqkf\" " with result "range_response_count:1 size:2116" took too long (971.241522ms) to execute
2022-01-12 08:35:31.392831 W | etcdserver: read-only range request "key:\"/registry/pods/apc-aggregator-id-live/aggregator-server-687d8f9cff-84lhm\" " with result "range_response_count:1 size:1977" took too long (782.368278ms) to execute
2022-01-12 08:35:31.392847 W | etcdserver: read-only range request "key:\"/registry/minions/\" range_end:\"/registry/minions0\" " with result "range_response_count:24 size:242681" took too long (887.134803ms) to execute
2022-01-12 08:35:31.392875 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-live/log-collector-v2-d685cd4c-8b9ht\" " with result "range_response_count:1 size:2308" took too long (283.345157ms) to execute
2022-01-12 08:35:31.392927 W | etcdserver: read-only range request "key:\"/registry/pods/apc-message-id-uat/apc-push-server-976949699-g5kt6\" " with result "range_response_count:1 size:1957" took too long (808.770635ms) to execute
2022-01-12 08:35:31.392955 W | etcdserver: read-only range request "key:\"/registry/pods/apc-account-id-live/apc-pay-server-674d7cbc95-bckc6\" " with result "range_response_count:1 size:2015" took too long (470.281315ms) to execute
2022-01-12 08:35:31.392984 W | etcdserver: read-only range request "key:\"/registry/pods/apc-infra-id-uat/apc-delay-queue-server-c9fbfd7bd-h8q9w\" " with result "range_response_count:1 size:2045" took too long (604.076876ms) to execute
2022-01-12 08:35:31.393017 W | etcdserver: read-only range reque`

2022-01-12 08:20:14.169194 W | wal: sync duration of 2.88067165s, expected less than 1s

Suggests that underlying disk is too slow. Some things to check:

• are there other processes writing a lot
• is it SSD
• is it big enough (if it's cloud disk) to support sufficient IO-ops / throughput.
• is the OS swapping on the disk...

https://github.com/etcd-io/etcd/issues/10215

etcd version: 3.3

2022-01-12 08:20:14.169194 W | wal: sync duration of 2.88067165s, expected less than 1s

Suggests that underlying disk is too slow. Some things to check:

• are there other processes writing a lot
• is it SSD
• is it big enough (if it's cloud disk) to support sufficient IO-ops / throughput.
• is the OS swapping on the disk...

#10215

We are in a production environment，It was normal a few days ago，the latency distributions of fsync called by WAL increased at some point (as shown in the figure above)，but I am not clear why it affect the election,Are there are other ways to alleviate it ?for example, increase heartbeat-interval and election-timeout can work ?。

@didihongsheng did you fix the issue, if yes please share how since i am facing the same

@didihongsheng did you fix the issue, if yes please share how since i am facing the same

yes，I replace the disk to ssd.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.