Flaking `TestLeaseGrantTimeToLiveExpired`

[ahrtr]

Which Github Action / Prow Jobs are flaking?

arm64 / test (linux-arm64-integration-1-cpu)

Which tests are flaking?

TestLeaseGrantTimeToLiveExpired

Github Action / Prow Job link

https://github.com/etcd-io/etcd/actions/runs/8078398584/job/22070629565

Reason for failure (if possible)

    lease_test.go:146: 
        	Error Trace:	/home/runner/actions-runner/_work/etcd/etcd/tests/common/lease_test.go:146
        	            				/home/runner/actions-runner/_work/etcd/etcd/tests/framework/testutils/execute.go:38
        	            				/home/runner/actions-runner/_work/_tool/go/1.21.6/arm64/src/runtime/asm_arm64.s:1197
        	Error:      	Not equal: 
        	            	expected: -1
        	            	actual  : 1
        	Test:       	TestLeaseGrantTimeToLiveExpired/PeerAutoTLS

Anything else we need to know?

No response

[jmhbnz]

Another instance https://github.com/etcd-io/etcd/actions/runs/8144423225/job/22282118668#step:5:1807.

[jmhbnz]

Another instance https://github.com/etcd-io/etcd/actions/runs/8500313218/job/23282034904?pr=17677.

[fuweid]

I think it's related to leader change. We can retry it 3 times if it's related to leader change. /assign

24-03-31T19:20:48.1185991Z     logger.go:146: 2024-03-31T19:16:04.271Z	INFO	m2.raft	c81b22754673dbd7 became follower at term 4	{"member": "m2"}
2024-03-31T19:20:48.1188332Z     logger.go:146: 2024-03-31T19:16:04.271Z	INFO	m2.raft	raft.node: c81b22754673dbd7 changed leader from 3f9a3c374f2c7e67 to f9a0f85dc7e1942c at term 4	{"member": "m2"}
2024-03-31T19:20:48.1190852Z     logger.go:146: 2024-03-31T19:16:04.272Z	WARN	m0	Failed to check current member's leadership	{"member": "m0", "error": "etcdserver: leader changed"}
2024-03-31T19:20:48.1197275Z     logger.go:146: 2024-03-31T19:16:04.272Z	WARN	m0	Ignore the lease revoking request because current member isn't a leader	{"member": "m0", "local-member-id": 4583041779052084839}
2024-03-31T19:20:48.1201217Z     logger.go:146: 2024-03-31T19:16:04.572Z	WARN	m1	leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk	{"member": "m1", "to": "c81b22754673dbd7", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "2.36516ms"}
2024-03-31T19:20:48.1205640Z     logger.go:146: 2024-03-31T19:16:04.572Z	WARN	m1	leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk	{"member": "m1", "to": "3f9a3c374f2c7e67", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "2.52152ms"}
2024-03-31T19:20:48.1210353Z     logger.go:146: 2024-03-31T19:16:04.602Z	WARN	m1	leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk	{"member": "m1", "to": "c81b22754673dbd7", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "685.36µs"}
2024-03-31T19:20:48.1215045Z     logger.go:146: 2024-03-31T19:16:04.603Z	WARN	m1	leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk	{"member": "m1", "to": "3f9a3c374f2c7e67", "heartbeat-interval": "10ms", "expected-duration": "20ms", "exceeded-duration": "811.24µs"}
2024-03-31T19:20:48.1217241Z     lease_test.go:146: 
2024-03-31T19:20:48.1218472Z         	Error Trace:	/home/runner/actions-runner/_work/etcd/etcd/tests/common/lease_test.go:146
2024-03-31T19:20:48.1220675Z         	            				/home/runner/actions-runner/_work/etcd/etcd/tests/framework/testutils/execute.go:38
2024-03-31T19:20:48.1222788Z         	            				/home/runner/actions-runner/_work/_tool/go/1.22.1/arm64/src/runtime/asm_arm64.s:1222
2024-03-31T19:20:48.1223781Z         	Error:      	Not equal: 
2024-03-31T19:20:48.1224504Z         	            	expected: -1
2024-03-31T19:20:48.1225333Z         	            	actual  : 1
2024-03-31T19:20:48.1226627Z         	Test:       	TestLeaseGrantTimeToLiveExpired/PeerAutoTLS
2024-03-31T19:20:48.1227698Z     cluster.go:1417: ========= Cluster termination started =====================