tidb panic after injection network partition between two AZ

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run mussel workload 2、inject network partition between two AZ

2. What did you expect to see? (Required)

no panic

3. What did you see instead (Required)

tidb panic 2024-06-30 10:15:40 log="\n" 2024-06-30 10:15:40 log="/tidb-server --store=tikv --advertise-address=tc-tidb-1.tc-tidb-peer.endless-ha-test-airbnb-tps-7510461-1-362.svc --host=0.0.0.0 --path=tc-pd:2379 --config=/etc/tidb/tidb.toml\n" 2024-06-30 10:15:40 log="start tidb-server ...\n" 2024-06-30 10:15:37 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/cmd/tidb-server/main.go:905 +0x37c\n" 2024-06-30 10:15:37 log="created by main.createServer in goroutine 1\n" 2024-06-30 10:15:37 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/expensivequery/expensivequery.go:98 +0xaa8\n" 2024-06-30 10:15:37 log="github.com/pingcap/tidb/pkg/util/expensivequery.(*Handle).Run(0xc003868408)\n" 2024-06-30 10:15:37 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:918 +0x515\n" 2024-06-30 10:15:37 log="github.com/pingcap/tidb/pkg/server.(*Server).Kill(0xc003cd3b00, 0x1241bfee, 0x1, 0x0?)\n" 2024-06-30 10:15:37 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:950 +0x2ca\n" 2024-06-30 10:15:37 log="github.com/pingcap/tidb/pkg/server.killQuery(0xc1a069e680, 0x1)\n" 2024-06-30 10:15:37 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/sqlkiller/sqlkiller.go:84\n" 2024-06-30 10:15:37 log="github.com/pingcap/tidb/pkg/util/sqlkiller.(*SQLKiller).FinishResultSet(...)\n" 2024-06-30 10:15:37 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2064 +0x1c\n" 2024-06-30 10:15:37 log="github.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt.func1()\n" 2024-06-30 10:15:37 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/internal/resultset/resultset.go:69 +0x33\n" 2024-06-30 10:15:37 log="github.com/pingcap/tidb/pkg/server/internal/resultset.(*tidbResultSet).Finish(0xa2b2e20?)\n" 2024-06-30 10:15:37 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:2358 +0x14\n" 2024-06-30 10:15:37 log="github.com/pingcap/tidb/pkg/session.(*execStmtResult).Finish(0xc0032f59a0?)\n" 2024-06-30 10:15:37 log="goroutine 12862 [running]:\n" 2024-06-30 10:15:37 log="\n" 2024-06-30 10:15:37 log="[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x540d174]\n" 2024-06-30 10:15:37 log="panic: runtime error: invalid memory address or nil pointer dereference\n" 2024-06-30 10:15:37 log="[conn.go:1162] [\"command dispatched failed\"] [conn=306298908] [session_alias=] [connInfo=\"id:306298908, addr:10.233.108.10:51888 status:10, collation:utf8mb4_general_ci, user:root\"] [command=Query] [status=\"inTxn:0, autocommit:1\"] [sql=\"select /*+ max_execution_time(400), set_var(tikv_client_read_timeout=100) */ pk, sk, ts, v from t1 as of timestamp now() - interval 10 second where pk = '151832271' and sk = 'y4_16' and ts >= '2024-06-30 02:14:36.647382' and ts < '2024-06-30 02:15:26.647382' order by ts desc limit 5\"] [txn_mode=PESSIMISTIC] [timestamp=450812634988544000] [err=\"context canceled\\ngithub.com/pingcap/errors.AddStack\\n\\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:178\\ngithub.com/pingcap/errors.Trace\\n\\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\\ngithub.com/pingcap/tidb/pkg/store/copr.(*copIterator).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/store/copr/coprocessor.go:1095\\ngithub.com/pingcap/tidb/pkg/distsql.(*selectResult).fetchResp\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/distsql/select_result.go:318\\ngithub.com/pingcap/tidb/pkg/distsql.(*selectResult).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/distsql/select_result.go:384\\ngithub.com/pingcap/tidb/pkg/executor.(*tableResultHandler).nextChunk\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/table_reader.go:607\\ngithub.com/pingcap/tidb/pkg/executor.(*TableReaderExecutor).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/table_reader.go:330\\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:410\\ngithub.com/pingcap/tidb/pkg/executor.(*LimitExec).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/executor.go:1366\\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:410\\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:1250\\ngithub.com/pingcap/tidb/pkg/executor.(*recordSet).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:175\\ngithub.com/pingcap/tidb/pkg/server/internal/resultset.(*tidbResultSet).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/internal/resultset/resultset.go:64\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).writeChunks\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2332\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).writeResultSet\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2275\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2068\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1785\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).dispatch\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1359\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).Run\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1125\\ngithub.com/pingcap/tidb/pkg/server.(*Server).onConn\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:739\\nruntime.goexit\\n\\t/usr/local/go/src/runtime/asm_amd64.s:1650\"]" 2024-06-30 10:15:37 log="[conn.go:1162] [\"command dispatched failed\"] [conn=306298862] [session_alias=] [connInfo=\"id:306298862, addr:10.233.119.153:60770 status:10, collation:utf8mb4_general_ci, user:root\"] [command=Query] [status=\"inTxn:0, autocommit:1\"] [sql=\"select /*+ max_execution_time(400), set_var(tikv_client_read_timeout=100) */ pk, sk, ts, v from t1 as of timestamp now() - interval 10 second where pk = '151832271' and sk = 'y4_16' and ts >= '2024-06-30 02:14:36.648815' and ts < '2024-06-30 02:15:26.648815' order by ts desc limit 5\"] [txn_mode=PESSIMISTIC] [timestamp=450812634988544000] [err=\"context canceled\\ngithub.com/pingcap/errors.AddStack\\n\\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:178\\ngithub.com/pingcap/errors.Trace\\n\\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\\ngithub.com/pingcap/tidb/pkg/store/copr.(*copIterator).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/store/copr/coprocessor.go:1095\\ngithub.com/pingcap/tidb/pkg/distsql.(*selectResult).fetchResp\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/distsql/select_result.go:318\\ngithub.com/pingcap/tidb/pkg/distsql.(*selectResult).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/distsql/select_result.go:384\\ngithub.com/pingcap/tidb/pkg/executor.(*tableResultHandler).nextChunk\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/table_reader.go:607\\ngithub.com/pingcap/tidb/pkg/executor.(*TableReaderExecutor).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/table_reader.go:330\\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:410\\ngithub.com/pingcap/tidb/pkg/executor.(*LimitExec).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/executor.go:1366\\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:410\\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:1250\\ngithub.com/pingcap/tidb/pkg/executor.(*recordSet).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:175\\ngithub.com/pingcap/tidb/pkg/server/internal/resultset.(*tidbResultSet).Next\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/internal/resultset/resultset.go:64\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).writeChunks\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2332\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).writeResultSet\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2275\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2068\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1785\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).dispatch\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1359\\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).Run\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1125\\ngithub.com/pingcap/tidb/pkg/server.(*Server).onConn\\n\\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:739\\nruntime.goexit\\n\\t/usr/local/go/src/runtime/asm_amd64.s:1650\"]" 2024-06-30 10:15:37 log="[sqlkiller.go:112] [\"kill finished\"] [conn=306298908]"

4. What is your TiDB version? (Required)

./tidb-server -V Release Version: v8.2.0-alpha Edition: Community Git Commit Hash: 7df4f66324905dfe2bf9e0f5288a0b2ce089098c Git Branch: heads/refs/tags/v8.2.0-alpha UTC Build Time: 2024-06-29 11:47:18 GoVersion: go1.21.10 Race Enabled: false Check Table Before Drop: false Store: unistore 2024-06-30T09:57:02.114+0800