cockroach icon indicating copy to clipboard operation
cockroach copied to clipboard

Published 20 hours ago • verified publisher icon cockroachdb

kv/kvserver: TestReplicateQueueRebalanceMultiStore failed

Open cockroach-teamcity opened this issue 1 year ago • 8 comments

kv/kvserver.TestReplicateQueueRebalanceMultiStore failed with artifacts on master @ 3eb2bb04abaaaaac92f3f5f2f6952a30ada78de5:

    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [54 6 7 6 6]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [53 6 7 7 6]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [52 6 7 8 6]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [51 6 7 9 6]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [50 7 7 9 6]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [49 9 7 8 6]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [48 10 7 7 7]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [47 10 7 6 9]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [46 10 7 6 10]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [45 9 8 8 9]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [44 8 10 8 9]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [43 7 11 8 10]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [42 7 10 9 11]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [41 7 9 10 12]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [40 7 8 12 12]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [39 6 9 13 12]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [38 7 11 12 11]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [37 10 10 11 11]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [36 11 9 10 13]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [35 10 12 10 12]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [34 9 13 12 11]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [33 10 12 14 10]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [32 13 11 13 10]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [31 15 10 12 11]
    replicate_queue_test.go:334: current replica state (want at least 67 replicas on all stores): [74 75 73 77 74]
    replicate_queue_test.go:345: current lease state (want at least 11 leases on all stores): [30 14 12 11 12]
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-43338

cockroach-teamcity avatar Oct 18 '24 10:10 cockroach-teamcity

kv/kvserver.TestReplicateQueueRebalanceMultiStore failed on master @ 9c3acd84e2fc639328b27a6cacd0e3893f1f90bb:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x142e811]

Stack:

goroutine 1537181 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc012fdb868?, {0x8d99218, 0xc00dfc07e0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x6895420?, 0xd0f5aa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc012fdb988?, {0x8d99218, 0xc00dfc0960})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x6895420?, 0xd0f5aa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8d99218, 0xc00dfc0fc0}, 0x1142)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc012fdc768, {0x8d99218?, 0xc00dfc0fc0?}, 0xc00635dce0?, 0x18?)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0x11, 0x5, 0xc007914a10, 0xc007914a80, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc00ef3c008, {0x8d99218, 0xc00dfc0960}, 0xc007095208, {0x7fe6eb9ee3b0, 0xc002243e30})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x76b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8d99218, 0xc00dfc0960})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0075cbd40, {0x8d99218, 0xc00dfc0960}, {0x746e8ec, 0x1e}, 0xc00b2f1280)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc00ef3c008, {0x8d991e0?, 0xc001a60500?}, 0xc007095208, {0x7fe6eb9ee388, 0xc002243e30})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc00de3c380, {0x8de74e8, 0xc002243e30})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x32d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x72a7fc0?, 0xc00de3c380}, {0x8dd19d0, 0xc0046e6000})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x72a7fc0, 0xc00de3c380}, {0x8dd1748, 0xc00dbaea50}, 0xc01324e108, 0x787ebd8)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x72a7fc0, 0xc00de3c380}, {0x8dd1748, 0xc00dbaea50})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x72a7fc0, 0xc00de3c380}, {0x8dd1748, 0xc00dbaea50}, 0xc01324e108?, 0xc0070bfc80)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x72a7fc0, 0xc00de3c380}, {0x8dd1748, 0xc00dbaea50})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc008764000?, {{0x470445?}, {0x8df41a0?, 0xc00ef6f740?}}}, {0x72a7fc0, 0xc00de3c380}, {0x8dd1748, 0xc00dbaea50}, 0xc01324e108, 0xc0070bfc40)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x72a7fc0, 0xc00de3c380}, {0x8dd1748, 0xc00dbaea50})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7fe6ac1b7a58?, 0x8d99218?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0075cbd40, {0x8d99218, 0xc00dfc07e0}, {0xc01427cd80, 0x29}, 0xc00af6aaa0)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x72a7fc0, 0xc00de3c380}, {0x8dd1748, 0xc00dbaea50}, 0xc01324e108, 0xc0070bfc00)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x72a7fc0, 0xc00de3c380}, {0x8dd1748, 0xc00dbaea50}, 0xc01324e108, 0xc002243de0?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc004e5ed20, {0x8df5940, 0xc012a3dd40}, 0xc007f939e0, 0xc0170c76e0, 0xd1f2240, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc004e5ed20, {0x8df5940, 0xc012a3dd40}, 0xc007f939e0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 1537205
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136
Log preceding fatal error 

*   | 	GOROOT/src/runtime/signal_unix.go:881
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140
*   | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472
*   | google.golang.org/grpc.(*Server).processStreamingRPC
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636
*   | google.golang.org/grpc.(*Server).handleStream
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716
*   | google.golang.org/grpc.(*Server).serveStreams.func1.1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters:

  • attempt=1
  • run=5
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 18 '24 11:10 cockroach-teamcity

kv/kvserver.TestReplicateQueueRebalanceMultiStore failed on master @ bb702bae5b316c3def3e7c9c2f409967672bfcd9:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x142e911]

Stack:

goroutine 1411972 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00cff1868?, {0x8da4b98, 0xc0170f99e0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x689f360?, 0xd105aa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00cff1988?, {0x8da4b98, 0xc018cd3a40})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x689f360?, 0xd105aa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8da4b98, 0xc018cd3b30}, 0x557)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc00cff2768, {0x8da4b98?, 0xc018cd3b30?}, 0xc01078adc0?, 0x98?)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc010bbfea0, 0xc010bbff10, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc004f91c08, {0x8da4b98, 0xc018cd3a40}, 0xc01442c488, {0x7f6549d8aca8, 0xc001f5b7d0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x76b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8da4b98, 0xc018cd3a40})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0068c4750, {0x8da4b98, 0xc018cd3a40}, {0x7478aaa, 0x1e}, 0xc0030d7280)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc004f91c08, {0x8da4b60?, 0xc011143a90?}, 0xc01442c488, {0x7f6549d8ac80, 0xc001f5b7d0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc008cc2fc0, {0x8df2d88, 0xc001f5b7d0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x32d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x72b1ea0?, 0xc008cc2fc0}, {0x8ddd350, 0xc0115bb400})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x72b1ea0, 0xc008cc2fc0}, {0x8ddd0c8, 0xc004028870}, 0xc011423a10, 0x7888f78)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x72b1ea0, 0xc008cc2fc0}, {0x8ddd0c8, 0xc004028870})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x72b1ea0, 0xc008cc2fc0}, {0x8ddd0c8, 0xc004028870}, 0xc011423a10?, 0xc013230d40)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x72b1ea0, 0xc008cc2fc0}, {0x8ddd0c8, 0xc004028870})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00817c000?, {{0x470445?}, {0x8dffa40?, 0xc0162255c0?}}}, {0x72b1ea0, 0xc008cc2fc0}, {0x8ddd0c8, 0xc004028870}, 0xc011423a10, 0xc013230d00)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x72b1ea0, 0xc008cc2fc0}, {0x8ddd0c8, 0xc004028870})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7f6547479c08?, 0x8da4b98?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0068c4750, {0x8da4b98, 0xc0170f99e0}, {0xc011cb67e0, 0x29}, 0xc005d3faa0)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x72b1ea0, 0xc008cc2fc0}, {0x8ddd0c8, 0xc004028870}, 0xc011423a10, 0xc013230cc0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x72b1ea0, 0xc008cc2fc0}, {0x8ddd0c8, 0xc004028870}, 0xc011423a10, 0xc001f5b7c0?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00564a000, {0x8e011e0, 0xc01124cea0}, 0xc0112d9440, 0xc0069e8690, 0xd202300, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc00564a000, {0x8e011e0, 0xc01124cea0}, 0xc0112d9440, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 1411957
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136
Log preceding fatal error 

*   | 	GOROOT/src/runtime/signal_unix.go:881
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140
*   | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472
*   | google.golang.org/grpc.(*Server).processStreamingRPC
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636
*   | google.golang.org/grpc.(*Server).handleStream
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716
*   | google.golang.org/grpc.(*Server).serveStreams.func1.1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters:

  • attempt=1
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

github-actions[bot] avatar Oct 18 '24 15:10 github-actions[bot]

This only fails when the metamorphic constant is set to true. It is set to false by default on release builds. So not a release blocker, but will fix this soon.

initialized metamorphic constant "kvadmission.store.snapshot_ingest_bandwidth_control.enabled" with value true

aadityasondhi avatar Oct 18 '24 15:10 aadityasondhi

This has me scratching my head a little.

Before calling Admit() in Pace() (the only caller of Admit()), we explicitly do a nil check on SnapshotQueue, but inside Admit() this panics on SnapshotQueue being nil.

aadityasondhi avatar Oct 18 '24 17:10 aadityasondhi

It is a flaky failure, only fails in ~1/20 runs even with the cluster setting manually set to true.

aadityasondhi avatar Oct 18 '24 17:10 aadityasondhi

kv/kvserver.TestReplicateQueueRebalanceMultiStore failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x142fcd1]

Stack:

goroutine 1758998 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc0116b7820?, {0x8db15f8, 0xc00ecb25a0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x68a8500?, 0xd11caa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc0116b7940?, {0x8db15f8, 0xc00ecb26f0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x68a8500?, 0xd11caa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8db15f8, 0xc00ecb27b0}, 0x565)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc0116b8720, {0x8db15f8?, 0xc00ecb27b0?}, 0xc0041002c0?, 0xf8?)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc0035fe380, 0xc0035fe3f0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc0095cdc08, {0x8db15f8, 0xc00ecb26f0}, 0xc006640fc8, {0x7f237c2a9f78, 0xc007eff8d0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x797
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8db15f8, 0xc00ecb26f0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0072d2000, {0x8db15f8, 0xc00ecb26f0}, {0x7483b44, 0x1e}, 0xc004db7268)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc0095cdc08, {0x8db15c0?, 0xc00d3b60f0?}, 0xc006640fc8, {0x7f237c2a9f50, 0xc007eff8d0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc00fb58e00, {0x8dff768, 0xc007eff8d0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x30d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x72bcd60?, 0xc00fb58e00}, {0x8de9d30, 0xc00f71f220})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x72bcd60, 0xc00fb58e00}, {0x8de9aa8, 0xc009792690}, 0xc0107b1950, 0x78943a8)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00fb58e00}, {0x8de9aa8, 0xc009792690})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x72bcd60, 0xc00fb58e00}, {0x8de9aa8, 0xc009792690}, 0xc0107b1950?, 0xc006619740)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00fb58e00}, {0x8de9aa8, 0xc009792690})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc009236000?, {{0x470445?}, {0x8e0c420?, 0xc00bfbfd40?}}}, {0x72bcd60, 0xc00fb58e00}, {0x8de9aa8, 0xc009792690}, 0xc0107b1950, 0xc006619700)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00fb58e00}, {0x8de9aa8, 0xc009792690})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7f237d1a6a48?, 0x8db15f8?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0072d2000, {0x8db15f8, 0xc00ecb25a0}, {0xc0021963f0, 0x29}, 0xc0077f5aa0)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x72bcd60, 0xc00fb58e00}, {0x8de9aa8, 0xc009792690}, 0xc0107b1950, 0xc0066196c0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x72bcd60, 0xc00fb58e00}, {0x8de9aa8, 0xc009792690}, 0xc0107b1950, 0xc007eff8c0?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00734ab40, {0x8e0dbc0, 0xc003f92000}, 0xc009173320, 0xc01222f830, 0xd2192e0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc00734ab40, {0x8e0dbc0, 0xc003f92000}, 0xc009173320, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 1758995
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136
Log preceding fatal error 

*   | 	GOROOT/src/runtime/signal_unix.go:881
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140
*   | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472
*   | google.golang.org/grpc.(*Server).processStreamingRPC
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636
*   | google.golang.org/grpc.(*Server).handleStream
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716
*   | google.golang.org/grpc.(*Server).serveStreams.func1.1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters:

  • attempt=1
  • run=1
  • shard=2
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

github-actions[bot] avatar Oct 18 '24 22:10 github-actions[bot]

kv/kvserver.TestReplicateQueueRebalanceMultiStore failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x142fcd1]

Stack:

goroutine 808251 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00ac9b820?, {0x8db15f8, 0xc009a26900})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x68a8500?, 0xd11caa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00ac9b940?, {0x8db15f8, 0xc00797f170})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x68a8500?, 0xd11caa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8db15f8, 0xc00797fbc0}, 0x557)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc00ac9c720, {0x8db15f8?, 0xc00797fbc0?}, 0xc011448840?, 0xf8?)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc0112e6770, 0xc0112e67e0, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc009eac808, {0x8db15f8, 0xc00797f170}, 0xc00f8e4488, {0x7f7ad0671828, 0xc012deb350})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x797
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8db15f8, 0xc00797f170})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00edea120, {0x8db15f8, 0xc00797f170}, {0x7483b44, 0x1e}, 0xc002db5268)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc009eac808, {0x8db15c0?, 0xc01433d0e0?}, 0xc00f8e4488, {0x7f7ad0671800, 0xc012deb350})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc002899a40, {0x8dff768, 0xc012deb350})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x30d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x72bcd60?, 0xc002899a40}, {0x8de9d30, 0xc007c2bd00})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x72bcd60, 0xc002899a40}, {0x8de9aa8, 0xc009550780}, 0xc006a9ff50, 0x78943a8)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc002899a40}, {0x8de9aa8, 0xc009550780})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x72bcd60, 0xc002899a40}, {0x8de9aa8, 0xc009550780}, 0xc006a9ff50?, 0xc0010a7080)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc002899a40}, {0x8de9aa8, 0xc009550780})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00ef86000?, {{0x470445?}, {0x8e0c420?, 0xc01ba79d40?}}}, {0x72bcd60, 0xc002899a40}, {0x8de9aa8, 0xc009550780}, 0xc006a9ff50, 0xc0010a6fc0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc002899a40}, {0x8de9aa8, 0xc009550780})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7f7ad43815f8?, 0x8db15f8?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00edea120, {0x8db15f8, 0xc009a26900}, {0xc00ac583c0, 0x29}, 0xc0083f8aa0)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x72bcd60, 0xc002899a40}, {0x8de9aa8, 0xc009550780}, 0xc006a9ff50, 0xc0010a6f80)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x72bcd60, 0xc002899a40}, {0x8de9aa8, 0xc009550780}, 0xc006a9ff50, 0xc012deb340?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0049f3a40, {0x8e0dbc0, 0xc006a431e0}, 0xc0120db0e0, 0xc00c2726c0, 0xd2192e0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc0049f3a40, {0x8e0dbc0, 0xc006a431e0}, 0xc0120db0e0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 808166
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136
Log preceding fatal error 

*   | 	GOROOT/src/runtime/signal_unix.go:881
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140
*   | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472
*   | google.golang.org/grpc.(*Server).processStreamingRPC
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636
*   | google.golang.org/grpc.(*Server).handleStream
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716
*   | google.golang.org/grpc.(*Server).serveStreams.func1.1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters:

  • attempt=1
  • run=1
  • shard=2
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #132995 kv/kvserver: TestReplicateQueueRebalanceMultiStore failed [C-test-failure O-robot T-kv branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 19 '24 13:10 cockroach-teamcity

kv/kvserver.TestReplicateQueueRebalanceMultiStore failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x142fcd1]

Stack:

goroutine 508705 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc010d07820?, {0x8db15f8, 0xc018fee5a0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x68a8500?, 0xd11caa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc010d07940?, {0x8db15f8, 0xc018fee6f0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x68a8500?, 0xd11caa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8db15f8, 0xc018feecf0}, 0x557)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc010d08720, {0x8db15f8?, 0xc018feecf0?}, 0xc0111bc2c0?, 0xf8?)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc009a047e0, 0xc009a04850, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc01206cc08, {0x8db15f8, 0xc018fee6f0}, 0xc012c1d208, {0x7f44d9a10f80, 0xc01be3af70})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x797
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8db15f8, 0xc018fee6f0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00f7407e0, {0x8db15f8, 0xc018fee6f0}, {0x7483b44, 0x1e}, 0xc0058db268)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc01206cc08, {0x8db15c0?, 0xc012412230?}, 0xc012c1d208, {0x7f44d9a10f58, 0xc01be3af70})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc00f757180, {0x8dff768, 0xc01be3af70})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x30d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x72bcd60?, 0xc00f757180}, {0x8de9d30, 0xc012a373e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x72bcd60, 0xc00f757180}, {0x8de9aa8, 0xc010c6b0e0}, 0xc015f01848, 0x78943a8)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00f757180}, {0x8de9aa8, 0xc010c6b0e0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x72bcd60, 0xc00f757180}, {0x8de9aa8, 0xc010c6b0e0}, 0xc015f01848?, 0xc00bf6c200)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00f757180}, {0x8de9aa8, 0xc010c6b0e0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00fbce000?, {{0x470445?}, {0x8e0c420?, 0xc00ef97800?}}}, {0x72bcd60, 0xc00f757180}, {0x8de9aa8, 0xc010c6b0e0}, 0xc015f01848, 0xc00bf6c1c0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00f757180}, {0x8de9aa8, 0xc010c6b0e0})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7f44d9a2d478?, 0x8db15f8?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00f7407e0, {0x8db15f8, 0xc018fee5a0}, {0xc00947b3e0, 0x29}, 0xc00b4bbaa0)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x72bcd60, 0xc00f757180}, {0x8de9aa8, 0xc010c6b0e0}, 0xc015f01848, 0xc00bf6c180)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x72bcd60, 0xc00f757180}, {0x8de9aa8, 0xc010c6b0e0}, 0xc015f01848, 0xc01be3af60?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0137cba40, {0x8e0dbc0, 0xc00befd1e0}, 0xc00f24f9e0, 0xc0016eaa80, 0xd2192e0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc0137cba40, {0x8e0dbc0, 0xc00befd1e0}, 0xc00f24f9e0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 508708
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136
Log preceding fatal error 

*   | 	GOROOT/src/runtime/signal_unix.go:881
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140
*   | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472
*   | google.golang.org/grpc.(*Server).processStreamingRPC
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636
*   | google.golang.org/grpc.(*Server).handleStream
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716
*   | google.golang.org/grpc.(*Server).serveStreams.func1.1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters:

  • attempt=1
  • run=17
  • shard=2
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #132995 kv/kvserver: TestReplicateQueueRebalanceMultiStore failed [C-test-failure O-robot T-kv branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 20 '24 11:10 cockroach-teamcity

kv/kvserver.TestReplicateQueueRebalanceMultiStore failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x142fcd1]

Stack:

goroutine 1578744 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc012497820?, {0x8db15f8, 0xc0136de5d0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x68a8500?, 0xd11caa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc012497940?, {0x8db15f8, 0xc0136de750})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x68a8500?, 0xd11caa0?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8db15f8, 0xc0136de840}, 0x111e)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc012498720, {0x8db15f8?, 0xc0136de840?}, 0xc00e17f1e0?, 0xf8?)
	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0x12, 0x6, 0xc00837e310, 0xc00837e380, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc00ec2e008, {0x8db15f8, 0xc0136de750}, 0xc00837ab48, {0x7f0dfaba8728, 0xc001c1c4e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x797
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8db15f8, 0xc0136de750})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00365d830, {0x8db15f8, 0xc0136de750}, {0x7483b44, 0x1e}, 0xc00a825268)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc00ec2e008, {0x8db15c0?, 0xc001109a90?}, 0xc00837ab48, {0x7f0dfaba8700, 0xc001c1c4e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc00e1b0540, {0x8dff768, 0xc001c1c4e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x30d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x72bcd60?, 0xc00e1b0540}, {0x8de9d30, 0xc00f9d8fc0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x72bcd60, 0xc00e1b0540}, {0x8de9aa8, 0xc010d16780}, 0xc00fb45878, 0x78943a8)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00e1b0540}, {0x8de9aa8, 0xc010d16780})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x72bcd60, 0xc00e1b0540}, {0x8de9aa8, 0xc010d16780}, 0xc00fb45878?, 0xc0086bc4c0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00e1b0540}, {0x8de9aa8, 0xc010d16780})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc007d38000?, {{0x470445?}, {0x8e0c420?, 0xc00cf09e00?}}}, {0x72bcd60, 0xc00e1b0540}, {0x8de9aa8, 0xc010d16780}, 0xc00fb45878, 0xc0086bc480)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x72bcd60, 0xc00e1b0540}, {0x8de9aa8, 0xc010d16780})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7f0dfb88e348?, 0x8db15f8?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00365d830, {0x8db15f8, 0xc0136de5d0}, {0xc00e9bf860, 0x29}, 0xc00ac90aa0)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x72bcd60, 0xc00e1b0540}, {0x8de9aa8, 0xc010d16780}, 0xc00fb45878, 0xc0086bc440)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x72bcd60, 0xc00e1b0540}, {0x8de9aa8, 0xc010d16780}, 0xc00fb45878, 0xc001c1c4d0?)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc006b3a960, {0x8e0dbc0, 0xc005e596c0}, 0xc0109a98c0, 0xc018217bf0, 0xd2192e0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc006b3a960, {0x8e0dbc0, 0xc005e596c0}, 0xc0109a98c0, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 1578656
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136
Log preceding fatal error 

*   | 	GOROOT/src/runtime/signal_unix.go:881
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165
*   | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace
*   | 	github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
*   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152
*   | google.golang.org/grpc.getChainStreamHandler.func1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349
*   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140
*   | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472
*   | google.golang.org/grpc.(*Server).processStreamingRPC
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636
*   | google.golang.org/grpc.(*Server).handleStream
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716
*   | google.golang.org/grpc.(*Server).serveStreams.func1.1
*   | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:947
*   | runtime.goexit
*   | 	src/runtime/asm_amd64.s:1695
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters:

  • attempt=1
  • run=1
  • shard=2
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

  • #132995 kv/kvserver: TestReplicateQueueRebalanceMultiStore failed [C-test-failure O-robot T-kv branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity avatar Oct 21 '24 12:10 cockroach-teamcity

ALso occurred in #132962 and #132990.

mgartner avatar Oct 21 '24 13:10 mgartner

https://github.com/cockroachdb/cockroach/pull/132954 has the fix, and will merge soon.

aadityasondhi avatar Oct 21 '24 15:10 aadityasondhi

Based on the specified backports for linked PR #132954, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] avatar Oct 21 '24 16:10 blathers-crl[bot]