milvus
milvus copied to clipboard
Published
20 hours ago •
milvus-io
milvus-io
[Bug]: [benchmark][standalone] Milvus standalone panic `context deadline exceeded` after test finished
Is there an existing issue for this?
- [X] I have searched the existing issues
Environment
- Milvus version: master-20240218-91b02b5d-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.0rc19
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
argo task：inverted-corn-phbc6 test case name：test_inverted_locust_partition_key_dml_standalone
server：
[2024-02-18 09:35:14,217 - INFO - fouram]: [Base] Deploy initial state:
I0218 06:17:47.450524 414 request.go:665] Waited for 1.166458237s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/events.k8s.io/v1?timeout=32s
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inverted-corn-phbc6-2-85-6150-etcd-0 1/1 Running 0 3m54s 10.104.15.68 4am-node20 <none> <none>
inverted-corn-phbc6-2-85-6150-milvus-standalone-99985d9f6-9sn65 1/1 Running 1 (113s ago) 3m54s 10.104.18.52 4am-node25 <none> <none>
inverted-corn-phbc6-2-85-6150-minio-7856dc48bc-8z4cl 1/1 Running 0 3m54s 10.104.15.67 4am-node20 <none> <none> (base.py:257)
[2024-02-18 09:35:14,217 - INFO - fouram]: [Cmd Exe] kubectl get pods -n qa-milvus -o wide | grep -E 'STATUS|inverted-corn-phbc6-2-85-6150-milvus|inverted-corn-phbc6-2-85-6150-minio|inverted-corn-phbc6-2-85-6150-etcd|inverted-corn-phbc6-2-85-6150-pulsar|inverted-corn-phbc6-2-85-6150-kafka|inverted-corn-phbc6-2-85-6150-log|inverted-corn-phbc6-2-85-6150-tikv' (util_cmd.py:14)
[2024-02-18 09:35:24,355 - INFO - fouram]: [CliClient] pod details of release(inverted-corn-phbc6-2-85-6150):
I0218 09:35:15.466282 522 request.go:665] Waited for 1.155585929s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/events.k8s.io/v1beta1?timeout=32s
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inverted-corn-phbc6-2-85-6150-etcd-0 1/1 Running 0 3h21m 10.104.15.68 4am-node20 <none> <none>
inverted-corn-phbc6-2-85-6150-milvus-standalone-99985d9f6-9sn65 0/1 Running 2 (8s ago) 3h21m 10.104.18.52 4am-node25 <none> <none>
inverted-corn-phbc6-2-85-6150-minio-7856dc48bc-8z4cl 1/1 Running 0 3h21m 10.104.15.67 4am-node20 <none> <none>
standalone panic log:
2024-02-18 14:15:42.057(no unique labels)I20240218 06:15:42.057615 115 knowhere_config.cc:94] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 163842024-02-18 14:15:42.057(no unique labels)I20240218 06:15:42.057714 115 knowhere_config.cc:105] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 02024-02-18 14:15:42.057(no unique labels)I20240218 06:15:42.057747 115 knowhere_config.cc:53] [KNOWHERE][ShowVersion][milvus] Knowhere Version: ad535c82024-02-18 14:15:52.057(no unique labels){"level":"warn","ts":"2024-02-18T06:15:52.057Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00123a700/inverted-corn-phbc6-2-85-6150-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.255.115.37:2379: connect: connection refused\""}2024-02-18 14:16:02.058(no unique labels){"level":"warn","ts":"2024-02-18T06:16:02.057Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00123a700/inverted-corn-phbc6-2-85-6150-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.255.115.37:2379: i/o timeout\""} | 2024-02-18 14:15:42.057 | (no unique labels) | I20240218 06:15:42.057615 115 knowhere_config.cc:94] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384 | | | | 2024-02-18 14:15:42.057 | (no unique labels) | I20240218 06:15:42.057714 115 knowhere_config.cc:105] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0 | | | | 2024-02-18 14:15:42.057 | (no unique labels) | I20240218 06:15:42.057747 115 knowhere_config.cc:53] [KNOWHERE][ShowVersion][milvus] Knowhere Version: ad535c8 | | | | 2024-02-18 14:15:52.057 | (no unique labels) | {"level":"warn","ts":"2024-02-18T06:15:52.057Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00123a700/inverted-corn-phbc6-2-85-6150-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.255.115.37:2379: connect: connection refused\""} | | | | 2024-02-18 14:16:02.058 | (no unique labels) | {"level":"warn","ts":"2024-02-18T06:16:02.057Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00123a700/inverted-corn-phbc6-2-85-6150-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.255.115.37:2379: i/o timeout\""} |
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
2024-02-18 14:15:42.057 | (no unique labels) | I20240218 06:15:42.057615 115 knowhere_config.cc:94] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384 |
| | 2024-02-18 14:15:42.057 | (no unique labels) | I20240218 06:15:42.057714 115 knowhere_config.cc:105] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0 |
| | 2024-02-18 14:15:42.057 | (no unique labels) | I20240218 06:15:42.057747 115 knowhere_config.cc:53] [KNOWHERE][ShowVersion][milvus] Knowhere Version: ad535c8 |
| | 2024-02-18 14:15:52.057 | (no unique labels) | {"level":"warn","ts":"2024-02-18T06:15:52.057Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00123a700/inverted-corn-phbc6-2-85-6150-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.255.115.37:2379: connect: connection refused\""} |
| | 2024-02-18 14:16:02.058 | (no unique labels) | {"level":"warn","ts":"2024-02-18T06:16:02.057Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00123a700/inverted-corn-phbc6-2-85-6150-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.255.115.37:2379: i/o timeout\""} |
2024-02-18 14:16:02.060stderrpanic: context deadline exceeded | | | 2024-02-18 14:16:02.060 | stderr | panic: context deadline exceeded |
| | 2024-02-18 14:16:02.060 | stderr | panic: context deadline exceeded |
2024-02-18 14:16:02.060(no unique labels)2024-02-18 14:16:02.060(no unique labels)goroutine 254 [running]:2024-02-18 14:16:02.060(no unique labels)panic({0x4f1cd60, 0xc0016d8858})2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/panic.go:987 +0x3bb fp=0xc001567f70 sp=0xc001567eb0 pc=0x1b34d9b2024-02-18 14:16:02.060(no unique labels)github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1()2024-02-18 14:16:02.060(no unique labels) /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:111 +0x105 fp=0xc001567fe0 sp=0xc001567f70 pc=0x47aa8452024-02-18 14:16:02.060(no unique labels)runtime.goexit()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001567fe8 sp=0xc001567fe0 pc=0x1b6ebe12024-02-18 14:16:02.060(no unique labels)created by github.com/milvus-io/milvus/cmd/roles.runComponent[...]2024-02-18 14:16:02.060(no unique labels) /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:102 +0x1582024-02-18 14:16:02.060(no unique labels)2024-02-18 14:16:02.060(no unique labels)goroutine 1 [semacquire, 1 minutes]:2024-02-18 14:16:02.060(no unique labels)runtime.gopark(0x5869bd0?, 0xc001646278?, 0x0?, 0x12?, 0xc0015c2180?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001d5b730 sp=0xc001d5b710 pc=0x1b383162024-02-18 14:16:02.060(no unique labels)runtime.goparkunlock(...)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:3872024-02-18 14:16:02.060(no unique labels)runtime.semacquire1(0xc00069d798, 0x20?, 0x1, 0x0, 0x1?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc001d5b798 sp=0xc001d5b730 pc=0x1b4a0ef2024-02-18 14:16:02.060(no unique labels)sync.runtime_Semacquire(0x5411a98?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc001d5b7d0 sp=0xc001d5b798 pc=0x1b6a1672024-02-18 14:16:02.060(no unique labels)sync.(*WaitGroup).Wait(0xc001540130?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc001d5b7f8 sp=0xc001d5b7d0 pc=0x1b9148b2024-02-18 14:16:02.060(no unique labels)github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc0004dafa0)2024-02-18 14:16:02.060(no unique labels) /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:382 +0x9e5 fp=0xc001d5bc78 sp=0xc001d5b7f8 pc=0x47a9c652024-02-18 14:16:02.060(no unique labels)github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0x0?, {0xc000222150?, 0x3, 0x3}, 0xc0004d8240)2024-02-18 14:16:02.060(no unique labels) /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:47 +0x2e5 fp=0xc001d5bd48 sp=0xc001d5bc78 pc=0x47b4ac52024-02-18 14:16:02.060(no unique labels)github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000222150?, 0x3, 0x3})2024-02-18 14:16:02.060(no unique labels) /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x20e fp=0xc001d5bdc0 sp=0xc001d5bd48 pc=0x47b474e2024-02-18 14:16:02.060(no unique labels)main.main()2024-02-18 14:16:02.060(no unique labels) /go/src/github.com/milvus-io/milvus/cmd/main.go:95 +0x3e5 fp=0xc001d5bf80 sp=0xc001d5bdc0 pc=0x47b94852024-02-18 14:16:02.060(no unique labels)runtime.main()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc001d5bfe0 sp=0xc001d5bf80 pc=0x1b37ee72024-02-18 14:16:02.060(no unique labels)runtime.goexit()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001d5bfe8 sp=0xc001d5bfe0 pc=0x1b6ebe12024-02-18 14:16:02.060(no unique labels)2024-02-18 14:16:02.060(no unique labels)goroutine 2 [force gc (idle), 1 minutes]:2024-02-18 14:16:02.060(no unique labels)runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000166fb0 sp=0xc000166f90 pc=0x1b383162024-02-18 14:16:02.060(no unique labels)runtime.goparkunlock(...)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:3872024-02-18 14:16:02.060(no unique labels)runtime.forcegchelper()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000166fe0 sp=0xc000166fb0 pc=0x1b381502024-02-18 14:16:02.060(no unique labels)runtime.goexit()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000166fe8 sp=0xc000166fe0 pc=0x1b6ebe12024-02-18 14:16:02.060(no unique labels)created by runtime.init.62024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:293 +0x252024-02-18 14:16:02.060(no unique labels)2024-02-18 14:16:02.060(no unique labels)goroutine 18 [GC sweep wait]:2024-02-18 14:16:02.060(no unique labels)runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000162780 sp=0xc000162760 pc=0x1b383162024-02-18 14:16:02.060(no unique labels)runtime.goparkunlock(...)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:3872024-02-18 14:16:02.060(no unique labels)runtime.bgsweep(0x0?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0001627c8 sp=0xc000162780 pc=0x1b209de2024-02-18 14:16:02.060(no unique labels)runtime.gcenable.func1()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0001627e0 sp=0xc0001627c8 pc=0x1b15aa62024-02-18 14:16:02.060(no unique labels)runtime.goexit()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001627e8 sp=0xc0001627e0 pc=0x1b6ebe12024-02-18 14:16:02.060(no unique labels)created by runtime.gcenable2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgc.go:178 +0x6b2024-02-18 14:16:02.060(no unique labels)2024-02-18 14:16:02.060(no unique labels)goroutine 19 [GC scavenge wait]:2024-02-18 14:16:02.060(no unique labels)runtime.gopark(0xc00020e000?, 0x584f808?, 0x0?, 0x0?, 0x0?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000162f70 sp=0xc000162f50 pc=0x1b383162024-02-18 14:16:02.060(no unique labels)runtime.goparkunlock(...)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:3872024-02-18 14:16:02.060(no unique labels)runtime.(*scavengerState).park(0x7848960)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000162fa0 sp=0xc000162f70 pc=0x1b1e8132024-02-18 14:16:02.060(no unique labels)runtime.bgscavenge(0x0?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000162fc8 sp=0xc000162fa0 pc=0x1b1ee252024-02-18 14:16:02.060(no unique labels)runtime.gcenable.func2()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000162fe0 sp=0xc000162fc8 pc=0x1b15a462024-02-18 14:16:02.060(no unique labels)runtime.goexit()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000162fe8 sp=0xc000162fe0 pc=0x1b6ebe12024-02-18 14:16:02.060(no unique labels)created by runtime.gcenable2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgc.go:179 +0xaa2024-02-18 14:16:02.060(no unique labels)2024-02-18 14:16:02.060(no unique labels)goroutine 20 [finalizer wait]:2024-02-18 14:16:02.060(no unique labels)runtime.gopark(0x0?, 0x5411b28?, 0xc0?, 0xe4?, 0x1000000010?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000867e28 sp=0xc000867e08 pc=0x1b383162024-02-18 14:16:02.060(no unique labels)runtime.runfinq()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000867fe0 sp=0xc000867e28 pc=0x1b14a672024-02-18 14:16:02.060(no unique labels)runtime.goexit()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000867fe8 sp=0xc000867fe0 pc=0x1b6ebe12024-02-18 14:16:02.060(no unique labels)created by runtime.createfing2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mfinal.go:163 +0x452024-02-18 14:16:02.060(no unique labels)2024-02-18 14:16:02.060(no unique labels)goroutine 21 [GC worker (idle), 1 minutes]:2024-02-18 14:16:02.060(no unique labels)runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000163750 sp=0xc000163730 pc=0x1b383162024-02-18 14:16:02.060(no unique labels)runtime.gcBgMarkWorker()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001637e0 sp=0xc000163750 pc=0x1b178112024-02-18 14:16:02.060(no unique labels)runtime.goexit()2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001637e8 sp=0xc0001637e0 pc=0x1b6ebe12024-02-18 14:16:02.060(no unique labels)created by runtime.gcBgMarkStartWorkers2024-02-18 14:16:02.060(no unique labels) /usr/local/go/src/runtime/mgc.go:1199 +0x252024-02-18 14:16:02.060(no unique labels)2024-02-18 14:16:02.060(no unique labels)goroutine 34 [GC worker (idle), 1 minutes]:2024-02-18 14:16:02.060(no unique labels)runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) | | | 2024-02-18 14:16:02.060 | (no unique labels) | | | | | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 254 [running]: | | | | 2024-02-18 14:16:02.060 | (no unique labels) | panic({0x4f1cd60, 0xc0016d8858}) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/panic.go:987 +0x3bb fp=0xc001567f70 sp=0xc001567eb0 pc=0x1b34d9b | | | | 2024-02-18 14:16:02.060 | (no unique labels) | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:111 +0x105 fp=0xc001567fe0 sp=0xc001567f70 pc=0x47aa845 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001567fe8 sp=0xc001567fe0 pc=0x1b6ebe1 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:102 +0x158 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | | | | | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 1 [semacquire, 1 minutes]: | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x5869bd0?, 0xc001646278?, 0x0?, 0x12?, 0xc0015c2180?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001d5b730 sp=0xc001d5b710 pc=0x1b38316 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goparkunlock(...) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:387 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.semacquire1(0xc00069d798, 0x20?, 0x1, 0x0, 0x1?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc001d5b798 sp=0xc001d5b730 pc=0x1b4a0ef | | | | 2024-02-18 14:16:02.060 | (no unique labels) | sync.runtime_Semacquire(0x5411a98?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc001d5b7d0 sp=0xc001d5b798 pc=0x1b6a167 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | sync.(*WaitGroup).Wait(0xc001540130?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc001d5b7f8 sp=0xc001d5b7d0 pc=0x1b9148b | | | | 2024-02-18 14:16:02.060 | (no unique labels) | github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc0004dafa0) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:382 +0x9e5 fp=0xc001d5bc78 sp=0xc001d5b7f8 pc=0x47a9c65 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0x0?, {0xc000222150?, 0x3, 0x3}, 0xc0004d8240) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:47 +0x2e5 fp=0xc001d5bd48 sp=0xc001d5bc78 pc=0x47b4ac5 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000222150?, 0x3, 0x3}) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x20e fp=0xc001d5bdc0 sp=0xc001d5bd48 pc=0x47b474e | | | | 2024-02-18 14:16:02.060 | (no unique labels) | main.main() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/main.go:95 +0x3e5 fp=0xc001d5bf80 sp=0xc001d5bdc0 pc=0x47b9485 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.main() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc001d5bfe0 sp=0xc001d5bf80 pc=0x1b37ee7 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001d5bfe8 sp=0xc001d5bfe0 pc=0x1b6ebe1 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | | | | | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 2 [force gc (idle), 1 minutes]: | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000166fb0 sp=0xc000166f90 pc=0x1b38316 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goparkunlock(...) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:387 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.forcegchelper() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000166fe0 sp=0xc000166fb0 pc=0x1b38150 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000166fe8 sp=0xc000166fe0 pc=0x1b6ebe1 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.init.6 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:293 +0x25 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | | | | | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 18 [GC sweep wait]: | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000162780 sp=0xc000162760 pc=0x1b38316 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goparkunlock(...) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:387 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.bgsweep(0x0?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0001627c8 sp=0xc000162780 pc=0x1b209de | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gcenable.func1() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0001627e0 sp=0xc0001627c8 pc=0x1b15aa6 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001627e8 sp=0xc0001627e0 pc=0x1b6ebe1 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.gcenable | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:178 +0x6b | | | | 2024-02-18 14:16:02.060 | (no unique labels) | | | | | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 19 [GC scavenge wait]: | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0xc00020e000?, 0x584f808?, 0x0?, 0x0?, 0x0?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000162f70 sp=0xc000162f50 pc=0x1b38316 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goparkunlock(...) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:387 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.(*scavengerState).park(0x7848960) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000162fa0 sp=0xc000162f70 pc=0x1b1e813 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.bgscavenge(0x0?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000162fc8 sp=0xc000162fa0 pc=0x1b1ee25 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gcenable.func2() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000162fe0 sp=0xc000162fc8 pc=0x1b15a46 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000162fe8 sp=0xc000162fe0 pc=0x1b6ebe1 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.gcenable | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:179 +0xaa | | | | 2024-02-18 14:16:02.060 | (no unique labels) | | | | | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 20 [finalizer wait]: | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x0?, 0x5411b28?, 0xc0?, 0xe4?, 0x1000000010?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000867e28 sp=0xc000867e08 pc=0x1b38316 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.runfinq() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000867fe0 sp=0xc000867e28 pc=0x1b14a67 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000867fe8 sp=0xc000867fe0 pc=0x1b6ebe1 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.createfing | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mfinal.go:163 +0x45 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | | | | | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 21 [GC worker (idle), 1 minutes]: | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000163750 sp=0xc000163730 pc=0x1b38316 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gcBgMarkWorker() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001637e0 sp=0xc000163750 pc=0x1b17811 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001637e8 sp=0xc0001637e0 pc=0x1b6ebe1 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.gcBgMarkStartWorkers | | | | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:1199 +0x25 | | | | 2024-02-18 14:16:02.060 | (no unique labels) | | | | | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 34 [GC worker (idle), 1 minutes]: | | | | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
| | 2024-02-18 14:16:02.060 | (no unique labels) | |
| | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 254 [running]: |
| | 2024-02-18 14:16:02.060 | (no unique labels) | panic({0x4f1cd60, 0xc0016d8858}) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/panic.go:987 +0x3bb fp=0xc001567f70 sp=0xc001567eb0 pc=0x1b34d9b |
| | 2024-02-18 14:16:02.060 | (no unique labels) | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:111 +0x105 fp=0xc001567fe0 sp=0xc001567f70 pc=0x47aa845 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001567fe8 sp=0xc001567fe0 pc=0x1b6ebe1 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:102 +0x158 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | |
| | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 1 [semacquire, 1 minutes]: |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x5869bd0?, 0xc001646278?, 0x0?, 0x12?, 0xc0015c2180?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc001d5b730 sp=0xc001d5b710 pc=0x1b38316 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goparkunlock(...) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:387 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.semacquire1(0xc00069d798, 0x20?, 0x1, 0x0, 0x1?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc001d5b798 sp=0xc001d5b730 pc=0x1b4a0ef |
| | 2024-02-18 14:16:02.060 | (no unique labels) | sync.runtime_Semacquire(0x5411a98?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc001d5b7d0 sp=0xc001d5b798 pc=0x1b6a167 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | sync.(*WaitGroup).Wait(0xc001540130?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc001d5b7f8 sp=0xc001d5b7d0 pc=0x1b9148b |
| | 2024-02-18 14:16:02.060 | (no unique labels) | github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc0004dafa0) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:382 +0x9e5 fp=0xc001d5bc78 sp=0xc001d5b7f8 pc=0x47a9c65 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0x0?, {0xc000222150?, 0x3, 0x3}, 0xc0004d8240) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:47 +0x2e5 fp=0xc001d5bd48 sp=0xc001d5bc78 pc=0x47b4ac5 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000222150?, 0x3, 0x3}) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x20e fp=0xc001d5bdc0 sp=0xc001d5bd48 pc=0x47b474e |
| | 2024-02-18 14:16:02.060 | (no unique labels) | main.main() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /go/src/github.com/milvus-io/milvus/cmd/main.go:95 +0x3e5 fp=0xc001d5bf80 sp=0xc001d5bdc0 pc=0x47b9485 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.main() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc001d5bfe0 sp=0xc001d5bf80 pc=0x1b37ee7 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc001d5bfe8 sp=0xc001d5bfe0 pc=0x1b6ebe1 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | |
| | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 2 [force gc (idle), 1 minutes]: |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000166fb0 sp=0xc000166f90 pc=0x1b38316 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goparkunlock(...) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:387 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.forcegchelper() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000166fe0 sp=0xc000166fb0 pc=0x1b38150 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000166fe8 sp=0xc000166fe0 pc=0x1b6ebe1 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.init.6 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:293 +0x25 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | |
| | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 18 [GC sweep wait]: |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000162780 sp=0xc000162760 pc=0x1b38316 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goparkunlock(...) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:387 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.bgsweep(0x0?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0001627c8 sp=0xc000162780 pc=0x1b209de |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gcenable.func1() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0001627e0 sp=0xc0001627c8 pc=0x1b15aa6 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001627e8 sp=0xc0001627e0 pc=0x1b6ebe1 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.gcenable |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:178 +0x6b |
| | 2024-02-18 14:16:02.060 | (no unique labels) | |
| | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 19 [GC scavenge wait]: |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0xc00020e000?, 0x584f808?, 0x0?, 0x0?, 0x0?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000162f70 sp=0xc000162f50 pc=0x1b38316 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goparkunlock(...) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:387 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.(*scavengerState).park(0x7848960) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000162fa0 sp=0xc000162f70 pc=0x1b1e813 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.bgscavenge(0x0?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000162fc8 sp=0xc000162fa0 pc=0x1b1ee25 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gcenable.func2() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000162fe0 sp=0xc000162fc8 pc=0x1b15a46 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000162fe8 sp=0xc000162fe0 pc=0x1b6ebe1 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.gcenable |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:179 +0xaa |
| | 2024-02-18 14:16:02.060 | (no unique labels) | |
| | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 20 [finalizer wait]: |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x0?, 0x5411b28?, 0xc0?, 0xe4?, 0x1000000010?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000867e28 sp=0xc000867e08 pc=0x1b38316 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.runfinq() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000867fe0 sp=0xc000867e28 pc=0x1b14a67 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000867fe8 sp=0xc000867fe0 pc=0x1b6ebe1 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.createfing |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mfinal.go:163 +0x45 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | |
| | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 21 [GC worker (idle), 1 minutes]: |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000163750 sp=0xc000163730 pc=0x1b38316 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gcBgMarkWorker() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001637e0 sp=0xc000163750 pc=0x1b17811 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.goexit() |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001637e8 sp=0xc0001637e0 pc=0x1b6ebe1 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | created by runtime.gcBgMarkStartWorkers |
| | 2024-02-18 14:16:02.060 | (no unique labels) | /usr/local/go/src/runtime/mgc.go:1199 +0x25 |
| | 2024-02-18 14:16:02.060 | (no unique labels) | |
| | 2024-02-18 14:16:02.060 | (no unique labels) | goroutine 34 [GC worker (idle), 1 minutes]: |
| | 2024-02-18 14:16:02.060 | (no unique labels) | runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
client pod name: inverted-corn-phbc6-1008373906
Expected Behavior
No response
Steps To Reproduce
concurrent test and calculation of RT and QPS
:purpose: `partition_key: scalar enable partition_key(num_partitions=128)`
verify concurrent DML scenario which
scalar `id`(pk) & `int64_1` created INVERTED index and enable partition_key on `int64_1` field
:test steps:
1. create collection with fields:
'float_vector': 128dim,
'int64_1': is_partition_key
2. build indexes:
IVF_FLAT: 'float_vector'
INVERTED: 'id', 'int64_1'
3. insert 5 million data
4. flush collection
5. build indexes again using the same params
6. load collection
7. concurrent request:
- insert
- delete
- flush
- release
Milvus Log
No response
Anything else?
test result:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'standalone',
'config_name': 'standalone_8c16m',
'config': {'standalone': {'resources': {'limits': {'cpu': '8.0',
'memory': '16Gi'},
'requests': {'cpu': '5.0',
'memory': '9Gi'}}},
'cluster': {'enabled': False},
'etcd': {'replicaCount': 1,
'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'minio': {'mode': 'standalone',
'metrics': {'podMonitor': {'enabled': True}}},
'pulsar': {'enabled': False},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': 'master-20240218-91b02b5d-amd64'}}},
'host': 'inverted-corn-phbc6-2-85-6150-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_inverted_locust_partition_key_dml_standalone',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'dim': 128,
'scalars_index': {'id': {'index_type': 'INVERTED'},
'int64_1': {'index_type': 'INVERTED'}},
'scalars_params': {'int64_1': {'params': {'is_partition_key': True}}},
'dataset_name': 'sift',
'dataset_size': 5000000,
'ni_per': 50000},
'collection_params': {'other_fields': ['int64_1'],
'shards_num': 2,
'num_partitions': 128},
'resource_groups_params': {'reset': False},
'database_user_params': {'reset_rbac': False,
'reset_db': False},
'index_params': {'index_type': 'IVF_FLAT',
'index_param': {'nlist': 1024}},
'concurrent_params': {'concurrent_number': 20,
'during_time': '3h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'insert',
'weight': 1,
'params': {'nb': 10,
'timeout': 180,
'random_id': True,
'random_vector': True,
'varchar_filled': False,
'start_id': 0}},
{'type': 'delete',
'weight': 1,
'params': {'expr': '',
'delete_length': 9,
'timeout': 30}},
{'type': 'flush',
'weight': 1,
'params': {'timeout': 180}},
{'type': 'release',
'weight': 1,
'params': {'timeout': 30}}]},
'run_id': 2024021868393359,
'datetime': '2024-02-18 06:13:59.518760',
'client_version': '2.4.0'},
'result': {'test_result': {'index': {'RT': 793.5038,
'id': {'RT': 1.0188},
'int64_1': {'RT': 0.5153}},
'insert': {'total_time': 143.0136,
'VPS': 34961.7099,
'batch_time': 1.4301,
'batch': 50000},
'flush': {'RT': 15.2762},
'load': {'RT': 8.6608},
'Locust': {'Aggregated': {'Requests': 22598,
'Fails': 427,
'RPS': 2.11,
'fail_s': 0.02,
'RT_max': 195975.14,
'RT_avg': 9443.15,
'TP50': 62,
'TP99': 181000.0},
'delete': {'Requests': 5654,
'Fails': 0,
'RPS': 0.53,
'fail_s': 0.0,
'RT_max': 284.16,
'RT_avg': 12.79,
'TP50': 3,
'TP99': 92},
'flush': {'Requests': 5638,
'Fails': 427,
'RPS': 0.53,
'fail_s': 0.08,
'RT_max': 195975.14,
'RT_avg': 33183.86,
'TP50': 14000.0,
'TP99': 183000.0},
'insert': {'Requests': 5683,
'Fails': 0,
'RPS': 0.53,
'fail_s': 0.0,
'RT_max': 40031.09,
'RT_avg': 4604.9,
'TP50': 2900.0,
'TP99': 29000.0},
'release': {'Requests': 5623,
'Fails': 0,
'RPS': 0.52,
'fail_s': 0.0,
'RT_max': 808.54,
'RT_avg': 11.32,
'TP50': 2,
'TP99': 87}}}}}
From log description, it panic when milvus start, connect to etcd failed and retry for many times. maybe env problem, keep watching and check whether reproduce next time
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
panic: context deadline exceeded
goroutine 261 gp=0xc0010a8380 m=5 mp=0xc0000de008 [running]: panic({0x50f18a0?, 0x7ff8e60?}) /usr/lib/go-1.22/src/runtime/panic.go:779 +0x158 fp=0xc001e03f70 sp=0xc001e03ec0 pc=0x1b3be98 github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() /users/sharmara/milvus/cmd/roles/roles.go:112 +0x108 fp=0xc001e03fe0 sp=0xc001e03f70 pc=0x4aa42a8 runtime.goexit({}) /usr/lib/go-1.22/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc001e03fe8 sp=0xc001e03fe0 pc=0x1b7ac81 created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] in goroutine 1 /users/sharmara/milvus/cmd/roles/roles.go:103 +0x129 Aborted
"Everytime I am trying to start milvus using the command, ./bin/milvus run standalone and I am getting "context deadline exceeded"
Any solution for the same?
@RajnandiniG most 'context deadline exceeded' errors just indicate something timeout, so we need the full logs for investigation. please file a new issue with milvus logs attached.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.