[Bug]: context deadline exceeded when inserting

[smallshallot]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:2.4.0-rc.1
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

我在用langchain-chatchat框架的init_database.py脚本对pdf数据进行建库的过程中，standalone服务宕机，etcd和minio服务正常

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

milvus-standalone | [2024/05/15 22:07:12.433 +00:00] [WARN] [datacoord/compaction.go:161] ["unable to alloc timestamp"] [error="context deadline exceeded"] 21031 milvus-standalone | [2024/05/15 22:07:12.433 +00:00] [WARN] [datacoord/compaction.go:145] ["fail to check result"] [error="context deadline exceeded"] 21032 milvus-standalone | [2024/05/15 22:07:12.444 +00:00] [WARN] [grpcclient/client.go:403] ["fail to grpc call because of unknown error"] [clientRole=querynode-27] [error="context deadline exceeded"] 21033 milvus-standalone | [2024/05/15 22:07:12.444 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="context deadline exceeded"] 21034 milvus-standalone | [2024/05/15 22:07:12.445 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = DeadlineExceeded desc = context de adline exceeded"] 21035 milvus-standalone | [2024/05/15 22:07:12.445 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] 21036 milvus-standalone | {"level":"warn","ts":"2024-05-15T22:07:12.424Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoin ts://0xc000709dc0/etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} 21037 milvus-standalone | [2024/05/15 22:07:12.453 +00:00] [ERROR] [config/refresher.go:73] ["can not pull configs"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*ref resher).refreshPeriodically\n\t/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:73"] 21038 milvus-standalone | [2024/05/15 22:07:12.460 +00:00] [WARN] [grpcclient/client.go:557] ["ClientBase Call grpc call get error"] [role=querynode-27] [address=172.18.0.4:21123] [error="stack trace: /go/ src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/mi lvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/grpcclient.(*Clien tBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/go/src/ github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:309 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetDataDistribution\n/go/src/github.com/milv us-io/milvus/internal/querycoordv2/session/cluster.go:208 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution.func1\n/go/src/github.com/milvus-io/milvus/inter nal/querycoordv2/session/cluster.go:271 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:203 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:226 github.com/milvus-io/ milvus/internal/querycoordv2/dist.(*distHandler).getDistribution\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:70 github.com/milvus-io/milvus/internal/querycoordv2/di st.(*distHandler).start: context deadline exceeded"] [errorVerbose="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: con text deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/go/src/github.com/milvus-io/milvus/interna l/util/grpcclient/client.go:556\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | \t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:90\n | github.com/mil vus-io/milvus/internal/distributed/querynode/client.(*Client).GetDataDistribution\n | \t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:309\n | github.com/milvus- io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution.func1\n | \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:208\n | github.com/milvus-io/milv us/internal/querycoordv2/session.(*QueryCluster).send\n | \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:271\n | github.com/milvus-io/milvus/internal/querycoordv2/ses sion.(*QueryCluster).GetDataDistribution\n | \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:203\n | github.com/milvus-io/milvus/internal/querycoordv2/dist.(*distHandl er).getDistribution\n | \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:226\n | github.com/milvus-io/milvus/internal/querycoordv2/dist.(*distHandler).start\n | \t/g o/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:70\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) stack trace: /go/src/github.com/milvus -io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/milvus-io/milvus/in ternal/util/grpcclient.(*ClientBase[...]).Call\n | /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).R eCall\n | /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | /go/src/githu b.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:309 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetDataDistribution\n | /go/src/github.com/milvu s-io/milvus/internal/querycoordv2/session/cluster.go:208 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution.func1\n | /go/src/github.com/milvus-io/milvus/in ternal/querycoordv2/session/cluster.go:271 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | /go/src/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster .go:203 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution\n | /go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:226 github.com /milvus-io/milvus/internal/querycoordv2/dist.(*distHandler).getDistribution\n | /go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:70 github.com/milvus-io/milvus/internal/ querycoordv2/dist.(*distHandler).start\nWraps: (3) context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) context.deadlineExceededError"] 21039 milvus-standalone | [2024/05/15 22:07:12.461 +00:00] [WARN] [runtime/asm_amd64.s:1598] ["failed to get data distribution"] [nodeID=27] [times=0] [lastHeartbeat=2024/05/15 21:54:26.927 +00:00] [error= "stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go :556 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/ grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcC all[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:309 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetDataDistribution\n/go/s rc/github.com/milvus-io/milvus/internal/querycoordv2/session/cluster.go:208 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution.func1\n/go/src/github.com/milv us-io/milvus/internal/querycoordv2/session/cluster.go:271 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/sess ion/cluster.go:203 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetDataDistribution\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:226 git hub.com/milvus-io/milvus/internal/querycoordv2/dist.(*distHandler).getDistribution\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/dist/dist_handler.go:70 github.com/milvus-io/milvus/intern al/querycoordv2/dist.(*distHandler).start: context deadline exceeded"] milvus-standalone | [2024/05/15 22:07:12.468 +00:00] [WARN] [proxy/channels_time_ticker.go:96] ["Proxy channelsTimeTickerImpl failed to get ts from tso"] [error="syncTimestamp Failed:stack trace: /go /src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/m ilvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/grpcclient.(*Clie ntBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/sr c/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:308 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp\n/go/src/github.com/milvus- io/milvus/internal/proxy/timestamp.go:60 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc\n/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:85 github.com/milvus-io /milvus/internal/proxy.(*timestampAllocator).AllocOne\n/go/src/github.com/milvus-io/milvus/internal/proxy/channels_time_ticker.go:94 github.com/milvus-io/milvus/internal/proxy.(*channelsTimeTickerImpl ).tick\n/go/src/github.com/milvus-io/milvus/internal/proxy/channels_time_ticker.go:166 github.com/milvus-io/milvus/internal/proxy.(*channelsTimeTickerImpl).tickLoop\n/usr/local/go/src/runtime/asm_amd6 4.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded"] 21042 milvus-standalone | [2024/05/15 22:07:12.468 +00:00] [WARN] [proxy/channels_time_ticker.go:168] [channelsTimeTickerImpl.tickLoop] [error="syncTimestamp Failed:stack trace: /go/src/github.com/milvus-i o/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/milvus-io/milvus/internal /util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go /src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/m ilvus/internal/distributed/rootcoord/client/client.go:308 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp\n/go/src/github.com/milvus-io/milvus/internal/proxy /timestamp.go:60 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc\n/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:85 github.com/milvus-io/milvus/internal/proxy.( *timestampAllocator).AllocOne\n/go/src/github.com/milvus-io/milvus/internal/proxy/channels_time_ticker.go:94 github.com/milvus-io/milvus/internal/proxy.(*channelsTimeTickerImpl).tick\n/go/src/github.c om/milvus-io/milvus/internal/proxy/channels_time_ticker.go:166 github.com/milvus-io/milvus/internal/proxy.(*channelsTimeTickerImpl).tickLoop\n/usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded"]

Anything else?

No response

[github-actions[bot]]

The title and description of this issue contains Chinese. Please use English to describe your issue.

[yanliang567]

looks like the etcd is not healthy or not fast enough, please allocate more resource for etcd and make sure etcd is running on ssd volumes. @smallshallot /assign @smallshallot /unassign

[smallshallot]

looks like the etcd is not healthy or not fast enough, please allocate more resource for etcd and make sure etcd is running on ssd volumes. @smallshallot /assign @smallshallot /unassign

Thank you. I'll try again

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.