milvus
milvus copied to clipboard
Published
20 hours ago •
milvus-io
milvus-io
[Bug]: [benchmark][cluster]Milvus recall search report an error"StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded"
Is there an existing issue for this?
- [X] I have searched the existing issues
Environment
- Milvus version:2.1.0-20220716-48b9b941
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus 2.1.0dev98
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
server-instance fouram-cron-1658160000-5 server-configmap server-cluster-8c16m client-configmap client-acc-glove-ivf-flat
fouram-cron-1658160000-5-etcd-0 1/1 Running 0 5m51s 10.104.6.252 4am-node13 <none> <none>
fouram-cron-1658160000-5-etcd-1 1/1 Running 0 5m50s 10.104.1.216 4am-node10 <none> <none>
fouram-cron-1658160000-5-etcd-2 1/1 Running 0 5m50s 10.104.4.114 4am-node11 <none> <none>
fouram-cron-1658160000-5-milvus-datacoord-587495ff7c-jlnxs 1/1 Running 1 (110s ago) 5m51s 10.104.5.192 4am-node12 <none> <none>
fouram-cron-1658160000-5-milvus-datanode-78cc965ff6-nwdjw 1/1 Running 1 (2m19s ago) 5m51s 10.104.5.193 4am-node12 <none> <none>
fouram-cron-1658160000-5-milvus-indexcoord-64cf9b9c44-8jlws 1/1 Running 1 (110s ago) 5m51s 10.104.5.191 4am-node12 <none> <none>
fouram-cron-1658160000-5-milvus-indexnode-76c4674469-hpxsd 1/1 Running 0 5m51s 10.104.1.208 4am-node10 <none> <none>
fouram-cron-1658160000-5-milvus-proxy-6d96956b5b-tjlxb 1/1 Running 1 (109s ago) 5m51s 10.104.5.195 4am-node12 <none> <none>
fouram-cron-1658160000-5-milvus-querycoord-649d4b8579-98ms6 1/1 Running 1 (109s ago) 5m51s 10.104.6.249 4am-node13 <none> <none>
fouram-cron-1658160000-5-milvus-querynode-9dd464d95-92bdr 1/1 Running 0 5m51s 10.104.9.49 4am-node14 <none> <none>
fouram-cron-1658160000-5-milvus-rootcoord-cfbf48f97-rwn69 1/1 Running 1 (2m19s ago) 5m51s 10.104.5.194 4am-node12 <none> <none>
fouram-cron-1658160000-5-minio-0 1/1 Running 0 5m51s 10.104.5.187 4am-node12 <none> <none>
fouram-cron-1658160000-5-minio-1 1/1 Running 0 5m51s 10.104.6.246 4am-node13 <none> <none>
fouram-cron-1658160000-5-minio-2 1/1 Running 0 5m51s 10.104.9.48 4am-node14 <none> <none>
fouram-cron-1658160000-5-minio-3 1/1 Running 0 5m51s 10.104.1.210 4am-node10 <none> <none>
fouram-cron-1658160000-5-pulsar-bookie-0 1/1 Running 0 5m51s 10.104.5.211 4am-node12 <none> <none>
fouram-cron-1658160000-5-pulsar-bookie-1 1/1 Running 0 5m51s 10.104.6.3 4am-node13 <none> <none>
fouram-cron-1658160000-5-pulsar-bookie-2 1/1 Running 0 5m50s 10.104.1.219 4am-node10 <none> <none>
fouram-cron-1658160000-5-pulsar-bookie-init-qkwsm 0/1 Completed 0 5m51s 10.104.5.190 4am-node12 <none> <none>
fouram-cron-1658160000-5-pulsar-broker-0 1/1 Running 0 5m51s 10.104.6.247 4am-node13 <none> <none>
fouram-cron-1658160000-5-pulsar-proxy-0 1/1 Running 0 5m51s 10.104.5.189 4am-node12 <none> <none>
fouram-cron-1658160000-5-pulsar-pulsar-init-jn6jm 0/1 Completed 0 5m51s 10.104.1.209 4am-node10 <none> <none>
fouram-cron-1658160000-5-pulsar-recovery-0 1/1 Running 0 5m51s 10.104.5.188 4am-node12 <none> <none>
fouram-cron-1658160000-5-pulsar-zookeeper-0 1/1 Running 0 5m51s 10.104.5.209 4am-node12 <none> <none>
fouram-cron-1658160000-5-pulsar-zookeeper-1 1/1 Running 0 5m7s 10.104.9.56 4am-node14 <none> <none>
fouram-cron-1658160000-5-pulsar-zookeeper-2 1/1 Running 0 4m6s 10.104.4.138 4am-node11 <none> <none>
client pod
[2022-07-18 17:35:34,359] [ ERROR] - RPC error: [search], <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658165734.357666421","description":"Error received from peer ipv4:10.255.3.187:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>, <Time:{'RPC start': '2022-07-18 17:30:34.078214', 'RPC error': '2022-07-18 17:35:34.359010'}> (pymilvus.decorators:94)
[2022-07-18 17:35:34,361] [ ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 446, in search
return self._execute_search_requests(requests, timeout, **_kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 410, in _execute_search_requests
raise pre_err
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 398, in _execute_search_requests
response = ft.result()
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 744, in result
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658165734.357666421","description":"Error received from peer ipv4:10.255.3.187:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 95, in run_suite
result = runner.run_case(case_metric, **case)
File "/src/milvus_benchmark/runners/accuracy.py", line 283, in run_case
self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"],
File "/src/milvus_benchmark/client.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 337, in query
result = self._milvus.search(tmp_collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 844, in search
return handler.search(collection_name, data, anns_field, param, limit, expression,
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 95, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 91, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 56, in handler
raise MilvusException(Status.UNEXPECTED_ERROR, str(e))
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658165734.357666421","description":"Error received from peer ipv4:10.255.3.187:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
No response
Anything else?
{
"config.yaml": "ann_accuracy:
collections:
-
milvus:
cache_config.cpu_cache_capacity: 16GB
engine_config.use_blas_threshold: 1100
server:
cpus: 12
source_file: /test/milvus/ann_hdf5/glove-200-angular.hdf5
collection_name: glove_200_angular
index_types: ['ivf_flat']
index_params:
nlist: [1024]
top_ks: [10]
nqs: [10000]
search_params:
nprobe: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
```"
}
This problem has been fixed before but has returned：#17434 The error is currently reported at nprobe 512, which means that the error occurs when nprobe is larger
server-instance fouram-cron-1658246400-5 server-configmap server-cluster-8c16m client-configmap client-acc-glove-ivf-flat
pymilvus 2.1.0dev103 2.1.0-20220719-69aea00c
fouram-cron-1658246400-5-etcd-0 1/1 Running 0 13m 10.104.4.239 4am-node11 <none> <none>
fouram-cron-1658246400-5-etcd-1 1/1 Running 0 13m 10.104.1.114 4am-node10 <none> <none>
fouram-cron-1658246400-5-etcd-2 1/1 Running 0 13m 10.104.9.154 4am-node14 <none> <none>
fouram-cron-1658246400-5-milvus-datacoord-6bbfd86785-xfvbs 1/1 Running 0 13m 10.104.1.95 4am-node10 <none> <none>
fouram-cron-1658246400-5-milvus-datanode-74fd9c49fb-mxg5x 1/1 Running 0 13m 10.104.5.97 4am-node12 <none> <none>
fouram-cron-1658246400-5-milvus-indexcoord-867d6547d8-qz8jz 1/1 Running 0 13m 10.104.4.229 4am-node11 <none> <none>
fouram-cron-1658246400-5-milvus-indexnode-cc5c88557-xs67g 1/1 Running 0 13m 10.104.9.145 4am-node14 <none> <none>
fouram-cron-1658246400-5-milvus-proxy-5cf9d8df88-zgwxm 1/1 Running 0 13m 10.104.5.98 4am-node12 <none> <none>
fouram-cron-1658246400-5-milvus-querycoord-589c5fccdc-wqpxw 1/1 Running 0 13m 10.104.1.102 4am-node10 <none> <none>
fouram-cron-1658246400-5-milvus-querynode-7fbb769975-8q7tp 1/1 Running 0 13m 10.104.1.103 4am-node10 <none> <none>
fouram-cron-1658246400-5-milvus-rootcoord-6c9d8d5649-j2dz5 1/1 Running 0 13m 10.104.1.104 4am-node10 <none> <none>
fouram-cron-1658246400-5-minio-0 1/1 Running 0 13m 10.104.4.228 4am-node11 <none> <none>
fouram-cron-1658246400-5-minio-1 1/1 Running 0 13m 10.104.1.100 4am-node10 <none> <none>
fouram-cron-1658246400-5-minio-2 1/1 Running 0 13m 10.104.5.99 4am-node12 <none> <none>
fouram-cron-1658246400-5-minio-3 1/1 Running 0 13m 10.104.6.74 4am-node13 <none> <none>
fouram-cron-1658246400-5-pulsar-bookie-0 1/1 Running 0 13m 10.104.1.116 4am-node10 <none> <none>
fouram-cron-1658246400-5-pulsar-bookie-1 1/1 Running 0 13m 10.104.4.241 4am-node11 <none> <none>
fouram-cron-1658246400-5-pulsar-bookie-2 1/1 Running 0 13m 10.104.6.83 4am-node13 <none> <none>
fouram-cron-1658246400-5-pulsar-bookie-init-rmcdl 0/1 Completed 0 13m 10.104.4.232 4am-node11 <none> <none>
fouram-cron-1658246400-5-pulsar-broker-0 1/1 Running 0 13m 10.104.4.231 4am-node11 <none> <none>
fouram-cron-1658246400-5-pulsar-proxy-0 1/1 Running 0 13m 10.104.5.100 4am-node12 <none> <none>
fouram-cron-1658246400-5-pulsar-pulsar-init-j6zdn 0/1 Completed 0 13m 10.104.4.230 4am-node11 <none> <none>
fouram-cron-1658246400-5-pulsar-recovery-0 1/1 Running 0 13m 10.104.1.96 4am-node10 <none> <none>
fouram-cron-1658246400-5-pulsar-zookeeper-0 1/1 Running 0 13m 10.104.4.237 4am-node11 <none> <none>
fouram-cron-1658246400-5-pulsar-zookeeper-1 1/1 Running 0 12m 10.104.5.116 4am-node12 <none> <none>
fouram-cron-1658246400-5-pulsar-zookeeper-2 1/1 Running 0 12m 10.104.1.127 4am-node10 <none> <none>
[2022-07-19 17:39:52,765] [ ERROR] - RPC error: [search], <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658252392.764521866","description":"Error received from peer ipv4:10.255.59.170:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>, <Time:{'RPC start': '2022-07-19 17:34:52.538377', 'RPC error': '2022-07-19 17:39:52.765409'}> (pymilvus.decorators:95)
[2022-07-19 17:39:52,767] [ ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 446, in search
return self._execute_search_requests(requests, timeout, **_kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 410, in _execute_search_requests
raise pre_err
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 398, in _execute_search_requests
response = ft.result()
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 744, in result
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658252392.764521866","description":"Error received from peer ipv4:10.255.59.170:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 95, in run_suite
result = runner.run_case(case_metric, **case)
File "/src/milvus_benchmark/runners/accuracy.py", line 283, in run_case
self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"],
File "/src/milvus_benchmark/client.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 337, in query
result = self._milvus.search(tmp_collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 844, in search
return handler.search(collection_name, data, anns_field, param, limit, expression,
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 56, in handler
raise MilvusException(Status.UNEXPECTED_ERROR, str(e))
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658252392.764521866","description":"Error received from peer ipv4:10.255.59.170:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>
(milvus_benchmark.main:98)
2.1.0-20220716-48b9b941 pymilvus 2.1.0dev103 Can run through nprobe's big scenarios
server-instance fouram-cron-bhkrs-5 server-configmap server-cluster-8c16m client-configmap client-acc-glove-ivf-flat 2.1.0-20220723-1e038a75 pymilvus 2.1.0dev103
fouram-cron-bhkrs-5-etcd-0 1/1 Running 0 5m52s 10.104.4.237 4am-node11 <none> <none>
fouram-cron-bhkrs-5-etcd-1 1/1 Running 0 5m52s 10.104.1.212 4am-node10 <none> <none>
fouram-cron-bhkrs-5-etcd-2 1/1 Running 0 5m52s 10.104.6.22 4am-node13 <none> <none>
fouram-cron-bhkrs-5-milvus-datacoord-77c7d89c8f-p68c2 1/1 Running 1 (108s ago) 5m52s 10.104.4.226 4am-node11 <none> <none>
fouram-cron-bhkrs-5-milvus-datanode-5dd5b55d44-gd2xh 1/1 Running 1 (111s ago) 5m53s 10.104.1.197 4am-node10 <none> <none>
fouram-cron-bhkrs-5-milvus-indexcoord-7566d59cd6-4c7t2 1/1 Running 1 (2m20s ago) 5m53s 10.104.1.198 4am-node10 <none> <none>
fouram-cron-bhkrs-5-milvus-indexnode-5f74599959-b4s7k 1/1 Running 0 5m53s 10.104.9.142 4am-node14 <none> <none>
fouram-cron-bhkrs-5-milvus-proxy-598dcc9886-lnhr4 1/1 Running 1 (109s ago) 5m53s 10.104.4.225 4am-node11 <none> <none>
fouram-cron-bhkrs-5-milvus-querycoord-5556b6bb85-bp54c 1/1 Running 1 (2m21s ago) 5m54s 10.104.1.195 4am-node10 <none> <none>
fouram-cron-bhkrs-5-milvus-querynode-54b6cd6989-c9cs2 1/1 Running 0 5m52s 10.104.5.234 4am-node12 <none> <none>
fouram-cron-bhkrs-5-milvus-rootcoord-7d4765897-dtclg 1/1 Running 1 (111s ago) 5m54s 10.104.1.196 4am-node10 <none> <none>
fouram-cron-bhkrs-5-minio-0 1/1 Running 0 5m54s 10.104.1.194 4am-node10 <none> <none>
fouram-cron-bhkrs-5-minio-1 1/1 Running 0 5m53s 10.104.9.141 4am-node14 <none> <none>
fouram-cron-bhkrs-5-minio-2 1/1 Running 0 5m53s 10.104.4.224 4am-node11 <none> <none>
fouram-cron-bhkrs-5-minio-3 1/1 Running 0 5m53s 10.104.5.235 4am-node12 <none> <none>
fouram-cron-bhkrs-5-pulsar-bookie-0 1/1 Running 0 5m52s 10.104.4.236 4am-node11 <none> <none>
fouram-cron-bhkrs-5-pulsar-bookie-1 1/1 Running 0 5m52s 10.104.1.213 4am-node10 <none> <none>
fouram-cron-bhkrs-5-pulsar-bookie-2 1/1 Running 0 5m51s 10.104.6.23 4am-node13 <none> <none>
fouram-cron-bhkrs-5-pulsar-bookie-init-88xlt 0/1 Completed 0 5m54s 10.104.5.232 4am-node12 <none> <none>
fouram-cron-bhkrs-5-pulsar-broker-0 1/1 Running 0 5m53s 10.104.1.199 4am-node10 <none> <none>
fouram-cron-bhkrs-5-pulsar-proxy-0 1/1 Running 0 5m54s 10.104.1.193 4am-node10 <none> <none>
fouram-cron-bhkrs-5-pulsar-pulsar-init-7fgzw 0/1 Completed 0 5m54s 10.104.1.191 4am-node10 <none> <none>
fouram-cron-bhkrs-5-pulsar-recovery-0 1/1 Running 0 5m53s 10.104.4.223 4am-node11 <none> <none>
fouram-cron-bhkrs-5-pulsar-zookeeper-0 1/1 Running 0 5m54s 10.104.1.208 4am-node10 <none> <none>
fouram-cron-bhkrs-5-pulsar-zookeeper-1 1/1 Running 0 4m40s 10.104.4.244 4am-node11 <none> <none>
fouram-cron-bhkrs-5-pulsar-zookeeper-2 1/1 Running 0 4m4s 10.104.9.152 4am-node14 <none> <none>
[2022-07-25 03:57:33,029] [ ERROR] - RPC error: [search], <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658721453.028636784","description":"Error received from peer ipv4:10.255.161.33:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>, <Time:{'RPC start': '2022-07-25 03:52:32.828528', 'RPC error': '2022-07-25 03:57:33.029483'}> (pymilvus.decorators:95)
[2022-07-25 03:57:33,032] [ ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 446, in search
return self._execute_search_requests(requests, timeout, **_kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 410, in _execute_search_requests
raise pre_err
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 398, in _execute_search_requests
response = ft.result()
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 744, in result
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658721453.028636784","description":"Error received from peer ipv4:10.255.161.33:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 95, in run_suite
result = runner.run_case(case_metric, **case)
File "/src/milvus_benchmark/runners/accuracy.py", line 283, in run_case
self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"],
File "/src/milvus_benchmark/client.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 337, in query
result = self._milvus.search(tmp_collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 844, in search
return handler.search(collection_name, data, anns_field, param, limit, expression,
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 56, in handler
raise MilvusException(Status.UNEXPECTED_ERROR, str(e))
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658721453.028636784","description":"Error received from peer ipv4:10.255.161.33:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>
(milvus_benchmark.main:98)
server-instance fouram-cron-1658937600-5 server-configmap server-cluster-8c16m client-configmap client-acc-glove-ivf-flat
2.1.0-20220727-7169256c pymilvus 2.1.0dev103
fouram-cron-1658937600-5-etcd-0 1/1 Running 0 4m39s 10.104.5.46 4am-node12 <none> <none>
fouram-cron-1658937600-5-etcd-1 1/1 Running 0 4m39s 10.104.6.88 4am-node13 <none> <none>
fouram-cron-1658937600-5-etcd-2 1/1 Running 0 4m38s 10.104.4.95 4am-node11 <none> <none>
fouram-cron-1658937600-5-milvus-datacoord-59479cc5f4-7q78g 1/1 Running 0 4m39s 10.104.6.77 4am-node13 <none> <none>
fouram-cron-1658937600-5-milvus-datanode-785cfb69f5-pb89c 1/1 Running 0 4m39s 10.104.6.78 4am-node13 <none> <none>
fouram-cron-1658937600-5-milvus-indexcoord-5887c78cff-9dxtr 1/1 Running 0 4m39s 10.104.5.32 4am-node12 <none> <none>
fouram-cron-1658937600-5-milvus-indexnode-f54c76c7d-d7tf9 1/1 Running 0 4m39s 10.104.1.7 4am-node10 <none> <none>
fouram-cron-1658937600-5-milvus-proxy-58b5fb67d9-7lwgx 1/1 Running 0 4m39s 10.104.5.33 4am-node12 <none> <none>
fouram-cron-1658937600-5-milvus-querycoord-7458456c96-z9xrl 1/1 Running 0 4m39s 10.104.4.80 4am-node11 <none> <none>
fouram-cron-1658937600-5-milvus-querynode-7895546b68-qbwsj 1/1 Running 0 4m39s 10.104.1.10 4am-node10 <none> <none>
fouram-cron-1658937600-5-milvus-rootcoord-79b7bf4f78-ljsb7 1/1 Running 0 4m39s 10.104.1.8 4am-node10 <none> <none>
fouram-cron-1658937600-5-minio-0 1/1 Running 0 4m39s 10.104.4.78 4am-node11 <none> <none>
fouram-cron-1658937600-5-minio-1 1/1 Running 0 4m39s 10.104.5.31 4am-node12 <none> <none>
fouram-cron-1658937600-5-minio-2 1/1 Running 0 4m39s 10.104.1.9 4am-node10 <none> <none>
fouram-cron-1658937600-5-minio-3 1/1 Running 0 4m39s 10.104.6.80 4am-node13 <none> <none>
fouram-cron-1658937600-5-pulsar-bookie-0 1/1 Running 0 4m39s 10.104.4.92 4am-node11 <none> <none>
fouram-cron-1658937600-5-pulsar-bookie-1 1/1 Running 0 4m39s 10.104.5.49 4am-node12 <none> <none>
fouram-cron-1658937600-5-pulsar-bookie-2 1/1 Running 0 4m38s 10.104.6.94 4am-node13 <none> <none>
fouram-cron-1658937600-5-pulsar-bookie-init-xgs86 0/1 Completed 0 4m39s 10.104.4.82 4am-node11 <none> <none>
fouram-cron-1658937600-5-pulsar-broker-0 1/1 Running 0 4m39s 10.104.6.79 4am-node13 <none> <none>
fouram-cron-1658937600-5-pulsar-proxy-0 1/1 Running 0 4m39s 10.104.5.34 4am-node12 <none> <none>
fouram-cron-1658937600-5-pulsar-pulsar-init-9lq6t 0/1 Completed 0 4m39s 10.104.4.81 4am-node11 <none> <none>
fouram-cron-1658937600-5-pulsar-recovery-0 1/1 Running 0 4m39s 10.104.4.79 4am-node11 <none> <none>
fouram-cron-1658937600-5-pulsar-zookeeper-0 1/1 Running 0 4m39s 10.104.4.94 4am-node11 <none> <none>
fouram-cron-1658937600-5-pulsar-zookeeper-1 1/1 Running 0 3m40s 10.104.1.27 4am-node10 <none> <none>
fouram-cron-1658937600-5-pulsar-zookeeper-2 1/1 Running 0 3m3s 10.104.6.103 4am-node13 <none> <none>
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658943110.117574001","description":"Error received from peer ipv4:10.255.234.75:19530","
file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 95, in run_suite
result = runner.run_case(case_metric, **case)
File "/src/milvus_benchmark/runners/accuracy.py", line 283, in run_case
self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"],
File "/src/milvus_benchmark/client.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 337, in query
result = self._milvus.search(tmp_collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 844, in search
return handler.search(collection_name, data, anns_field, param, limit, expression,
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 56, in handler
raise MilvusException(Status.UNEXPECTED_ERROR, str(e))
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1658943110.117574001","description":"Error received from peer ipv4:10.255.234.75:19530","
file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>
(milvus_benchmark.main:98)
fouram-cron-1661097600-5 server-configmap server-cluster-8c16m client-configmap client-acc-glove-ivf-flat
master-20220820-e712b23
fouram-cron-1661097600-5-etcd-0 1/1 Running 0 5m53s 10.104.9.162 4am-node14 <none> <none>
fouram-cron-1661097600-5-etcd-1 1/1 Running 0 5m52s 10.104.1.252 4am-node10 <none> <none>
fouram-cron-1661097600-5-etcd-2 1/1 Running 0 5m51s 10.104.6.133 4am-node13 <none> <none>
fouram-cron-1661097600-5-milvus-datacoord-5f74688f8f-lmsdl 1/1 Running 1 (108s ago) 5m52s 10.104.1.243 4am-node10 <none> <none>
fouram-cron-1661097600-5-milvus-datanode-56ff4f445b-wmgvz 1/1 Running 1 (2m20s ago) 5m51s 10.104.4.244 4am-node11 <none> <none>
fouram-cron-1661097600-5-milvus-indexcoord-97d5f44dd-mjzp2 1/1 Running 1 (108s ago) 5m51s 10.104.1.241 4am-node10 <none> <none>
fouram-cron-1661097600-5-milvus-indexnode-6649484665-bc65z 1/1 Running 0 5m51s 10.104.1.239 4am-node10 <none> <none>
fouram-cron-1661097600-5-milvus-proxy-6d6fdf9b4d-cxkcl 1/1 Running 1 (2m17s ago) 5m52s 10.104.1.244 4am-node10 <none> <none>
fouram-cron-1661097600-5-milvus-querycoord-866d44f9f-zmnr8 1/1 Running 1 (2m19s ago) 5m51s 10.104.5.162 4am-node12 <none> <none>
fouram-cron-1661097600-5-milvus-querynode-86d894684c-dqcq9 1/1 Running 0 5m51s 10.104.6.124 4am-node13 <none> <none>
fouram-cron-1661097600-5-milvus-rootcoord-bb5667c85-vjbzr 1/1 Running 1 (111s ago) 5m52s 10.104.6.121 4am-node13 <none> <none>
fouram-cron-1661097600-5-minio-0 1/1 Running 0 5m52s 10.104.1.238 4am-node10 <none> <none>
fouram-cron-1661097600-5-minio-1 1/1 Running 0 5m52s 10.104.6.122 4am-node13 <none> <none>
fouram-cron-1661097600-5-minio-2 1/1 Running 0 5m52s 10.104.4.242 4am-node11 <none> <none>
fouram-cron-1661097600-5-minio-3 1/1 Running 0 5m51s 10.104.5.159 4am-node12 <none> <none>
fouram-cron-1661097600-5-pulsar-bookie-0 1/1 Running 0 5m51s 10.104.1.254 4am-node10 <none> <none>
fouram-cron-1661097600-5-pulsar-bookie-1 1/1 Running 0 5m50s 10.104.9.165 4am-node14 <none> <none>
fouram-cron-1661097600-5-pulsar-bookie-2 1/1 Running 0 5m50s 10.104.4.3 4am-node11 <none> <none>
fouram-cron-1661097600-5-pulsar-bookie-init-ffjbx 0/1 Completed 0 5m53s 10.104.9.151 4am-node14 <none> <none>
fouram-cron-1661097600-5-pulsar-broker-0 1/1 Running 0 5m53s 10.104.6.119 4am-node13 <none> <none>
fouram-cron-1661097600-5-pulsar-proxy-0 1/1 Running 0 5m53s 10.104.1.233 4am-node10 <none> <none>
fouram-cron-1661097600-5-pulsar-pulsar-init-pf2x5 0/1 Completed 0 5m53s 10.104.9.150 4am-node14 <none> <none>
fouram-cron-1661097600-5-pulsar-recovery-0 1/1 Running 0 5m53s 10.104.1.234 4am-node10 <none> <none>
fouram-cron-1661097600-5-pulsar-zookeeper-0 1/1 Running 0 5m53s 10.104.9.161 4am-node14 <none> <none>
fouram-cron-1661097600-5-pulsar-zookeeper-1 1/1 Running 0 4m50s 10.104.6.140 4am-node13 <none> <none>
fouram-cron-1661097600-5-pulsar-zookeeper-2 1/1 Running 0 4m12s 10.104.4.10 4am-node11 <none> <none>
[2022-08-21 17:30:59,154] [ INFO] - {'acc': 0.979, 'search_rps': 161.38094186782837, 'rps_pv': 16.138094186782837} (milvus_benchmark.main:99)
[2022-08-21 17:30:59,155] [ DEBUG] - {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 161.38094186782837, 'rps_pv': 16.138094186782837}} (milvus_benchmark.main:107)
[2022-08-21 17:30:59,155] [ DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1661102587, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7f98a80474f0>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7f98a8047400>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7f98a8047370>, 'status': 'RUN_SUCC', 'err_message': '', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 256}, 'filter': [], 'guarantee_timestamp': None}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 161.38094186782837, 'rps_pv': 16.138094186782837}}, 'datetime': '2022-08-21 16:07:21.386754', 'type': 'metric'} (milvus_benchmark.metric.api:30)
[2022-08-21 17:35:59,379] [ ERROR] - RPC error: [search], <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1661103359.378227499","description":"Error received from peer ipv4:10.255.14.141:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>, <Time:{'RPC start': '2022-08-21 17:30:59.164640', 'RPC error': '2022-08-21 17:35:59.379105'}> (pymilvus.decorators:95)
[2022-08-21 17:35:59,381] [ ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 446, in search
return self._execute_search_requests(requests, timeout, **_kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 410, in _execute_search_requests
raise pre_err
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 398, in _execute_search_requests
response = ft.result()
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 744, in result
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1661103359.378227499","description":"Error received from peer ipv4:10.255.14.141:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 95, in run_suite
result = runner.run_case(case_metric, **case)
File "/src/milvus_benchmark/runners/accuracy.py", line 283, in run_case
self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"],
File "/src/milvus_benchmark/client.py", line 52, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 338, in query
result = self._milvus.search(tmp_collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 844, in search
return handler.search(collection_name, data, anns_field, param, limit, expression,
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 56, in handler
raise MilvusException(Status.UNEXPECTED_ERROR, str(e))
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1661103359.378227499","description":"Error received from peer ipv4:10.255.14.141:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>
(milvus_benchmark.main:98)
server-instance fouram-cron-1661788800-5 server-configmap server-cluster-8c16m client-configmap client-acc-glove-ivf-flat
master-20220829-d28114c pymilvus 2.1.2 dev2
fouram-cron-1661788800-5-etcd-0 1/1 Running 0 5m52s 10.104.6.218 4am-node13 <none> <none>
fouram-cron-1661788800-5-etcd-1 1/1 Running 0 5m52s 10.104.9.178 4am-node14 <none> <none>
fouram-cron-1661788800-5-etcd-2 1/1 Running 0 5m51s 10.104.1.165 4am-node10 <none> <none>
fouram-cron-1661788800-5-milvus-datacoord-64b8659bf7-q8sr9 1/1 Running 1 (111s ago) 5m53s 10.104.6.192 4am-node13 <none> <none>
fouram-cron-1661788800-5-milvus-datanode-84b9648cd6-c9928 1/1 Running 1 (112s ago) 5m53s 10.104.4.4 4am-node11 <none> <none>
fouram-cron-1661788800-5-milvus-indexcoord-55d9bdbdbb-tkqck 1/1 Running 1 (112s ago) 5m53s 10.104.4.2 4am-node11 <none> <none>
fouram-cron-1661788800-5-milvus-indexnode-77797c744f-zknnb 1/1 Running 0 5m52s 10.104.5.252 4am-node12 <none> <none>
fouram-cron-1661788800-5-milvus-proxy-5698548bbd-r5mb5 1/1 Running 1 (112s ago) 5m53s 10.104.4.3 4am-node11 <none> <none>
fouram-cron-1661788800-5-milvus-querycoord-75dbb5444f-m7ms5 1/1 Running 1 (111s ago) 5m52s 10.104.6.191 4am-node13 <none> <none>
fouram-cron-1661788800-5-milvus-querynode-6fd4787495-lv4vv 1/1 Running 0 5m52s 10.104.6.194 4am-node13 <none> <none>
fouram-cron-1661788800-5-milvus-rootcoord-9d7d4b8bf-jfkdw 1/1 Running 1 (2m19s ago) 5m52s 10.104.4.6 4am-node11 <none> <none>
fouram-cron-1661788800-5-minio-0 1/1 Running 0 5m54s 10.104.4.254 4am-node11 <none> <none>
fouram-cron-1661788800-5-minio-1 1/1 Running 0 5m54s 10.104.6.196 4am-node13 <none> <none>
fouram-cron-1661788800-5-minio-2 1/1 Running 0 5m53s 10.104.5.250 4am-node12 <none> <none>
fouram-cron-1661788800-5-minio-3 1/1 Running 0 5m53s 10.104.1.161 4am-node10 <none> <none>
fouram-cron-1661788800-5-pulsar-bookie-0 1/1 Running 0 5m53s 10.104.6.216 4am-node13 <none> <none>
fouram-cron-1661788800-5-pulsar-bookie-1 1/1 Running 0 5m52s 10.104.9.175 4am-node14 <none> <none>
fouram-cron-1661788800-5-pulsar-bookie-2 1/1 Running 0 5m52s 10.104.4.18 4am-node11 <none> <none>
fouram-cron-1661788800-5-pulsar-bookie-init-vbmj5 0/1 Completed 0 5m54s 10.104.4.253 4am-node11 <none> <none>
fouram-cron-1661788800-5-pulsar-broker-0 1/1 Running 0 5m52s 10.104.1.162 4am-node10 <none> <none>
fouram-cron-1661788800-5-pulsar-proxy-0 1/1 Running 0 5m52s 10.104.5.251 4am-node12 <none> <none>
fouram-cron-1661788800-5-pulsar-pulsar-init-6fm6f 0/1 Completed 0 5m54s 10.104.6.197 4am-node13 <none> <none>
fouram-cron-1661788800-5-pulsar-recovery-0 1/1 Running 0 5m52s 10.104.1.163 4am-node10 <none> <none>
fouram-cron-1661788800-5-pulsar-zookeeper-0 1/1 Running 0 5m53s 10.104.6.215 4am-node13 <none> <none>
fouram-cron-1661788800-5-pulsar-zookeeper-1 1/1 Running 0 4m37s 10.104.9.181 4am-node14 <none> <none>
fouram-cron-1661788800-5-pulsar-zookeeper-2 1/1 Running 0 4m2s 10.104.4.22 4am-node11 <none> <none>
[2022-08-29 17:30:17,557] [ ERROR] - RPC error: [search], <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1661794217.556340665","description":"Error received from peer ipv4:10.255.157.164:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>, <Time:{'RPC start': '2022-08-29 17:25:17.356693', 'RPC error': '2022-08-29 17:30:17.557461'}> (pymilvus.decorators:100)
[2022-08-29 17:30:17,560] [ ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 50, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 473, in search
return self._execute_search_requests(requests, timeout, **_kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 437, in _execute_search_requests
raise pre_err
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 425, in _execute_search_requests
response = ft.result()
File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 744, in result
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1661794217.556340665","description":"Error received from peer ipv4:10.255.157.164:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 95, in run_suite
result = runner.run_case(case_metric, **case)
File "/src/milvus_benchmark/runners/accuracy.py", line 292, in run_case
self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"],
File "/src/milvus_benchmark/client.py", line 52, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 345, in query
result = self._milvus.search(tmp_collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 844, in search
return handler.search(collection_name, data, anns_field, param, limit, expression,
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 101, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 97, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 127, in handler
ret = func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 59, in handler
raise MilvusException(Status.UNEXPECTED_ERROR, str(e))
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1661794217.556340665","description":"Error received from peer ipv4:10.255.157.164:19530","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>)>
(milvus_benchmark.main:98)
