milvus
milvus copied to clipboard
Published
20 hours ago •
milvus-io
milvus-io
[Bug]: [benchmark][standalone]Milvus load failed ,rasie an error"collection has not been loaded to memory or load failed"
Is there an existing issue for this?
- [X] I have searched the existing issues
Environment
- Milvus version: master-20220802-efc5406
- Deployment mode(standalone or cluster):standalone
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus 2.1.0dev103
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
server-instance fouram-cluster-cron-1659474000-1 server-configmap server-single-32c128m client-configmap client-get-local50m-512 server
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouram-cluster-cron-1659474000-1-etcd-0 1/1 Running 0 116s 10.104.6.226 4am-node13 <none> <none>
fouram-cluster-cron-1659474000-1-milvus-standalone-587c67bkzpc9 1/1 Running 0 116s 10.104.9.125 4am-node14 <none> <none>
fouram-cluster-cron-1659474000-1-minio-98d84644-j4bsx 1/1 Running 0 116s 10.104.9.121 4am-node14 <none> <none>
client
2022-08-02 22:50:28,207] [ DEBUG] - Start load collection (milvus_benchmark.runners.get:130)
[2022-08-02 22:53:10,781] [ ERROR] - RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-02 22:50:28.436100', 'RPC error': '2022-08-02 22:53:10.781008'}> (pymilvus.decorators:95)
[2022-08-02 22:53:10,781] [ ERROR] - RPC error: [load_collection], <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-02 22:50:28.207749', 'RPC error': '2022-08-02 22:53:10.781320'}> (pymilvus.decorators:95)
[2022-08-02 22:53:10,781] [ ERROR] - <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or load failed)> (milvus_benchmark.main:118)
[2022-08-02 22:53:10,782] [ ERROR] - Traceback (most recent call last):
File "main.py", line 87, in run_suite
runner.prepare(**cases[0])
File "/src/milvus_benchmark/runners/get.py", line 131, in prepare
self.milvus.load_collection(replica_number=replica_number, timeout=1200)
File "/src/milvus_benchmark/client.py", line 52, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 581, in load_collection
return self._milvus.load_collection(collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 195, in load_collection
return handler.load_collection(collection_name=collection_name, replica_number=replica_number, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 673, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 682, in wait_for_loading_collection
return self._wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 711, in _wait_for_loading_collection
progress = self.get_collection_loading_progress(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 690, in get_collection_loading_progress
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or load failed)>
(milvus_benchmark.main:119)
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
No response
Anything else?
No response
server-instance fouram-cluster-cron-1659474000-5 server-configmap server-single-16c64m client-configmap client-search-filter-sift50m-ivf-flat-2048
master-20220802-efc5406 pymilvus 2.1.0dev103
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouram-cluster-cron-1659474000-5-etcd-0 1/1 Running 0 2m9s 10.104.5.117 4am-node12 <none> <none>
fouram-cluster-cron-1659474000-5-milvus-standalone-6658867dfjgm 1/1 Running 0 2m9s 10.104.1.115 4am-node10 <none> <none>
fouram-cluster-cron-1659474000-5-minio-7cdbfd85f6-stcq5 1/1 Running 0 2m9s 10.104.1.104 4am-node10 <none> <none>
[2022-08-02 21:40:24,134] [ WARNING] - [93m[wait_for_loading_collection] retry:4, cost: 0.27s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses>[0m (pymilvus.decorators:69)
[2022-08-02 21:40:24,406] [ WARNING] - [93m[wait_for_loading_collection] retry:5, cost: 0.81s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses>[0m (pymilvus.decorators:69)
[2022-08-02 21:40:25,217] [ WARNING] - [93m[wait_for_loading_collection] retry:6, cost: 2.43s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses>[0m (pymilvus.decorators:69)
[2022-08-02 21:40:27,649] [ WARNING] - [93m[wait_for_loading_collection] retry:7, cost: 7.290000000000001s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses>[0m (pymilvus.decorators:69)
[2022-08-02 21:40:34,941] [ WARNING] - [93m[wait_for_loading_collection] retry:8, cost: 21.870000000000005s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses>[0m (pymilvus.decorators:69)
[2022-08-02 21:40:56,813] [ WARNING] - [93m[wait_for_loading_collection] retry:9, cost: 60s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses>[0m (pymilvus.decorators:69)
[2022-08-02 21:41:56,814] [ WARNING] - [93m[wait_for_loading_collection] retry:10, cost: 60s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses>[0m (pymilvus.decorators:69)
[2022-08-02 21:43:16,758] [ ERROR] - RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-02 21:37:32.243568', 'RPC error': '2022-08-02 21:43:16.758765'}> (pymilvus.decorators:95)
[2022-08-02 21:43:16,759] [ ERROR] - RPC error: [load_collection], <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-02 21:37:31.047842', 'RPC error': '2022-08-02 21:43:16.758991'}> (pymilvus.decorators:95)
[2022-08-02 21:43:16,759] [ ERROR] - <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)> (milvus_benchmark.main:118)
[2022-08-02 21:43:16,760] [ ERROR] - Traceback (most recent call last):
File "main.py", line 87, in run_suite
runner.prepare(**cases[0])
File "/src/milvus_benchmark/runners/search.py", line 295, in prepare
self.milvus.load_collection(replica_number=replica_number, timeout=1200)
File "/src/milvus_benchmark/client.py", line 52, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 581, in load_collection
return self._milvus.load_collection(collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 195, in load_collection
return handler.load_collection(collection_name=collection_name, replica_number=replica_number, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 673, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 682, in wait_for_loading_collection
return self._wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 711, in _wait_for_loading_collection
progress = self.get_collection_loading_progress(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 690, in get_collection_loading_progress
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>
(milvus_benchmark.main:119)
server-instance fouram-cron-1659456000-6 server-configmap server-single-16c64m client-configmap client-search-sift50m-ivf-flat-2048
master-20220802-efc5406
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouram-cron-1659456000-6-etcd-0 1/1 Running 0 2m1s 10.104.6.194 4am-node13 <none> <none>
fouram-cron-1659456000-6-milvus-standalone-7479799b86-lvvzv 1/1 Running 0 2m1s 10.104.1.55 4am-node10 <none> <none>
fouram-cron-1659456000-6-minio-6db566b56c-2flcg 1/1 Running 0 2m1s 10.104.6.175 4am-node13 <none> <none>
2022-08-02 16:45:54,081] [ ERROR] - RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-02 16:44:10.970257', 'RPC error': '2022-08-02 16:45:54.081357'}> (pymilvus.decorators:95)
[2022-08-02 16:45:54,081] [ ERROR] - RPC error: [load_collection], <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-02 16:44:10.871152', 'RPC error': '2022-08-02 16:45:54.081651'}> (pymilvus.decorators:95)
[2022-08-02 16:45:54,081] [ ERROR] - <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)> (milvus_benchmark.main:118)
[2022-08-02 16:45:54,082] [ ERROR] - Traceback (most recent call last):
File "main.py", line 87, in run_suite
runner.prepare(**cases[0])
File "/src/milvus_benchmark/runners/search.py", line 295, in prepare
self.milvus.load_collection(replica_number=replica_number, timeout=1200)
File "/src/milvus_benchmark/client.py", line 52, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 581, in load_collection
return self._milvus.load_collection(collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 195, in load_collection
return handler.load_collection(collection_name=collection_name, replica_number=replica_number, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 673, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 682, in wait_for_loading_collection
return self._wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 711, in _wait_for_loading_collection
progress = self.get_collection_loading_progress(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 690, in get_collection_loading_progress
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>
(milvus_benchmark.main:119)
server-instance fouram-cluster-cron-1659560400-1 server-configmap server-single-32c128m client-configmap client-get-local50m-512 master-20220803-f84d176 pymilvus:2.1.0.dev103
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouram-cluster-cron-1659560400-1-etcd-0 1/1 Running 0 2m9s 10.104.9.226 4am-node14 <none> <none>
fouram-cluster-cron-1659560400-1-milvus-standalone-7dc49fdn2ppp 1/1 Running 0 2m9s 10.104.5.51 4am-node12 <none> <none>
fouram-cluster-cron-1659560400-1-minio-6c6b59fcf6-4pdbl 1/1 Running 0 2m9s 10.104.4.2 4am-node11 <none> <none>
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 673, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 682, in wait_for_loading_collection
return self._wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 711, in _wait_for_loading_collection
progress = self.get_collection_loading_progress(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 690, in get_collection_loading_progress
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or
load failed)>
(milvus_benchmark.main:119)
server-instance fouram-cluster-cron-1659646800-1 server-configmap server-single-32c128m client-configmap client-get-local50m-512
master-20220804-ff02002 pymilvus 2.1.0dev103
NAME READY STATUS RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
fouram-cluster-cron-1659646800-1-etcd-0 1/1 Running 0 110s 10.104.6.64 4am-nod
e13 <none> <none>
fouram-cluster-cron-1659646800-1-milvus-standalone-746cf58rb892 1/1 Running 0 110s 10.104.4.108 4am-nod
e11 <none> <none>
fouram-cluster-cron-1659646800-1-minio-7584cbb856-pwc97 1/1 Running 0 110s 10.104.4.105 4am-nod
e11 <none> <none>
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 195, in load_collection
return handler.load_collection(collection_name=collection_name, replica_number=replica_number, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 673, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 682, in wait_for_loading_collection
return self._wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 711, in _wait_for_loading_collection
progress = self.get_collection_loading_progress(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 690, in get_collection_loading_progress
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or
load failed)>
(milvus_benchmark.main:119)
server-instance fouram-cron-1659974400-6 server-configmap server-single-16c64m client-configmap client-search-sift50m-ivf-flat-2048 master-20220805-c5c3392 2.1.1dev3
fouram-cron-1659974400-6-etcd-0 1/1 Running 0 113s 10.104.1.62 4am-node10 <none> <none>
fouram-cron-1659974400-6-milvus-standalone-8fc9d59dc-6j8hz 1/1 Running 0 113s 10.104.5.107 4am-node12 <none> <none>
fouram-cron-1659974400-6-minio-5d4cfb9b75-59hwg 1/1 Running 0 113s 10.104.1.48 4am-node10 <none> <none>
[2022-08-08 16:38:19,077] [ INFO] - Start load collection (milvus_benchmark.runners.search:293)
[2022-08-08 16:41:17,752] [ ERROR] - RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-08 16:38:19.180060', 'RPC error': '2022-08-08 16:41:17.752029'}> (pymilvus.decorators:95)
[2022-08-08 16:41:17,752] [ ERROR] - RPC error: [load_collection], <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-08 16:38:19.077343', 'RPC error': '2022-08-08 16:41:17.752401'}> (pymilvus.decorators:95)
[2022-08-08 16:41:17,752] [ ERROR] - <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)> (milvus_benchmark.main:118)
[2022-08-08 16:41:17,754] [ ERROR] - Traceback (most recent call last):
File "main.py", line 87, in run_suite
runner.prepare(**cases[0])
File "/src/milvus_benchmark/runners/search.py", line 295, in prepare
self.milvus.load_collection(replica_number=replica_number, timeout=1200)
File "/src/milvus_benchmark/client.py", line 52, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 581, in load_collection
return self._milvus.load_collection(collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 195, in load_collection
return handler.load_collection(collection_name=collection_name, replica_number=replica_number, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 673, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 682, in wait_for_loading_collection
return self._wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 711, in _wait_for_loading_collection
progress = self.get_collection_loading_progress(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 690, in get_collection_loading_progress
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>
(milvus_benchmark.main:119)
server-instance fouram-cluster-cron-1661115600-1 server-configmap server-single-32c128m client-configmap client-get-local50m-512 master-20220820-e712b23
2.2.0dev3
fouram-cluster-cron-1661115600-1-etcd-0 1/1 Running 0 2m3s 10.104.1.11 4am-node10 <none> <none>
fouram-cluster-cron-1661115600-1-milvus-standalone-64bdc94vw5mx 1/1 Running 0 2m3s 10.104.4.19 4am-node11 <none> <none>
fouram-cluster-cron-1661115600-1-minio-6d54578987-5tmxr 1/1 Running 0 2m3s 10.104.4.13 4am-node11 <none> <none>
[2022-08-21 23:05:18,705] [ ERROR] - RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-21 22:57:57.068736', 'RPC error': '2022-08-21 23:05:18.705288'}> (pymilvus.decorators:95)
[2022-08-21 23:05:18,706] [ ERROR] - RPC error: [load_collection], <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-08-21 22:57:56.625427', 'RPC error': '2022-08-21 23:05:18.706330'}> (pymilvus.decorators:95)
[2022-08-21 23:05:18,706] [ ERROR] - <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or load failed)> (milvus_benchmark.main:118)
[2022-08-21 23:05:18,710] [ ERROR] - Traceback (most recent call last):
File "main.py", line 87, in run_suite
runner.prepare(**cases[0])
File "/src/milvus_benchmark/runners/get.py", line 131, in prepare
self.milvus.load_collection(replica_number=replica_number, timeout=1200)
File "/src/milvus_benchmark/client.py", line 52, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 581, in load_collection
return self._milvus.load_collection(collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 195, in load_collection
return handler.load_collection(collection_name=collection_name, replica_number=replica_number, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 673, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 96, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 74, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 48, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 682, in wait_for_loading_collection
return self._wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 711, in _wait_for_loading_collection
progress = self.get_collection_loading_progress(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 690, in get_collection_loading_progress
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection local_50m_512_l2 has not been loaded to memory or load failed)>
(milvus_benchmark.main:119)
server-instance fouram-cron-1664208000-7 server-configmap server-cluster-16c64m client-configmap client-search-sift50m-ivf-flat-2048
master-20220926-911ea6c2 2.2.0.dev30
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
fouram-cron-1664208000-7-etcd-0 1/1 Running 0 5m52s 10.104.1.45 4am-node10 <none> <none>
fouram-cron-1664208000-7-etcd-1 1/1 Running 0 5m52s 10.104.4.28 4am-node11 <none> <none>
fouram-cron-1664208000-7-etcd-2 1/1 Running 0 5m51s 10.104.6.79 4am-node13 <none> <none>
fouram-cron-1664208000-7-milvus-datacoord-dc96c859c-96mrw 1/1 Running 1 (109s ago) 5m54s 10.104.9.227 4am-node14 <none> <none>
fouram-cron-1664208000-7-milvus-datanode-8446d59777-7rqc2 1/1 Running 1 (111s ago) 5m53s 10.104.9.218 4am-node14 <none> <none>
fouram-cron-1664208000-7-milvus-indexcoord-6df478bf57-7z9hd 1/1 Running 1 (111s ago) 5m54s 10.104.9.219 4am-node14 <none> <none>
fouram-cron-1664208000-7-milvus-indexnode-5cfcdc685d-n6fdh 1/1 Running 0 5m53s 10.104.5.132 4am-node12 <none> <none>
fouram-cron-1664208000-7-milvus-proxy-7cb59f86b5-cmcn9 1/1 Running 1 (110s ago) 5m53s 10.104.9.224 4am-node14 <none> <none>
fouram-cron-1664208000-7-milvus-querycoord-6c98b97b7d-vf2dg 1/1 Running 1 (110s ago) 5m54s 10.104.9.225 4am-node14 <none> <none>
fouram-cron-1664208000-7-milvus-querynode-845cdb7488-zbbc6 1/1 Running 0 5m54s 10.104.6.65 4am-node13 <none> <none>
fouram-cron-1664208000-7-milvus-rootcoord-7dc98b4f74-x4222 1/1 Running 1 (110s ago) 5m53s 10.104.9.223 4am-node14 <none> <none>
fouram-cron-1664208000-7-minio-0 1/1 Running 0 5m53s 10.104.9.229 4am-node14 <none> <none>
fouram-cron-1664208000-7-minio-1 1/1 Running 0 5m53s 10.104.1.29 4am-node10 <none> <none>
fouram-cron-1664208000-7-minio-2 1/1 Running 0 5m52s 10.104.4.10 4am-node11 <none> <none>
fouram-cron-1664208000-7-minio-3 1/1 Running 0 5m52s 10.104.6.66 4am-node13 <none> <none>
fouram-cron-1664208000-7-pulsar-bookie-0 1/1 Running 0 5m53s 10.104.1.44 4am-node10 <none> <none>
fouram-cron-1664208000-7-pulsar-bookie-1 1/1 Running 0 5m52s 10.104.4.27 4am-node11 <none> <none>
fouram-cron-1664208000-7-pulsar-bookie-2 1/1 Running 0 5m51s 10.104.6.80 4am-node13 <none> <none>
fouram-cron-1664208000-7-pulsar-bookie-init-jjwh6 0/1 Completed 0 5m54s 10.104.9.216 4am-node14 <none> <none>
fouram-cron-1664208000-7-pulsar-broker-0 1/1 Running 0 5m54s 10.104.9.222 4am-node14 <none> <none>
fouram-cron-1664208000-7-pulsar-proxy-0 1/1 Running 0 5m54s 10.104.1.28 4am-node10 <none> <none>
fouram-cron-1664208000-7-pulsar-pulsar-init-nf8ng 0/1 Completed 0 5m54s 10.104.9.217 4am-node14 <none> <none>
fouram-cron-1664208000-7-pulsar-recovery-0 1/1 Running 0 5m54s 10.104.9.221 4am-node14 <none> <none>
fouram-cron-1664208000-7-pulsar-zookeeper-0 1/1 Running 0 5m53s 10.104.9.245 4am-node14 <none> <none>
fouram-cron-1664208000-7-pulsar-zookeeper-1 1/1 Running 0 4m27s 10.104.6.85 4am-node13 <none> <none>
fouram-cron-1664208000-7-pulsar-zookeeper-2 1/1 Running 0 3m47s 10.104.4.33 4am-node11 <none> <none>
[2022-09-26 17:10:35,053] [ ERROR] - RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-09-26 17:00:33.915303', 'RPC error': '2022-09-26 17:10:35.052906'}> (pymilvus.decorators:112)
[2022-09-26 17:10:35,053] [ ERROR] - RPC error: [load_collection], <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2022-09-26 17:00:33.853554', 'RPC error': '2022-09-26 17:10:35.053350'}> (pymilvus.decorators:112)
[2022-09-26 17:10:35,053] [ ERROR] - <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)> (milvus_benchmark.main:118)
[2022-09-26 17:10:35,055] [ ERROR] - Traceback (most recent call last):
File "main.py", line 87, in run_suite
runner.prepare(**cases[0])
File "/src/milvus_benchmark/runners/search.py", line 295, in prepare
self.milvus.load_collection(replica_number=replica_number, timeout=1200)
File "/src/milvus_benchmark/client.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 592, in load_collection
return self._milvus.load_collection(collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 195, in load_collection
return handler.load_collection(collection_name=collection_name, replica_number=replica_number, timeout=timeout, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
ret = func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 664, in load_collection
self.wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
ret = func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 673, in wait_for_loading_collection
return self._wait_for_loading_collection(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 702, in _wait_for_loading_collection
progress = self.get_collection_loading_progress(collection_name, timeout)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 681, in get_collection_loading_progress
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection sift_50m_128_l2 has not been loaded to memory or load failed)>
(milvus_benchmark.main:119)
server-instance fouram-r6z57-1 server-configmap server-single-16c64m client-configmap client-search-sift50m-ivf-flat-2048、
master-20220929-3a770f7d 2.2.0dev32
fouram-r6z57-1-etcd-0 1/1 Running 0 17m 10.104.4.67 4am-node11 <none> <none>
fouram-r6z57-1-milvus-standalone-779cb8b9d-mqlf2 1/1 Running 0 17m 10.104.1.18 4am-node10 <none> <none>
fouram-r6z57-1-minio-7d6b5696df-52kbx 1/1 Running 0 17m 10.104.4.68 4am-node11 <none> <none>
[2022-09-29 08:36:09,765] [ ERROR] - RPC error: [search], <MilvusException: (code=1, message=collection:sift_50m_128_l2 or partition:[] not loaded into memory when search)>, <Time:{'RPC start': '2022-09-29 08:36:09.758899', 'RPC error': '2022-09-29 08:36:09.765572'}> (pymilvus.decorators:112)
[2022-09-29 08:36:09,765] [ ERROR] - Traceback (most recent call last):
File "main.py", line 95, in run_suite
result = runner.run_case(case_metric, **case)
File "/src/milvus_benchmark/runners/search.py", line 309, in run_case
_query_res = self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"],
File "/src/milvus_benchmark/client.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/src/milvus_benchmark/client.py", line 346, in query
result = self._milvus.search(tmp_collection_name, **params)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 844, in search
return handler.search(collection_name, data, anns_field, param, limit, expression,
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
ret = func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 472, in search
return self._execute_search_requests(requests, timeout, **_kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 436, in _execute_search_requests
raise pre_err
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 427, in _execute_search_requests
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection:sift_50m_128_l2 or partition:[] not loaded into memory when search)>
(milvus_benchmark.main:98)