milvus
milvus copied to clipboard
Published
20 hours ago •
milvus-io
milvus-io
[Bug]: [benchmark][varchar]Milvus hangs when creating indexes for scalar field
Is there an existing issue for this?
- [X] I have searched the existing issues
Environment
- Milvus version:2.2.0-20221213-1c309a74
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):2.2.1dev4
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
deploy_config: fouramf-server-cluster-8c16m-test case_params: fouramf-client-varchar-search-maxlen5000 other_params: --milvus_tag=2.2.0-20221213-1c309a74 -k test_search_custom_parameters -s --case_skip_clean_collection --deploy_mode=cluster --sync_report
server:
fouramf-wr4nn-72-4455-etcd-0 1/1 Running 0 14h 10.104.6.206 4am-node13 <none> <none>
fouramf-wr4nn-72-4455-etcd-1 1/1 Running 0 14h 10.104.4.19 4am-node11 <none> <none>
fouramf-wr4nn-72-4455-etcd-2 1/1 Running 0 14h 10.104.5.73 4am-node12 <none> <none>
fouramf-wr4nn-72-4455-milvus-datacoord-75659cf6f7-9wcjw 1/1 Running 1 (14h ago) 14h 10.104.6.194 4am-node13 <none> <none>
fouramf-wr4nn-72-4455-milvus-datanode-54cb75c947-b5gr6 1/1 Running 1 (14h ago) 14h 10.104.4.17 4am-node11 <none> <none>
fouramf-wr4nn-72-4455-milvus-indexcoord-f86c78d7c-x5mt4 1/1 Running 1 (14h ago) 14h 10.104.5.63 4am-node12 <none> <none>
fouramf-wr4nn-72-4455-milvus-indexnode-5554cc9c8-pssxx 1/1 Running 0 14h 10.104.9.130 4am-node14 <none> <none>
fouramf-wr4nn-72-4455-milvus-proxy-598fc8fbcc-5w2b4 1/1 Running 1 (14h ago) 14h 10.104.1.135 4am-node10 <none> <none>
fouramf-wr4nn-72-4455-milvus-querycoord-77c97f9dcd-474cr 1/1 Running 1 (14h ago) 14h 10.104.1.134 4am-node10 <none> <none>
fouramf-wr4nn-72-4455-milvus-querynode-77df67496d-hdsl8 1/1 Running 0 14h 10.104.1.136 4am-node10 <none> <none>
fouramf-wr4nn-72-4455-milvus-rootcoord-5447554bd4-k5x5l 1/1 Running 1 (14h ago) 14h 10.104.5.61 4am-node12 <none> <none>
fouramf-wr4nn-72-4455-minio-0 1/1 Running 0 14h 10.104.6.203 4am-node13 <none> <none>
fouramf-wr4nn-72-4455-minio-1 1/1 Running 0 14h 10.104.5.68 4am-node12 <none> <none>
fouramf-wr4nn-72-4455-minio-2 1/1 Running 0 14h 10.104.4.22 4am-node11 <none> <none>
fouramf-wr4nn-72-4455-minio-3 1/1 Running 0 14h 10.104.1.140 4am-node10 <none> <none>
fouramf-wr4nn-72-4455-pulsar-bookie-0 1/1 Running 0 14h 10.104.6.204 4am-node13 <none> <none>
fouramf-wr4nn-72-4455-pulsar-bookie-1 1/1 Running 0 14h 10.104.5.71 4am-node12 <none> <none>
fouramf-wr4nn-72-4455-pulsar-bookie-2 1/1 Running 0 14h 10.104.1.141 4am-node10 <none> <none>
fouramf-wr4nn-72-4455-pulsar-bookie-init-lxmwd 0/1 Completed 0 14h 10.104.6.196 4am-node13 <none> <none>
fouramf-wr4nn-72-4455-pulsar-broker-0 1/1 Running 0 14h 10.104.5.62 4am-node12 <none> <none>
fouramf-wr4nn-72-4455-pulsar-proxy-0 1/1 Running 0 14h 10.104.6.193 4am-node13 <none> <none>
fouramf-wr4nn-72-4455-pulsar-pulsar-init-nfvrb 0/1 Completed 0 14h 10.104.6.195 4am-node13 <none> <none>
fouramf-wr4nn-72-4455-pulsar-recovery-0 1/1 Running 0 14h 10.104.5.64 4am-node12 <none> <none>
fouramf-wr4nn-72-4455-pulsar-zookeeper-0 1/1 Running 0 14h 10.104.5.66 4am-node12 <none> <none>
fouramf-wr4nn-72-4455-pulsar-zookeeper-1 1/1 Running 0 14h 10.104.6.208 4am-node13 <none> <none>
fouramf-wr4nn-72-4455-pulsar-zookeeper-2 1/1 Running 0 14h 10.104.4.24 4am-node11 <none> <none>
client log: It has been more than 10 hours since the scalar field was indexed
2022-12-15 12:44:58,503 - INFO - fouram]: [Base] Start inserting, ids: 999900 - 999999, data size: 1,000,000 (base.py:133)
[2022-12-15 12:44:58,548 - INFO - fouram]: [Time] Collection.insert run in 0.0442s (api_request.py:41)
[2022-12-15 12:44:58,551 - INFO - fouram]: [Base] Number of vectors in the collection(fouram_LRViMFK3): 979309 (base.py:281)
[2022-12-15 12:44:58,556 - INFO - fouram]: [Base] Total time of insert: 622.0892s, average number of vector bars inserted per secon
d: 1607.4865, average time to insert 100 vectors per time: 0.0622s (base.py:200)
[2022-12-15 12:44:58,557 - INFO - fouram]: [Base] Start flush collection fouram_LRViMFK3 (base.py:104)
[2022-12-15 12:45:03,233 - INFO - fouram]: [Base] Params of index: {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 8, '
efConstruction': 200}} (base.py:259)
[2022-12-15 12:45:03,233 - INFO - fouram]: [Base] Start build index of HNSW for collection fouram_LRViMFK3, params:{'index_type': '
HNSW', 'metric_type': 'L2', 'params': {'M': 8, 'efConstruction': 200}} (base.py:246)
[2022-12-15 12:45:17,459 - INFO - fouram]: [Time] Index run in 14.2253s (api_request.py:41)
[2022-12-15 12:45:17,459 - INFO - fouram]: [CommonCases] RT of build index HNSW: 14.2253s (common_cases.py:81)
[2022-12-15 12:45:17,466 - INFO - fouram]: [Base] Params of index: {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 8, '
efConstruction': 200}} (base.py:259)
[2022-12-15 12:45:17,466 - INFO - fouram]: [CommonCases] Prepare index HNSW done. (common_cases.py:84)
[2022-12-15 12:45:17,472 - INFO - fouram]: [Base] Params of index: {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 8, '
efConstruction': 200}} (base.py:259)
[2022-12-15 12:45:17,472 - INFO - fouram]: [Base] Start build scalar index of varchar_1 (base.py:253)
Expected Behavior
No response
Steps To Reproduce
1. create a collection
2. build index on vector column
3. insert a 1m of vectors
4. flush collection
5. build index on vector column with the same parameters
6. build index on on scalars
7. count the total number of rows
8. load collection
9. search collection
Milvus Log
No response
Anything else?
scene_search required params: {'dataset_params': {'dim': 128, 'max_length
': 5000, 'varchar_filled': True, 'scalars_index': ['varchar_1'], 'dataset_name': 'sift', 'dataset_size': '1m', 'ni_per': 100, 'metri
c_type': 'L2', 'req_run_counts': 100000}, 'collection_params': {'other_fields': ['varchar_1']}, 'load_params': {}, 'search_params':
{'top_k': [1], 'nq': [1], 'search_param': {'ef': [64], 'output_fields': ['varchar_1']}}, 'index_params': {'index_type': 'HNSW', 'ind
ex_param': {'M': 8, 'efConstruction': 200}}} (params_check.py:31)
Collection schema: {'auto_id': False, 'description': '', 'fields': [{'name': 'id'
, 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '',
'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}, {'name': 'varchar_1', 'description': '', 'type': <DataType.VARCHAR:
21>, 'params': {'max_length': 5000}}]} (base.py:118)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Since proto has a 2G size limit, data exchange between go and c++ will fail, resulting in build index failure
Another issue is that the estimated size of the string field is inaccurate，resulting in the size of the string column in a segment exceeding 2G
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
image: 2.2.0-20230814-27fe2a45
server:
fouramf-wpnx4-1-3666-etcd-0 1/1 Running 0 17h 10.104.21.94 4am-node24 <none> <none>
fouramf-wpnx4-1-3666-etcd-1 1/1 Running 0 17h 10.104.13.149 4am-node16 <none> <none>
fouramf-wpnx4-1-3666-etcd-2 1/1 Running 0 17h 10.104.1.165 4am-node10 <none> <none>
fouramf-wpnx4-1-3666-milvus-datacoord-7644c8f696-r4tdm 1/1 Running 0 17h 10.104.19.5 4am-node28 <none> <none>
fouramf-wpnx4-1-3666-milvus-datanode-6cfbdcd67-jphgt 1/1 Running 1 (17h ago) 17h 10.104.18.166 4am-node25 <none> <none>
fouramf-wpnx4-1-3666-milvus-indexcoord-77fd496d99-drwp4 1/1 Running 1 (17h ago) 17h 10.104.18.169 4am-node25 <none> <none>
fouramf-wpnx4-1-3666-milvus-indexnode-68cc85549b-gg5ng 0/1 Running 129 (5m37s ago) 17h 10.104.21.90 4am-node24 <none> <none>
fouramf-wpnx4-1-3666-milvus-proxy-5b65cc979c-2ff4m 1/1 Running 1 (17h ago) 17h 10.104.16.67 4am-node21 <none> <none>
fouramf-wpnx4-1-3666-milvus-querycoord-867689b644-w8g2p 1/1 Running 1 (17h ago) 17h 10.104.24.158 4am-node29 <none> <none>
fouramf-wpnx4-1-3666-milvus-querynode-6f54f6cccd-845tr 1/1 Running 0 17h 10.104.24.160 4am-node29 <none> <none>
fouramf-wpnx4-1-3666-milvus-rootcoord-657dcc657d-7kbs6 1/1 Running 1 (17h ago) 17h 10.104.18.168 4am-node25 <none> <none>
fouramf-wpnx4-1-3666-minio-0 1/1 Running 0 17h 10.104.21.92 4am-node24 <none> <none>
fouramf-wpnx4-1-3666-minio-1 1/1 Running 0 17h 10.104.16.71 4am-node21 <none> <none>
fouramf-wpnx4-1-3666-minio-2 1/1 Running 0 17h 10.104.1.163 4am-node10 <none> <none>
fouramf-wpnx4-1-3666-minio-3 1/1 Running 0 17h 10.104.14.67 4am-node18 <none> <none>
fouramf-wpnx4-1-3666-pulsar-bookie-0 1/1 Running 0 17h 10.104.19.10 4am-node28 <none> <none>
fouramf-wpnx4-1-3666-pulsar-bookie-1 1/1 Running 0 17h 10.104.16.76 4am-node21 <none> <none>
fouramf-wpnx4-1-3666-pulsar-bookie-2 1/1 Running 0 17h 10.104.20.155 4am-node22 <none> <none>
fouramf-wpnx4-1-3666-pulsar-bookie-init-ctlf4 0/1 Completed 0 17h 10.104.21.89 4am-node24 <none> <none>
fouramf-wpnx4-1-3666-pulsar-broker-0 1/1 Running 0 17h 10.104.18.167 4am-node25 <none> <none>
fouramf-wpnx4-1-3666-pulsar-proxy-0 1/1 Running 0 17h 10.104.24.159 4am-node29 <none> <none>
fouramf-wpnx4-1-3666-pulsar-pulsar-init-cnx2p 0/1 Completed 0 17h 10.104.21.85 4am-node24 <none> <none>
fouramf-wpnx4-1-3666-pulsar-recovery-0 1/1 Running 0 17h 10.104.13.142 4am-node16 <none> <none>
fouramf-wpnx4-1-3666-pulsar-zookeeper-0 1/1 Running 0 17h 10.104.16.70 4am-node21 <none> <none>
fouramf-wpnx4-1-3666-pulsar-zookeeper-1 1/1 Running 0 17h 10.104.18.175 4am-node25 <none> <none>
fouramf-wpnx4-1-3666-pulsar-zookeeper-2 1/1 Running 0 17h 10.104.13.152 4am-node16 <none> <none>
client log:
[2023-08-15 09:23:57,005 - INFO - fouram]: [Time] Collection.insert run in 0.0777s (api_request.py:45)
[2023-08-15 09:23:57,007 - INFO - fouram]: [Base] Number of vectors in the collection(fouram_t6u09APP): 75500 (base.py:483)
[2023-08-15 09:23:57,012 - INFO - fouram]: [Base] Total time of insert: 81.0848s, average number of vector bars inserted per second
: 1233.2768, average time to insert 100 vectors per time: 0.0811s (base.py:394)
[2023-08-15 09:23:57,012 - INFO - fouram]: [Base] Start flush collection fouram_t6u09APP (base.py:292)
[2023-08-15 09:24:10,101 - INFO - fouram]: [Base] Params of index: [{'float_vector': {'index_type': 'HNSW', 'metric_type': 'L2', 'p
arams': {'M': 8, 'efConstruction': 200}}}, {'varchar_1': {}}] (base.py:456)
[2023-08-15 09:24:10,101 - INFO - fouram]: [Base] Start release collection fouram_t6u09APP (base.py:303)
[2023-08-15 09:24:10,104 - INFO - fouram]: [Base] Start build index of HNSW for collection fouram_t6u09APP, params:{'index_type': '
HNSW', 'metric_type': 'L2', 'params': {'M': 8, 'efConstruction': 200}} (base.py:442)
[2023-08-15 09:25:28,952 - INFO - fouram]: [Time] Index run in 78.8476s (api_request.py:45)
[2023-08-15 09:25:28,953 - INFO - fouram]: [CommonCases] RT of build index HNSW: 78.8476s (common_cases.py:96)
[2023-08-15 09:25:28,962 - INFO - fouram]: [Base] Params of index: [{'float_vector': {'index_type': 'HNSW', 'metric_type': 'L2', 'p
arams': {'M': 8, 'efConstruction': 200}}}, {'varchar_1': {}}] (base.py:456)
[2023-08-15 09:25:28,962 - INFO - fouram]: [CommonCases] Prepare index HNSW done. (common_cases.py:99)
[2023-08-15 09:25:28,972 - INFO - fouram]: [Base] Params of index: [{'float_vector': {'index_type': 'HNSW', 'metric_type': 'L2', 'p
arams': {'M': 8, 'efConstruction': 200}}}, {'varchar_1': {}}] (base.py:456)
[2023-08-15 09:25:28,972 - INFO - fouram]: [Base] Start build scalar index of varchar_1 (base.py:449)
On Linux, read() (and similar system calls) will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred. (This is true on both 32-bit and 64-bit systems.). so String index failed when read index data from fd.
https://github.com/milvus-io/milvus/blob/27fe2a45a1d1a8de47ed2409b4a367b0d7a054a1/internal/core/src/index/StringIndexMarisa.cpp#L132-L138
server：
fouramf-7jnr8-79-3106-etcd-0 1/1 Running 0 40m [10.104.19.189](http://10.104.19.189/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-etcd-1 1/1 Running 0 39m [10.104.4.39](http://10.104.4.39/) 4am-node11 <none> <none>
fouramf-7jnr8-79-3106-etcd-2 1/1 Running 0 39m [10.104.24.45](http://10.104.24.45/) 4am-node29 <none> <none>
fouramf-7jnr8-79-3106-milvus-datacoord-774bc75d4b-ns75z 1/1 Running 0 40m [10.104.19.178](http://10.104.19.178/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-milvus-datanode-6b69d8cbd9-qqtrc 1/1 Running 1 (35m ago) 39m [10.104.19.184](http://10.104.19.184/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-milvus-indexcoord-f9b5c6675-bk7xg 1/1 Running 1 (35m ago) 40m [10.104.19.176](http://10.104.19.176/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-milvus-indexnode-74d9cf9db6-vb4zg 0/1 Running 1 (30s ago) 39m [10.104.17.208](http://10.104.17.208/) 4am-node23 <none> <none>
fouramf-7jnr8-79-3106-milvus-proxy-58bdbcc475-xg594 1/1 Running 1 (35m ago) 40m [10.104.19.179](http://10.104.19.179/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-milvus-querycoord-74cd6fd64d-r9l79 1/1 Running 1 (35m ago) 40m [10.104.13.221](http://10.104.13.221/) 4am-node16 <none> <none>
fouramf-7jnr8-79-3106-milvus-querynode-7d77f6bf84-lx5fb 1/1 Running 0 39m [10.104.4.37](http://10.104.4.37/) 4am-node11 <none> <none>
fouramf-7jnr8-79-3106-milvus-rootcoord-565bff96f8-jbqh8 1/1 Running 1 (35m ago) 40m [10.104.19.177](http://10.104.19.177/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-minio-0 1/1 Running 0 39m [10.104.19.193](http://10.104.19.193/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-minio-1 1/1 Running 0 39m [10.104.14.51](http://10.104.14.51/) 4am-node18 <none> <none>
fouramf-7jnr8-79-3106-minio-2 1/1 Running 0 39m [10.104.5.233](http://10.104.5.233/) 4am-node12 <none> <none>
fouramf-7jnr8-79-3106-minio-3 1/1 Running 0 39m [10.104.24.47](http://10.104.24.47/) 4am-node29 <none> <none>
fouramf-7jnr8-79-3106-pulsar-bookie-0 1/1 Running 0 40m [10.104.19.191](http://10.104.19.191/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-pulsar-bookie-1 1/1 Running 0 39m [10.104.12.106](http://10.104.12.106/) 4am-node17 <none> <none>
fouramf-7jnr8-79-3106-pulsar-bookie-2 1/1 Running 0 39m [10.104.5.234](http://10.104.5.234/) 4am-node12 <none> <none>
fouramf-7jnr8-79-3106-pulsar-bookie-init-ch4rp 0/1 Completed 0 40m [10.104.13.219](http://10.104.13.219/) 4am-node16 <none> <none>
fouramf-7jnr8-79-3106-pulsar-broker-0 1/1 Running 0 40m [10.104.19.180](http://10.104.19.180/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-pulsar-proxy-0 1/1 Running 0 39m [10.104.19.183](http://10.104.19.183/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-pulsar-pulsar-init-drzwd 0/1 Completed 0 40m [10.104.13.220](http://10.104.13.220/) 4am-node16 <none> <none>
fouramf-7jnr8-79-3106-pulsar-recovery-0 1/1 Running 0 40m [10.104.19.181](http://10.104.19.181/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-pulsar-zookeeper-0 1/1 Running 0 40m [10.104.19.190](http://10.104.19.190/) 4am-node28 <none> <none>
fouramf-7jnr8-79-3106-pulsar-zookeeper-1 1/1 Running 0 39m [10.104.12.109](http://10.104.12.109/) 4am-node17 <none> <none>
fouramf-7jnr8-79-3106-pulsar-zookeeper-2 1/1 Running 0 38m [10.104.13.225](http://10.104.13.225/) 4am-node16 <none> <none>
image：2.2.0-20230822-408fad8f
client log：
2023-08-22 09:18:16,734 - INFO - fouram]: [Base] Start release collection fouram_daK3KQfH (base.py:303)
[2023-08-22 09:18:16,735 - INFO - fouram]: [Base] Start build index of HNSW for collection fouram_daK3KQfH, params:{'index_type': '
HNSW', 'metric_type': 'L2', 'params': {'M': 8, 'efConstruction': 200}} (base.py:444)
[2023-08-22 09:19:27,762 - INFO - fouram]: [Time] Index run in 71.0262s (api_request.py:45)
[2023-08-22 09:19:27,762 - INFO - fouram]: [CommonCases] RT of build index HNSW: 71.0262s (common_cases.py:97)
[2023-08-22 09:19:27,770 - INFO - fouram]: [Base] Params of index: [{'varchar_1': {}}, {'float_vector': {'index_type': 'HNSW', 'met
ric_type': 'L2', 'params': {'M': 8, 'efConstruction': 200}}}] (base.py:458)
[2023-08-22 09:19:27,770 - INFO - fouram]: [CommonCases] Prepare index HNSW done. (common_cases.py:100)
[2023-08-22 09:19:27,777 - INFO - fouram]: [Base] Params of index: [{'float_vector': {'index_type': 'HNSW', 'metric_type': 'L2', 'p
arams': {'M': 8, 'efConstruction': 200}}}, {'varchar_1': {}}] (base.py:458)
[2023-08-22 09:19:27,778 - INFO - fouram]: [Base] Start build scalar index of varchar_1 (base.py:451)
[2023-08-22 09:21:16,547 - INFO - fouram]: [Time] Index run in 108.7684s (api_request.py:45)
[2023-08-22 09:21:16,547 - INFO - fouram]: [CommonCases] RT of build scalar index varchar_1: 108.7684s (common_cases.py:124)
[2023-08-22 09:21:16,554 - INFO - fouram]: [Base] Collection:fouram_daK3KQfH indexes (base.py:466)
[2023-08-22 09:21:16,554 - INFO - fouram]: [CommonCases] Prepare scalars ['varchar_1'] index done. (common_cases.py:126)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
@yanliang567 Can you ask the test students to confirm whether this problem still exists? /assign @yanliang567
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.