milvus icon indicating copy to clipboard operation
milvus copied to clipboard
Published 4 months ago • verified publisher icon milvus-io

[Bug]: [bulk insert v2]In the "list import job" interface, the dbName parameter is not taking effect. Regardless of the different values set, the results returned are the same

Open zhuwenxing opened this issue 2 years ago • 4 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Environment

- Milvus version:master
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

when payload is {}, {"dbName": "bd1"}, {"dbName": "db2"} or {"dbName": "default"}, all results are same

[2024-03-14 17:17:54 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.13.156:19530/v2/vectordb/jobs/import/list, 
cost time: 0.039526939392089844, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bd84db72-e1e3-11ee-93e4-acde48001122'}, 
payload: {}, 
response: {"code":200,"data":{"records":[{"collectionName":"test_collection_2024_03_14_14_44_26_278620UxhOrhNv","jobId":"448368158535519818","progress":0,"reason":"value of field 'book_id' is missed: importing ...bId":"448368158536986792","progress":100,"state":"Completed"},{"collectionName":"test_collection_2024_03_14_17_12_19_069494qwClffvt","jobId":"448368158537216778","progress":100,"state":"Completed"}]}} (milvus.py:17)
[2024-03-14 17:17:54 - INFO - ci_test]: job num: 71 (test_jobs_operation.py:1446)
[2024-03-14 17:17:54 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.13.156:19530 (connectionpool.py:245)
[2024-03-14 17:17:54 - DEBUG - urllib3.connectionpool]: http://10.104.13.156:19530 "POST /v2/vectordb/jobs/import/list HTTP/1.1" 200 None (connectionpool.py:547)
[2024-03-14 17:17:54 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.13.156:19530/v2/vectordb/jobs/import/list, 
cost time: 0.06205582618713379, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bd8b08a8-e1e3-11ee-93e4-acde48001122'}, 
payload: {"dbName": "db1"}, 
response: {"code":200,"data":{"records":[{"collectionName":"test_collection_2024_03_14_16_43_55_344620plDuLUau","jobId":"448368158537133817","progress":100,"state":"Completed"},{"collectionName":"test_collectio...bId":"448368158537084025","progress":100,"state":"Completed"},{"collectionName":"test_collection_2024_03_14_14_57_08_043129wJhtNcTO","jobId":"448368158537030072","progress":100,"state":"Completed"}]}} (milvus.py:17)
[2024-03-14 17:17:54 - INFO - ci_test]: job num: 71 (test_jobs_operation.py:1446)
[2024-03-14 17:17:54 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.13.156:19530 (connectionpool.py:245)
[2024-03-14 17:17:54 - DEBUG - urllib3.connectionpool]: http://10.104.13.156:19530 "POST /v2/vectordb/jobs/import/list HTTP/1.1" 200 None (connectionpool.py:547)
[2024-03-14 17:17:54 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.13.156:19530/v2/vectordb/jobs/import/list, 
cost time: 0.04655003547668457, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bd94b970-e1e3-11ee-93e4-acde48001122'}, 
payload: {"dbName": "db2"}, 
response: {"code":200,"data":{"records":[{"collectionName":"test_collection_2024_03_14_14_50_14_386351qKhyXCic","jobId":"448368158535954139","progress":100,"state":"Completed"},{"collectionName":"test_collectio...bId":"448368158537076712","progress":100,"state":"Completed"},{"collectionName":"test_collection_2024_03_14_16_43_55_344620plDuLUau","jobId":"448368158537133814","progress":100,"state":"Completed"}]}} (milvus.py:17)
[2024-03-14 17:17:55 - INFO - ci_test]: job num: 71 (test_jobs_operation.py:1446)
[2024-03-14 17:17:55 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.13.156:19530 (connectionpool.py:245)
[2024-03-14 17:17:55 - DEBUG - urllib3.connectionpool]: http://10.104.13.156:19530 "POST /v2/vectordb/jobs/import/list HTTP/1.1" 200 None (connectionpool.py:547)
[2024-03-14 17:17:55 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.13.156:19530/v2/vectordb/jobs/import/list, 
cost time: 0.04587435722351074, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bd9c0e32-e1e3-11ee-93e4-acde48001122'}, 
payload: {"dbName": "default"}, 
response: {"code":200,"data":{"records":[{"collectionName":"test_collection_2024_03_14_14_44_34_369948tsqIQuZS","jobId":"448368158535519856","progress":100,"state":"Completed"},{"collectionName":"test_collectio...bId":"448368158537239285","progress":100,"state":"Completed"},{"collectionName":"test_collection_2024_03_14_14_57_08_043129wJhtNcTO","jobId":"448368158537030076","progress":100,"state":"Completed"}]}} (milvus.py:17)
[2024-03-14 17:17:55 - INFO - ci_test]: job num: 71 (test_jobs_operation.py:1446)

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

zhuwenxing avatar Mar 14 '24 09:03 zhuwenxing

/assign @bigsheeper /unassign

yanliang567 avatar Mar 14 '24 14:03 yanliang567

dbName is only utilized for RBAC purposes, and will be fixed by PR https://github.com/milvus-io/milvus/pull/31298

bigsheeper avatar Mar 15 '24 07:03 bigsheeper

/assign @zhuwenxing please help to make a verify

bigsheeper avatar Mar 20 '24 06:03 bigsheeper

@bigsheeper reproduced in 2.4-20240326-35664fa3-amd64

create 2 tasks in db1 create 2 tasks in db2

when listing import jobs, no matter which dbName is, the result always is 4

[2024-03-27 11:11:34 - INFO - ci_test]: all database: ['default'] (testbase.py:132)
[2024-03-27 11:11:34 - INFO - ci_test]: create database: db1 (testbase.py:134)
[2024-03-27 11:11:34 - INFO - ci_test]: all database: ['default', 'db1'] (testbase.py:132)
[2024-03-27 11:11:34 - INFO - ci_test]: create database: db2 (testbase.py:134)
[2024-03-27 11:11:35 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:35 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/collections/create HTTP/1.1" 200 22 (connectionpool.py:547)
[2024-03-27 11:11:35 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/collections/create, 
cost time: 0.3169589042663574, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'b813c686-ebe7-11ee-862f-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_03_27_11_11_34_330056NYGSKnDb", "schema": {"autoId": true, "enableDynamicField": true, "fields": [{"fieldName": "book_id", "dataType": "Int64", "isPrimary": tr...": "book_intro", "dataType": "FloatVector", "elementTypeParams": {"dim": "128"}}]}, "indexParams": [{"fieldName": "book_intro", "indexName": "book_intro_vector", "metricType": "L2"}], "dbName": "db1"}, 
response: {"code":200,"data":{}} (milvus.py:17)
[2024-03-27 11:11:36 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:36 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/collections/create HTTP/1.1" 200 22 (connectionpool.py:547)
[2024-03-27 11:11:36 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/collections/create, 
cost time: 0.27849817276000977, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'b8dd5b90-ebe7-11ee-862f-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_03_27_11_11_34_330056NYGSKnDb", "schema": {"autoId": true, "enableDynamicField": true, "fields": [{"fieldName": "book_id", "dataType": "Int64", "isPrimary": tr...": "book_intro", "dataType": "FloatVector", "elementTypeParams": {"dim": "128"}}]}, "indexParams": [{"fieldName": "book_intro", "indexName": "book_intro_vector", "metricType": "L2"}], "dbName": "db2"}, 
response: {"code":200,"data":{}} (milvus.py:17)
[2024-03-27 11:11:39 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.30.25:9000 (connectionpool.py:245)
[2024-03-27 11:11:40 - DEBUG - urllib3.connectionpool]: http://10.104.30.25:9000 "GET /milvus-bucket?location= HTTP/1.1" 200 0 (connectionpool.py:547)
[2024-03-27 11:11:40 - DEBUG - urllib3.connectionpool]: http://10.104.30.25:9000 "POST /milvus-bucket/bulk_insert_data_1711509097.json?uploads= HTTP/1.1" 200 0 (connectionpool.py:547)
[2024-03-27 11:11:40 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (2): 10.104.30.25:9000 (connectionpool.py:245)
[2024-03-27 11:11:40 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (3): 10.104.30.25:9000 (connectionpool.py:245)
[2024-03-27 11:11:41 - DEBUG - urllib3.connectionpool]: http://10.104.30.25:9000 "PUT /milvus-bucket/bulk_insert_data_1711509097.json?partNumber=3&uploadId=MzMxMTEwNzgtY2QzNy00NjE4LTk5YjAtMjQxNmJjYWM4ZDBhLjFhMmQyMjlhLTBkNDktNGViOC1iMWJhLTgyZTA5NzdkYzg5Nw HTTP/1.1" 200 0 (connectionpool.py:547)
[2024-03-27 11:11:41 - DEBUG - urllib3.connectionpool]: http://10.104.30.25:9000 "PUT /milvus-bucket/bulk_insert_data_1711509097.json?partNumber=2&uploadId=MzMxMTEwNzgtY2QzNy00NjE4LTk5YjAtMjQxNmJjYWM4ZDBhLjFhMmQyMjlhLTBkNDktNGViOC1iMWJhLTgyZTA5NzdkYzg5Nw HTTP/1.1" 200 0 (connectionpool.py:547)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: http://10.104.30.25:9000 "PUT /milvus-bucket/bulk_insert_data_1711509097.json?partNumber=1&uploadId=MzMxMTEwNzgtY2QzNy00NjE4LTk5YjAtMjQxNmJjYWM4ZDBhLjFhMmQyMjlhLTBkNDktNGViOC1iMWJhLTgyZTA5NzdkYzg5Nw HTTP/1.1" 200 0 (connectionpool.py:547)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: http://10.104.30.25:9000 "POST /milvus-bucket/bulk_insert_data_1711509097.json?uploadId=MzMxMTEwNzgtY2QzNy00NjE4LTk5YjAtMjQxNmJjYWM4ZDBhLjFhMmQyMjlhLTBkNDktNGViOC1iMWJhLTgyZTA5NzdkYzg5Nw HTTP/1.1" 200 0 (connectionpool.py:547)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/jobs/import/create HTTP/1.1" 200 50 (connectionpool.py:547)
[2024-03-27 11:11:42 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/jobs/import/create, 
cost time: 0.11962509155273438, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bc2ba400-ebe7-11ee-862f-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_03_27_11_11_34_330056NYGSKnDb", "files": [["bulk_insert_data_1711509097.json"]], "dbName": "db1"}, 
response: {"code":200,"data":{"jobId":"448661240466124804"}} (milvus.py:17)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/jobs/import/create HTTP/1.1" 200 50 (connectionpool.py:547)
[2024-03-27 11:11:42 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/jobs/import/create, 
cost time: 0.1380617618560791, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bc3e11bc-ebe7-11ee-862f-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_03_27_11_11_34_330056NYGSKnDb", "files": [["bulk_insert_data_1711509097.json"]], "dbName": "db1"}, 
response: {"code":200,"data":{"jobId":"448661240466124807"}} (milvus.py:17)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/jobs/import/create HTTP/1.1" 200 50 (connectionpool.py:547)
[2024-03-27 11:11:42 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/jobs/import/create, 
cost time: 0.13666486740112305, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bc535798-ebe7-11ee-862f-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_03_27_11_11_34_330056NYGSKnDb", "files": [["bulk_insert_data_1711509097.json"]], "dbName": "db2"}, 
response: {"code":200,"data":{"jobId":"448661240466124811"}} (milvus.py:17)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/jobs/import/create HTTP/1.1" 200 50 (connectionpool.py:547)
[2024-03-27 11:11:42 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/jobs/import/create, 
cost time: 0.1778411865234375, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bc68531e-ebe7-11ee-862f-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_03_27_11_11_34_330056NYGSKnDb", "files": [["bulk_insert_data_1711509097.json"]], "dbName": "db2"}, 
response: {"code":200,"data":{"jobId":"448661240466124815"}} (milvus.py:17)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/jobs/import/list HTTP/1.1" 200 561 (connectionpool.py:547)
[2024-03-27 11:11:42 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/jobs/import/list, 
cost time: 0.08938288688659668, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bc83b8fc-ebe7-11ee-862f-acde48001122'}, 
payload: {}, 
response: {"code":200,"data":{"records":[{"collectionName":"test_collection_2024_03_27_11_11_34_330056NYGSKnDb","jobId":"448661240466124804","progress":0,"state":"Pending"},{"collectionName":"test_collection_20...nDb","jobId":"448661240466124811","progress":0,"state":"Pending"},{"collectionName":"test_collection_2024_03_27_11_11_34_330056NYGSKnDb","jobId":"448661240466124815","progress":0,"state":"Pending"}]}} (milvus.py:17)
[2024-03-27 11:11:42 - INFO - ci_test]: job num: 4 (test_jobs_operation.py:1509)
[2024-03-27 11:11:42 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:43 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/jobs/import/list HTTP/1.1" 200 573 (connectionpool.py:547)
[2024-03-27 11:11:43 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/jobs/import/list, 
cost time: 0.5196182727813721, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bc931536-ebe7-11ee-862f-acde48001122'}, 
payload: {"dbName": "db1"}, 
response: {"code":200,"data":{"records":[{"collectionName":"test_collection_2024_03_27_11_11_34_330056NYGSKnDb","jobId":"448661240466124804","progress":10,"state":"Importing"},{"collectionName":"test_collection...jobId":"448661240466124811","progress":10,"state":"Importing"},{"collectionName":"test_collection_2024_03_27_11_11_34_330056NYGSKnDb","jobId":"448661240466124815","progress":10,"state":"Importing"}]}} (milvus.py:17)
[2024-03-27 11:11:43 - INFO - ci_test]: job num: 4 (test_jobs_operation.py:1509)
[2024-03-27 11:11:43 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:43 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/jobs/import/list HTTP/1.1" 200 573 (connectionpool.py:547)
[2024-03-27 11:11:43 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/jobs/import/list, 
cost time: 0.2700080871582031, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bce28a12-ebe7-11ee-862f-acde48001122'}, 
payload: {"dbName": "db2"}, 
response: {"code":200,"data":{"records":[{"collectionName":"test_collection_2024_03_27_11_11_34_330056NYGSKnDb","jobId":"448661240466124804","progress":10,"state":"Importing"},{"collectionName":"test_collection...jobId":"448661240466124811","progress":10,"state":"Importing"},{"collectionName":"test_collection_2024_03_27_11_11_34_330056NYGSKnDb","jobId":"448661240466124815","progress":10,"state":"Importing"}]}} (milvus.py:17)
[2024-03-27 11:11:43 - INFO - ci_test]: job num: 4 (test_jobs_operation.py:1509)
[2024-03-27 11:11:43 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.26.207:19530 (connectionpool.py:245)
[2024-03-27 11:11:43 - DEBUG - urllib3.connectionpool]: http://10.104.26.207:19530 "POST /v2/vectordb/jobs/import/list HTTP/1.1" 200 573 (connectionpool.py:547)
[2024-03-27 11:11:43 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.26.207:19530/v2/vectordb/jobs/import/list, 
cost time: 0.2100389003753662, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'bd0bf9c4-ebe7-11ee-862f-acde48001122'}, 
payload: {"dbName": "default"}, 
response: {"code":200,"data":{"records":[{"collectionName":"test_collection_2024_03_27_11_11_34_330056NYGSKnDb","jobId":"448661240466124804","progress":10,"state":"Importing"},{"collectionName":"test_collection...jobId":"448661240466124811","progress":10,"state":"Importing"},{"collectionName":"test_collection_2024_03_27_11_11_34_330056NYGSKnDb","jobId":"448661240466124815","progress":10,"state":"Importing"}]}} (milvus.py:17)
[2024-03-27 11:11:43 - INFO - ci_test]: job num: 4 (test_jobs_operation.py:1509)

zhuwenxing avatar Mar 27 '24 03:03 zhuwenxing

@bigsheeper any updates?

yanliang567 avatar Oct 09 '24 06:10 yanliang567

@bigsheeper any updates?

Currently, the dbName parameter for import is not supported. It's not an urgent issue, and we can add support for it later.

bigsheeper avatar Oct 09 '24 06:10 bigsheeper