milvus icon indicating copy to clipboard operation
milvus copied to clipboard

Published 20 hours ago • verified publisher icon milvus-io

[Bug]: RESTFUL, auto-incremented primary keys require special handling

Open PowderLi opened this issue 1 year ago • 3 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Environment

- Milvus version: 2.3.4
- Deployment mode(standalone or cluster): both

Current Behavior

  1. create a collection with auto-incremented primary key
  2. use restful api: insert to insert new data which wasn't filled with primary key, it return err, like this
[error="unable to cast \"\" of type string to int64: invalid parameter[expected=Int64][actual=]"]
[errorVerbose="unable to cast \"\" of type string to int64: invalid parameter[expected=Int64][actual=]\n(1) attached stack trace\n  -- stack trace:\n  | 
github.com/milvus-io/milvus/pkg/util/merr.WrapErrParameterInvalid[...]\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/merr/utils.go:744\n  | 
github.com/milvus-io/milvus/internal/distributed/proxy/httpserver.checkAndSetData\n  | \t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/httpserver/utils.go:244\n  | 
github.com/milvus-io/milvus/internal/distributed/proxy/httpserver.(*HandlersV1).insert\n  | \t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/httpserver/handler_v1.go:654\n  | 
github.com/gin-gonic/gin.(*Context).Next\n  | \t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\n  | 
github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).startHTTPServer.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:195\n  | 
github.com/gin-gonic/gin.(*Context).Next\n  | \t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\n  | 
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n  | \t/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102\n  | 
github.com/gin-gonic/gin.(*Context).Next\n  | \t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\n  | 
github.com/gin-gonic/gin.LoggerWithConfig.func1\n  | \t/go/pkg/mod/github.com/gin-gonic/[email protected]/logger.go:240\n  | 
github.com/gin-gonic/gin.(*Context).Next\n  | \t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174\n  | 
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n  | \t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620\n  | 
github.com/gin-gonic/gin.(*Engine).ServeHTTP\n  | \t/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576\n  | 
net/http.serverHandler.ServeHTTP\n  | \t/usr/local/go/src/net/http/server.go:2936\n  | 
net/http.(*conn).serve\n  | \t/usr/local/go/src/net/http/server.go:1995\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) unable to cast \"\" of type string to int64\nWraps: (3) invalid parameter[expected=Int64][actual=]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

Expected Behavior

can insert new data without primary key successfully using restful api, if the primary key is auto-incremented

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

PowderLi avatar Jan 15 '24 11:01 PowderLi

/assign @PowderLi /unassign

yanliang567 avatar Jan 15 '24 12:01 yanliang567 

[2024-01-16 17:57:30 - INFO - ci_test]: method: post, url: http://10.104.21.149:19530/v1/vector/collections/create, cost time: 0.040171146392822266, header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'a97ac51c-b455-11ee-a2e7-acde48001122'}, payload: {"collectionName": "test_collection_2024_01_16_17_57_29_365434rBtJegQa", "dimension": 128, "primaryField": "url", "vectorField": "embedding", "dbName": "prod"}, response: {"code":200,"data":{}} (milvus.py:14)
[2024-01-16 17:57:30 - INFO - ci_test]: method: get, url: http://10.104.21.149:19530/v1/vector/collections/describe, cost time: 0.024565935134887695, header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'a981114c-b455-11ee-a2e7-acde48001122'}, payload: null, response: {"code":200,"data":{"collectionName":"test_collection_2024_01_16_17_57_29_365434rBtJegQa","description":"","enableDynamicField":true,"fields":[{"autoId":true,"description":"","name":"url","primaryKey"...ription":"","name":"embedding","primaryKey":false,"type":"FloatVector(128)"}],"indexes":[{"fieldName":"embedding","indexName":"vector_idx","metricType":"L2"}],"load":"LoadStateLoading","shardsNum":1}} (milvus.py:14)
[2024-01-16 17:57:30 - INFO - ci_test]: rsp: {'code': 200, 'data': {'collectionName': 'test_collection_2024_01_16_17_57_29_365434rBtJegQa', 'description': '', 'enableDynamicField': True, 'fields': [{'autoId': True, 'description': '', 'name': 'url', 'primaryKey': True, 'type': 'Int64'}, {'autoId': False, 'description': '', 'name': 'embedding', 'primaryKey': False, 'type': 'FloatVector(128)'}], 'indexes': [{'fieldName': 'embedding', 'indexName': 'vector_idx', 'metricType': 'L2'}], 'load': 'LoadStateLoading', 'shardsNum': 1}} (test_vector_operations.py:40)
[2024-01-16 17:57:30 - INFO - ci_test]: body size: 0.003162384033203125 MB (test_vector_operations.py:50)
[2024-01-16 17:57:30 - ERROR - ci_test]: method: post, url: http://10.104.21.149:19530/v1/vector/insert, cost time: 0.027186870574951172, header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'a9852a7a-b455-11ee-a2e7-acde48001122'}, payload: {"collectionName": "test_collection_2024_01_16_17_57_29_365434rBtJegQa", "data": [{"embedding": [0.10503232820987377, 0.10524573273330146, 0.10724004682248506, 0.0741073962974074, 0.038609044157231744, 0.10931114153979368, 0.15557623068620974, 0.01432725918610439, 0.02197866219249412, 0.017727693277791147, 0.025845687606599898, 0.14972004982862272, 0.11239422949587603, 0.07926265188888922, 0.08759827124035152, 0.08784323348046186, 0.09684541967033324, 0.12705489475737516, 0.06040848454182098, 0.14034941482744398, 0.12526207723992616, 0.028241969687915744, 0.01672942385356011, 0.12221269400505728, 0.14807597287275207, 0.1418158505256834, 0.07218966841604807, 0.026522032831382984, 0.05185050313365804, 0.06560472968468906, 0.11530875313479227, 0.0585363084087966, 0.05127294893731234, 0.13059095535530268, 0.03270146707010754, 0.12237238638467877, 0.02024320103432553, 0.05889271110478854, 0.11001402515491387, 0.15129929516739157, 0.04370046027041673, 0.03959578447756104, 0.01723595036923567...32850845428002, 0.004694894200694867, 0.07896870560141372, 0.09025741739187243, 0.027798488890081626, 0.07734476786193972, 0.08343643000369567, 0.10570610549027386, 0.04196930761538659, 0.0932158738346877, 0.12620846839328398, 0.021429717642132046, 0.09010652978556873, 0.05412695453512159, 0.14450465890489816, 0.13282821459031866, 0.009432060140506257, 0.11170154604155254, 0.01823146079478023, 0.09957685930080297, 0.14018120856859612, 0.05230382759592277, 0.05111101823776989, 0.010285409890881982], "uid": 0, "name": "Albert Smith", "address": "663 Abigail Underpass\nRodriguezmouth, WI 97312", "text": "Their safe send defense firm artist say summer. Very modern generation city side rock.\nDoctor single image weight. Phone enjoy particular.", "email": "[email protected]", "phone_number": "(355)707-4219", "json": {"name": "Jordan Harris", "address": "5542 Villegas Overpass Suite 850\nPort Janetport, VT 02431"}, "key35647": "value77008", "key71128": "value70697"}], "dbName": "prod"}, response: {"code":1100,"message":"the number of fields is less than needed: invalid parameter[expected=2][actual=3]"} (milvus.py:17)

create a collection with payload

{
    "collectionName": "test_collection_2024_01_16_17_57_29_365434rBtJegQa",
    "dimension": 128,
    "primaryField": "url",
    "vectorField": "embedding",
    "dbName": "prod"
}

collection describe

{
    "collectionName": "test_collection_2024_01_16_17_57_29_365434rBtJegQa",
    "description": "",
    "enableDynamicField": True,
    "fields": [
        {
            "autoId": True,
            "description": "",
            "name": "url",
            "primaryKey": True,
            "type": "Int64"
        },
        {
            "autoId": False,
            "description": "",
            "name": "embedding",
            "primaryKey": False,
            "type": "FloatVector(128)"
        }
    ],
    "indexes": [
        {
            "fieldName": "embedding",
            "indexName": "vector_idx",
            "metricType": "L2"
        }
    ],
    "load": "LoadStateLoading",
    "shardsNum": 1
}

insert data

[
    {
        "embedding": [
            0.10503232820987377,
            0.10524573273330146,
            0.10724004682248506,
            0.0741073962974074,
            0.038609044157231744,
            0.10931114153979368,
            0.15557623068620974,
            0.01432725918610439,
            0.02197866219249412,
            0.017727693277791147,
            0.025845687606599898,
            0.14972004982862272,
            0.11239422949587603,
            0.07926265188888922,
            0.08759827124035152,
            0.08784323348046186,
            0.09684541967033324,
            0.12705489475737516,
            0.06040848454182098,
            0.14034941482744398,
            0.12526207723992616,
            0.028241969687915744,
            0.01672942385356011,
            0.12221269400505728,
            0.14807597287275207,
            0.1418158505256834,
            0.07218966841604807,
            0.026522032831382984,
            0.05185050313365804,
            0.06560472968468906,
            0.11530875313479227,
            0.0585363084087966,
            0.05127294893731234,
            0.13059095535530268,
            0.03270146707010754,
            0.12237238638467877,
            0.02024320103432553,
            0.05889271110478854,
            0.11001402515491387,
            0.15129929516739157,
            0.04370046027041673,
            0.03959578447756104,
            0.01723595036923567...32850845428002,
            0.004694894200694867,
            0.07896870560141372,
            0.09025741739187243,
            0.027798488890081626,
            0.07734476786193972,
            0.08343643000369567,
            0.10570610549027386,
            0.04196930761538659,
            0.0932158738346877,
            0.12620846839328398,
            0.021429717642132046,
            0.09010652978556873,
            0.05412695453512159,
            0.14450465890489816,
            0.13282821459031866,
            0.009432060140506257,
            0.11170154604155254,
            0.01823146079478023,
            0.09957685930080297,
            0.14018120856859612,
            0.05230382759592277,
            0.05111101823776989,
            0.010285409890881982
        ],
        "uid": 0,
        "name": "Albert Smith",
        "address": "663 Abigail Underpass\nRodriguezmouth, WI 97312",
        "text": "Their safe send defense firm artist say summer. Very modern generation city side rock.\nDoctor single image weight. Phone enjoy particular.",
        "email": "[email protected]",
        "phone_number": "(355)707-4219",
        "json": {
            "name": "Jordan Harris",
            "address": "5542 Villegas Overpass Suite 850\nPort Janetport, VT 02431"
        },
        "key35647": "value77008",
        "key71128": "value70697"
    }
]

but get error

{"code":1100,"message":"the number of fields is less than needed: invalid parameter[expected=2][actual=3]"}

zhuwenxing avatar Jan 16 '24 11:01 zhuwenxing

embedding field is for vector, url is auto id, others are for dynamic fields.

zhuwenxing avatar Jan 16 '24 11:01 zhuwenxing

/unassign /assign @yanliang567 already fixed, need to recheck

PowderLi avatar Mar 01 '24 01:03 PowderLi

/assign @zhuwenxing /unassign

yanliang567 avatar Mar 02 '24 02:03 yanliang567