[Bug]: [JSON] Wrong errors when input None in search expression

[binbinlv]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:master-20230515-30415e1b
- Deployment mode(standalone or cluster):both
- MQ type(rocksmq, pulsar or kafka):    all
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.0.dev31
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Not specified JSON field, but it reports Json related errors, it is not valid: expression: int64 == None, error: there is multi json field in schema, need to specified field name

>>> res = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "int64 == None")
RPC error: [search], <MilvusException: (code=1, message=failed to create query plan: cannot parse expression: int64 == None, error: there is multi json field in schema, need to specified field name)>, <Time:{'RPC start': '2023-05-17 17:50:07.394037', 'RPC error': '2023-05-17 17:50:07.475324'}>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 666, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 521, in search
    return self._execute_search_requests(requests, timeout, round_decimal=round_decimal, auto_id=auto_id, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 490, in _execute_search_requests
    raise pre_err
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 481, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to create query plan: cannot parse expression: int64 == None, error: there is multi json field in schema, need to specified field name)>

Expected Behavior

reports the true error if not supports None

Steps To Reproduce

from pymilvus import CollectionSchema, FieldSchema
from pymilvus import Collection
from pymilvus import connections
from pymilvus import DataType
from pymilvus import Partition
from pymilvus import utility
import time

connections.connect()

dim = 128
int64_field = FieldSchema(name="int64", dtype=DataType.INT64, is_primary=True)
float_field = FieldSchema(name="float", dtype=DataType.FLOAT)
bool_field = FieldSchema(name="bool", dtype=DataType.BOOL)
string_field = FieldSchema(name="string", dtype=DataType.VARCHAR, max_length=65535)
json_field = FieldSchema(name="json_field", dtype=DataType.JSON)
json_field_1 = FieldSchema(name="json_field_1", dtype=DataType.JSON)
float_vector = FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
schema = CollectionSchema(fields=[int64_field, float_field, bool_field, json_field, json_field_1, float_vector])
collection = Collection("test_search_collection_multiple_json_0", schema=schema)
import random
nb=3000
vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
import numpy as np
res = collection.insert([[i for i in range(nb)], [np.float32(i) for i in range(nb)], [np.bool_(i) for i in range(nb)], [{"number": i, "string": "milvus", "bool": True, "list": [j for j in range(0, i)], "json": {"number": i, "string": "milvus", "bool": True}} for i in range(0, nb)], [{"number": i, "string": "milvus", "bool": True, "list": [j for j in range(0, i)], "json": {"number": i, "string": "milvus", "bool": True}} for i in range(0, nb)], vectors])
index_param = {"index_type": "IVF_SQ8", "metric_type": "L2", "params": {"nlist": 8192}}
collection.create_index("float_vector", index_param, index_name="index_name_1")
collection.load()
default_search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
limit = 10
nq =1
res = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "int64 == 1")
res = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "int64 == None")

Milvus Log

No response

Anything else?

No response

[binbinlv]

And the same: collection.search(vectors[:nq], "float_vector", default_search_params, limit, "json_field["string"] == None")

>>> res = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "json_field[\"string\"] == None")
RPC error: [search], <MilvusException: (code=1, message=failed to create query plan: cannot parse expression: json_field["string"] == None, error: there is multi json field in schema, need to specified field name)>, <Time:{'RPC start': '2023-05-17 17:59:17.706949', 'RPC error': '2023-05-17 17:59:17.772554'}>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 666, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 521, in search
    return self._execute_search_requests(requests, timeout, round_decimal=round_decimal, auto_id=auto_id, **kwargs)
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 490, in _execute_search_requests
    raise pre_err
  File "/Users/binbin/towhee-test-env-3.8-0.9.0/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 481, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to create query plan: cannot parse expression: json_field["string"] == None, error: there is multi json field in schema, need to specified field name)>

[xiaocai2333]

Fixed the error message, but currently, the use of "None" as a keyword is not supported. In such cases, it will be interpreted as $meta["None"]. so the expression will be parsed in this manner "json_field["string"] == json_field["None"]" . Please note that this format is also not supported.

[xiaocai2333]

@binbinlv

[binbinlv]

@xiaocai2333

you mean the following error?

>>> res = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "json_field['string'] == json_field['None']")
RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Search, QueryNode ID=1, reason=Search 1 failed, reason [UnexpectedError] Assert "unsupported right datatype of compare expr" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/visitors/ExecExprVisitor.cpp:1082
 err %!w(<nil>)
attempt #2:context canceled
)>, <Time:{'RPC start': '2023-05-31 07:10:29.863155', 'RPC error': '2023-05-31 07:10:29.872094'}>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 628, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 543, in search
    return self._execute_search_requests(requests, timeout, round_decimal=round_decimal, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 523, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 514, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Search, QueryNode ID=1, reason=Search 1 failed, reason [UnexpectedError] Assert "unsupported right datatype of compare expr" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/visitors/ExecExprVisitor.cpp:1082
 err %!w(<nil>)
attempt #2:context canceled

[binbinlv]

And

>>> collection.search(vectors[:nq], "float_vector", default_search_params, limit, "json_field['string'] == None")
RPC error: [search], <MilvusException: (code=1, message=failed to create query plan: cannot parse expression: json_field['string'] == None, error: there is no dynamic json field in schema, need to specified field name)>, <Time:{'RPC start': '2023-05-31 07:12:30.082413', 'RPC error': '2023-05-31 07:12:30.087481'}>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 628, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 543, in search
    return self._execute_search_requests(requests, timeout, round_decimal=round_decimal, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 523, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 514, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to create query plan: cannot parse expression: json_field['string'] == None, error: there is no dynamic json field in schema, need to specified field name)

[xiaocai2333]

yes, we currently do not support comparing between two JSON keys. Once we support 'None', it will be used as a keyword for matching.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[xiaocai2333]

/keep it

[binbinlv]

The error now for None is: pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to create query plan: cannot parse expression: int64 == None, error: there is no dynamic json field in schema, need to specified field name)>

It seems that the error message did not show the error is that we do not support None?

[binbinlv]

@xiaocai2333

[xiaocai2333]

The error now for None is: pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to create query plan: cannot parse expression: int64 == None, error: there is no dynamic json field in schema, need to specified field name)>

It seems that the error message did not show the error is that we do not support None?

In this case, we cannot directly report an error saying that None is not supported, because None may be a field, or it may be a key in $meta. And I have fixed the error message to field None not exist

[xiaocai2333]

@binbinlv

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.