[Bug]: The data in released partition could be searched without flush()

[binbinlv]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: master latest
- Deployment mode(standalone or cluster):both
- MQ type(rocksmq, pulsar or kafka):    all
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus 2.2.7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The data in released partition could be searched if has no flush operation

before release partition:
[0, 1428, 592, 4120, 3163, 2414, 581, 545, 1244, 4708]
after release partition:
[0, 1428, 592, 4120, 3163, 2414, 581, 545, 1244, 4708]

Expected Behavior

The data in released partition could not be searched even has no flush operation and report not loaded error.

Steps To Reproduce

1. create collection with two partitions
2. insert both partitions
3. create index
4. collection load
5. search
6. release one partition
7. search

from pymilvus import CollectionSchema, FieldSchema
from pymilvus import Collection
from pymilvus import connections
from pymilvus import DataType
from pymilvus import Partition
from pymilvus import utility

connections.connect()

dim = 128
int64_field = FieldSchema(name="int64", dtype=DataType.INT64, is_primary=True)
float_field = FieldSchema(name="float", dtype=DataType.FLOAT)
bool_field = FieldSchema(name="bool", dtype=DataType.BOOL)
string_field = FieldSchema(name="string", dtype=DataType.VARCHAR, max_length=65535)
float_vector = FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
schema = CollectionSchema(fields=[int64_field, float_field, bool_field, float_vector])

collection = Collection("test_search_collection_binbin_tmp_4", schema=schema)


collection.create_partition(partition_name="search_partition_1", description="search partition")
par = collection.partitions
import numpy as np
nb=5000
import random
vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
res = collection.insert([[i for i in range(nb)], [np.float32(i) for i in range(nb)], [np.bool_(i) for i in range(nb)], vectors])
res = collection.insert([[i for i in range(nb)], [np.float32(i) for i in range(nb)], [np.bool_(i) for i in range(nb)], vectors], par[1].name)
#collection.flush()

index_param = {"index_type": "HNSW", "metric_type": "L2", "params": {"M": 48, "efConstruction": 500}}
collection.create_index("float_vector", index_param, index_name="index_name_1")

collection.load()
par[1].name
limit = 10
nq = 1
default_search_params = {"metric_type": "L2", "params": {"ef": 500}}
res = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "int64 >= 0", [par[1].name])
print(res[0].ids)

par[1].release()
res = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "int64 >= 0", [par[1].name])
print(res[0].ids)

Milvus Log

No response

Anything else?

No response

[binbinlv]

if call collection.flush() before, the result could not be searched, but still not report error, the result is empty.

[xiaofan-luan]

should be fixed in 2.3

[bigsheeper]

@binbin should be fixed， please help to make a verify

[bigsheeper]

/unssign /assign @binbinlv

[binbinlv]

Milvus: master-20230509-f6d3b4f7 Fixed.

>>> par[1].release()
>>> res = collection.search(vectors[:nq], "float_vector", default_search_params, limit, "int64 >= 0", [par[1].name])
RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=partition=[441346631963472220]: partition not loaded)>, <Time:{'RPC start': '2023-05-09 16:33:17.114195', 'RPC error': '2023-05-09 16:33:17.243954'}>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 666, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 522, in search
    return self._execute_search_requests(requests, timeout, round_decimal=round_decimal, auto_id=auto_id, **kwargs)
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 491, in _execute_search_requests
    raise pre_err
  File "/Users/binbin/milvus-test-env-3.8/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 482, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=partition=[441346631963472220]: partition not loaded)>