[Bug]: The query count(*) does not decrease after dropping partition

[ThreadDao]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: master-20230411-ad7f3d4c
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): pulsar 
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.4.0.dev4
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

1. create collection -> insert 3000 entities into _default partition
2. flush -> index -> load collection
3. create partition p1 -> insert 3000 into p1
4. query count() return 30002
5. drop p1 partition
6. count() return 30002 ??

Case:

    def test_count_alias_insert_delete_drop(self):
        """
        target: test count after alias insert and load
        method: 1. init collection
                2. alias insert more entities
                3. count and alias count
        expected: verify count
        """
        # create -> insert -> index -> load
        collection_w = self.init_collection_general(insert_data=True)[0]

        # create alias

        alias = cf.gen_unique_str("alias")
        self.utility_wrap.create_alias(collection_w.name, alias)
        collection_w_alias = self.init_collection_wrap(name=alias)

        # new insert partitions and count
        p_name = "p_alias"
        collection_w_alias.create_partition(p_name)
        collection_w_alias.insert(cf.gen_default_dataframe_data(start=ct.default_nb), partition_name=p_name)
        collection_w_alias.query(expr=default_expr, output_fields=[ct.default_count_output],
                                 check_task=CheckTasks.check_query_results,
                                 check_items={exp_res: [{count: ct.default_nb * 2}]})

        # alias drop partition
        collection_w_alias.drop_partition(p_name)
        res, _ = collection_w_alias.has_partition(p_name)
        assert res is False
        collection_w_alias.query(expr=default_expr, output_fields=[ct.default_count_output],
                                 check_task=CheckTasks.check_query_results,
                                 check_items={exp_res: [{count: ct.default_nb}]})

        # alias delete and count
        collection_w_alias.delete(f"{ct.default_int64_field_name} in {[i for i in range(ct.default_nb)]}")
        collection_w_alias.query(expr=default_expr, output_fields=[ct.default_count_output],
                                 check_task=CheckTasks.check_query_results,
                                 check_items={exp_res: [{count: 0}]})

        collection_w_alias.drop()

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @longjiquan /unassign

[longjiquan]

/assign @bigsheeper

[longjiquan]

partition can be seen by the query cluster even after the partition has been droppped, maybe we need to translate the query on collection to query on the partitions in proxy? @bigsheeper

[bigsheeper]

I prefer adding the constraint that error will be returned at the attempt to drop the partition when the partition is/are already loaded. Users need to release partitions before dropping.

[bigsheeper]

Added a constraint of releasing partitions before dropping, please modify the case and release it first @ThreadDao

[bigsheeper]

/assign @ThreadDao /unassign

[ThreadDao]

/assign @bigsheeper image: 'master-20230519-c75e7a5d'

create partition failed:

[2023-05-19 15:50:02,674 - DEBUG - ci_test]: (api_request)  : [Collection.create_partition] args: ['p_alias_JFlF19DE', ''], kwargs: {} (api_request.py:56)
[2023-05-19 15:50:12,873 - ERROR - ci_test]: Traceback (most recent call last):
  File "/Users/nausicca/projects/milvus/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
    res = func(*args, **_kwargs)
  File "/Users/nausicca/projects/milvus/tests/python_client/utils/api_request.py", line 57, in api_request
    return func(*arg, **kwargs)
  File "/Users/nausicca/.virtualenvs/milvus/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 834, in create_partition
    return Partition(self, partition_name, description=description, **kwargs)
  File "/Users/nausicca/.virtualenvs/milvus/lib/python3.8/site-packages/pymilvus/orm/partition.py", line 43, in __init__
    conn.create_partition(self._collection.name, self.name, **kwargs)
  File "/Users/nausicca/.virtualenvs/milvus/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
    raise e
  File "/Users/nausicca/.virtualenvs/milvus/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
    return func(*args, **kwargs)
  File "/Users/nausicca/.virtualenvs/milvus/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
    ret = func(self, *args, **kwargs)
  File "/Users/nausicca/.virtualenvs/milvus/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
    raise e
  File "/Users/nausicca/.virtualenvs/milvus/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
    return func(self, *args, **kwargs)
  File "/Users/nausicca/.virtualenvs/milvus/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 294, in create_partition
    raise MilvusException(response.error_code, response.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=sync new partition failed, reason: failed to sync new created partition, err=context deadline exceeded)>
 (api_request.py:39)
[2023-05-19 15:50:12,873 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=sync new partition failed, reason: failed to sync new created partition, err=context deadline exceeded)> (api_request.py:40)
FAILED

Case:

    def test_count_alias_insert_delete_drop(self):
        """
        target: test count after alias insert and load
        method: 1. init collection
                2. alias insert more entities
                3. count and alias count
        expected: verify count
        """
        # create -> insert -> index -> load
        collection_w = self.init_collection_general(insert_data=True)[0]
    
        # create alias
    
        alias = cf.gen_unique_str("alias")
        self.utility_wrap.create_alias(collection_w.name, alias)
        collection_w_alias = self.init_collection_wrap(name=alias)
    
        # new insert partitions and count
        p_name = cf.gen_unique_str("p_alias")
        collection_w_alias.create_partition(p_name)

[bigsheeper]

should be fixed, please help to verify

/assign @ThreadDao

[bigsheeper]

/assign @ThreadDao

[longjiquan]

/unassign