[Bug]: non-growing segments are not searchable if handed-off after collection.load()

[soothing-rain]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: master
- Deployment mode(standalone or cluster):
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.1.0dev83
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

#BAD CASE If we collection.load() before bulk_load(), search() always returns nothing. The new segments will be added and loaded by QueryNodes. But these newly added segments will not be accepted by the shard leader, no matter how long we wait.

#GOOD CASE However if we do collection.load() AFTER bulk_load(), everything is fine. Because calling collection.load() will trigger SyncReplicaSegments which can do stuff like:

[2022/06/23 23:30:35.594 +08:00] [DEBUG] [impl.go:736] ["Received SyncReplicaSegments request"] [vchannelName=by-dev-rootcoord-dml_10_434110000414326785v0]
[2022/06/23 23:30:35.595 +08:00] [INFO] [shard_cluster.go:278] ["ShardCluster sync segments"] ["replica segments"="[{\"node_id\":2,\"partition_id\":434110000414326786,\"segment_ids\":[434110000428482561]}]"] [state=3]
[2022/06/23 23:30:35.595 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_10_434110000414326785v0] [distribution="[{\"node_id\":2,\"partition_id\":434110000414326786,\"segment_ids\":[434110000428482561]}]"]

#BAD CASE In the bad case, when the collection.load() was called before segments are added. SyncReplicaSegments will be triggered with nothing to sync and will never be trigger afterwards:

[2022/06/24 02:24:18.721 +08:00] [DEBUG] [impl.go:736] ["Received SyncReplicaSegments request"] [vchannelName=by-dev-rootcoord-dml_20_434112733794533377v0]
[2022/06/24 02:24:18.721 +08:00] [INFO] [shard_cluster.go:278] ["ShardCluster sync segments"] ["replica segments"=null] [state=3]
[2022/06/24 02:24:18.721 +08:00] [INFO] [shard_cluster_service.go:142] ["successfully sync segments"] [channel=by-dev-rootcoord-dml_20_434112733794533377v0] [distribution=null]

In this bad case, search will be applied on no segments:

[2022/06/24 02:24:37.384 +08:00] [DEBUG] [impl.go:454] ["Received SearchRequest"] [msgID=434112441141458425] [fromShardLeader=false] [vChannel=by-dev-rootcoord-dml_20_434112733794533377v0] [segmentIDs="[]"] [guaranteeTimestamp=434112738715762689] [timeTravel=18446744073709551615]
[2022/06/24 02:24:37.384 +08:00] [DEBUG] [impl.go:478] ["start do search"] [msgID=434112441141458425] [fromShardLeader=false] [vChannel=by-dev-rootcoord-dml_20_434112733794533377v0] [segmentIDs="[]"]

Expected Behavior

data is searchable no matter load is called before or afterwards.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[soothing-rain]

/assign @congqixia

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[yanliang567]

any updates

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.