[Bug]: find a strange behavior when insert vector

[cydrain]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: Milvus 2.2.2 (0fdc1a04)
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.2.1
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

During debugging #21359, find a strange behavior when insert vectors.

query_node.log

in above log file, it runs the script "21359_create_insert.py" 3 times:

== 1st run =======
CYD - seg id 438290154915564879, 69595
CYD - seg id 438290154915564880, 69686
CYD - seg id 438290154915564941, 69593
CYD - seg id 438290154915564942, 69679
== 2nd run =======
CYD - seg id 438290154915765084, 69595
CYD - seg id 438290154915765085, 69686
CYD - seg id 438290154915965147, 69679
CYD - seg id 438290154915965148, 69593
== 3rd run =======
CYD - seg id 438290154915965248, 209051
== 4th run =======
CYD - seg id 438290154915965359, 69595
CYD - seg id 438290154915965360, 69686
CYD - seg id 438290154916165422, 69593
CYD - seg id 438290154916165423, 69679

Why in the 3rd run, the vectors are not evenly inserted into 4 segments as the 1st run and 2nd run, and the row count is not the sum of all vector rows ? Is this expected ?

Expected Behavior

input vectors should be evenly inserted into 4 segments

Steps To Reproduce

1. add following debug code, then build and launch Milvus

diff --git a/internal/core/src/segcore/segment_c.cpp b/internal/core/src/segcore/segment_c.cpp
index c43804991..b9440586d 100644
--- a/internal/core/src/segcore/segment_c.cpp
+++ b/internal/core/src/segcore/segment_c.cpp
@@ -76,6 +76,15 @@ Search(CSegmentInterface c_segment,
                 dis *= -1;
             }
         }
+        {
+            static std::mutex mu;
+            std::lock_guard<std::mutex> lock(mu);
+            std::cout << "CYD - seg id " << segment->get_segment_id() << ", " << segment->get_row_count() << std::endl;
+            for (int i = 0; i < 3; i++) {
+                std::cout << "  (" << search_result->seg_offsets_[i] << ", " << search_result->distances_[i] << ")" << std::endl;
+            }
+            std::fflush(stdout);
+        }
         *result = search_result.release();
         return milvus::SuccessCStatus();
     } catch (std::exception& e) {

2. run this script several times 21359_create_insert.py.txt

3. check the log file to see how many rows in each segment

### Milvus Log

_No response_

### Anything else?

_No response_

[yanliang567]

/assign @soothing-rain /unassign

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[yanliang567]

keep it active

[xiaofan-luan]

@cydrain any ideas?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.