[Bug]: [Nightly] Load balance failed reporting "segment 436130972956669944 not found in source node 6"

[NicoYuan1986]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:master(5141e05)
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):2.2.0.dev30
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Load balance failed reporting "segment 436130972956669944 not found in source node 6":

[2022-09-21T01:05:42.164Z] [2022-09-21 00:40:40 - ERROR - pymilvus.decorators]: RPC error: [load_balance], <MilvusException: (code=1, message=failed to balance segments, err=segment 436130972956669944 not found in source node 6)>, <Time:{'RPC start': '2022-09-21 00:40:40.710390', 'RPC error': '2022-09-21 00:40:40.711552'}> (decorators.py:100)
[2022-09-21T01:05:42.165Z] [2022-09-21 00:40:40 - ERROR - ci_test]: Traceback (most recent call last):
[2022-09-21T01:05:42.165Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
[2022-09-21T01:05:42.165Z]     res = func(*args, **_kwargs)
[2022-09-21T01:05:42.165Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
[2022-09-21T01:05:42.165Z]     return func(*arg, **kwargs)
[2022-09-21T01:05:42.165Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/utility.py", line 543, in load_balance
[2022-09-21T01:05:42.165Z]     return _get_connection(using).load_balance(collection_name, src_node_id, dst_node_ids, sealed_segment_ids, timeout=timeout)
[2022-09-21T01:05:42.165Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 101, in handler
[2022-09-21T01:05:42.165Z]     raise e
[2022-09-21T01:05:42.165Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 97, in handler
[2022-09-21T01:05:42.165Z]     return func(*args, **kwargs)
[2022-09-21T01:05:42.165Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 127, in handler
[2022-09-21T01:05:42.165Z]     ret = func(self, *args, **kwargs)
[2022-09-21T01:05:42.165Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 79, in handler
[2022-09-21T01:05:42.165Z]     raise e
[2022-09-21T01:05:42.165Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2022-09-21T01:05:42.165Z]     return func(self, *args, **kwargs)
[2022-09-21T01:05:42.165Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1042, in load_balance
[2022-09-21T01:05:42.165Z]     raise MilvusException(status.error_code, status.reason)
[2022-09-21T01:05:42.165Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to balance segments, err=segment 436130972956669944 not found in source node 6)>
[2022-09-21T01:05:42.165Z]  (api_request.py:39)
[2022-09-21T01:05:42.165Z] [2022-09-21 00:40:40 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=failed to balance segments, err=segment 436130972956669944 not found in source node 6)> (api_request.py:40)

Expected Behavior

PASSED

Steps To Reproduce

nightly: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/111/pipeline/171

Milvus Log

1. FAILED timeline: [2022-09-21T00:41:04.304Z] [gw0] [ 99%] FAILED testcases/test_utility.py::TestUtilityAdvanced::test_load_balance_normal
2. collection name: utility_DYiCyEtH
3. milvus log: artifacts-milvus-distributed-pulsar-nightly-111-pymilvus-e2e-logs.tar.gz

Anything else?

No response

[yanliang567]

/assign @sunby /unassign

[NicoYuan1986]

Recently, the issue occurs in https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/125/pipeline/187.

1. FAILED timeline: [2022-09-28T01:37:36.184Z] [gw3] [ 99%] FAILED testcases/test_utility.py::TestUtilityAdvanced::test_load_balance_normal
2. collection name: utility_DYLjHVdM
3. milvus log: artifacts-milvus-distributed-pulsar-nightly-125-pymilvus-e2e-logs.tar.gz
4. error message:

[2022-09-28 01:38:33,456 - ERROR - ci_test]: Traceback (most recent call last):
  File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
    res = func(*args, **_kwargs)
  File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
    return func(*arg, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/utility.py", line 543, in load_balance
    return _get_connection(using).load_balance(collection_name, src_node_id, dst_node_ids, sealed_segment_ids, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 113, in handler
    raise e
  File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 139, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 89, in handler
    raise e
  File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 51, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1034, in load_balance
    raise MilvusException(status.error_code, status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to balance segments, err=segment 436289534272515381 not found in source node 1)>
 (api_request.py:39)
[2022-09-28 01:38:33,456 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=failed to balance segments, err=segment 436289534272515381 not found in source node 1)> (api_request.py:40)