[Bug]: [Nightly] Fail to balance segments reporting a service internal error

[NicoYuan1986]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: be1470a
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.3.5rc8
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Fail to balance segments reporting a service internal error.

[2024-01-18T22:46:21.331Z] [2024-01-18 22:44:43 - DEBUG - ci_test]: (api_request)  : [load_balance] args: ['utility_x2FNX6KS', 2, [4], [447115841529299971], 20, 'default'], kwargs: {} (api_request.py:62)
[2024-01-18T22:46:21.331Z] [2024-01-18 22:44:43 - ERROR - pymilvus.decorators]: RPC error: [load_balance], <MilvusException: (code=5, message=failed to balance segments: service internal error: task with the same segment exists)>, <Time:{'RPC start': '2024-01-18 22:44:43.752692', 'RPC error': '2024-01-18 22:44:43.755083'}> (decorators.py:128)
[2024-01-18T22:46:21.331Z] [2024-01-18 22:44:43 - ERROR - ci_test]: Traceback (most recent call last):
[2024-01-18T22:46:21.331Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-01-18T22:46:21.331Z]     res = func(*args, **_kwargs)
[2024-01-18T22:46:21.331Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-01-18T22:46:21.331Z]     return func(*arg, **kwargs)
[2024-01-18T22:46:21.331Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/utility.py", line 600, in load_balance
[2024-01-18T22:46:21.331Z]     return _get_connection(using).load_balance(
[2024-01-18T22:46:21.331Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 129, in handler
[2024-01-18T22:46:21.331Z]     raise e from e
[2024-01-18T22:46:21.331Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 125, in handler
[2024-01-18T22:46:21.331Z]     return func(*args, **kwargs)
[2024-01-18T22:46:21.331Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 164, in handler
[2024-01-18T22:46:21.331Z]     return func(self, *args, **kwargs)
[2024-01-18T22:46:21.331Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 104, in handler
[2024-01-18T22:46:21.331Z]     raise e from e
[2024-01-18T22:46:21.331Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 68, in handler
[2024-01-18T22:46:21.331Z]     return func(*args, **kwargs)
[2024-01-18T22:46:21.331Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1352, in load_balance
[2024-01-18T22:46:21.331Z]     check_status(status)
[2024-01-18T22:46:21.331Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-01-18T22:46:21.331Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-01-18T22:46:21.331Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=5, message=failed to balance segments: service internal error: task with the same segment exists)>
[2024-01-18T22:46:21.331Z]  (api_request.py:45)
[2024-01-18T22:46:21.331Z] [2024-01-18 22:44:43 - ERROR - ci_test]: (api_response) : <MilvusException: (code=5, message=failed to balance segments: service internal error: task with the same segment exists)> (api_request.py:46)

Expected Behavior

load balance successfully

Steps To Reproduce

No response

Milvus Log

1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/73/pipeline/237
2. log: artifacts-milvus-distributed-pulsar-nightly-73-pymilvus-e2e-logs.tar.gz
3. collection name: utility_x2FNX6KS
4. failed time: [2024-01-18T22:44:44.396Z] [gw5] [ 99%] FAILED testcases/test_utility.py::TestUtilityAdvanced::test_load_balance_normal

Anything else?

No response

[yanliang567]

/assign @sunby /unassign

[NicoYuan1986]

The issue has not reproduced for a long time. So close it first. I will reopen it once it occurs. latest image: master-20240321-09281a07-amd64