[Bug]: Continued writing at a QPS of about 4000 led to reaching the backlogSize limit, and further writing was no longer possible.

[linggong2011]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:2.2.5
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[Bug]: Continued writing at a QPS of about 4000 led to reaching the backlogSize limit, and further writing was no longer possible.

Expected Behavior

It is able to continue writing and functioning without interruption.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

/assign @bigsheeper

[yanliang567]

@linggong2011 One quick question: are you talking about 4000 insert requests per second or 4000 vectors inserted per second? and what is the dim of the vectors？ Also Could you please refer this script to export the whole Milvus logs for investigation?

/assign @linggong2011

[JackLCL]

After offline discussions, it was found that the issue was caused by not carrying the configuration parameters of the related dependent components during the upgrade. After investigation, the parameter "subscriptionExpirationTimeMinutes: "30"" was missing in Pulsar. After configuring this parameter in Pulsar, the backlog size returned to normal.

[xiaofan-luan]

should we just set to very low value by default? say 3 minutes

[LoveEachDay]

By default the subscription expiration check interval is 5 minutes, so change to 3 minutes may not take effect.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.