apache
[Bug] The consumption speed of routine load job shake severely if the FDB pressure is high
Search before asking
- [x] I had searched in the issues and found no similar issues.
Version
cloud
What's Wrong?
At present, for a single routine load job, the begin and commit transactions in the job are serial, which brings the problem that if RPC is slow, the task scheduling cycle will become particularly long.
For example, in the case of the actual problems encountered, each job has 24 concurrent tasks. When the FDB pressure caused by backup is high, resulting in RPC delay to 1 second, the scheduling interval of tasks becomes (begin transaction delay+commit delay) * concurrency, which is about 2 minutes, causing severe jitter in the consumption of routine load. Some users have particularly high concurrency, such as reaching 512, which exponentially amplifies the impact.
Based on this issue, if parallel begin and commit transactions can be performed, the task scheduling interval will be reduced to (the delay of begin transaction +the delay of commit transaction).
What You Expected?
No obvious shaking.
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
