eventmesh icon indicating copy to clipboard operation
eventmesh copied to clipboard

[Enhancement] Define and standardize some common configurations for all Sources

Open cnzakii opened this issue 5 months ago • 0 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

Enhancement Request

In the various implementations of Source, there are often some common configurations scattered across different places, such as the capacity of the storage queue, the number of elements to poll each time, etc. Additionally, in some source implementations, the poll method can block for up to several seconds in the worst-case scenario, reaching maxBatchSize * pollTimeout. Therefore, it is necessary to define common parameters that support configuration and also limit the maximum waiting time for poll.

Describe the solution you'd like

  1. Added a PollConfig configuration class that includes parameters for capacity, maxBatchSize, and maxWaitTime, and incorporated it as a variable in SourceConfig to support user configuration with default values.
  2. Replaced the fixed constants for capacity, maxBatchSize, and maxWaitTime in the original source implementations with the corresponding variables from PollConfig.
  3. Optimized some poll methods to ensure the maximum waiting time does not exceed maxWaitTime, preventing blocking for several seconds.

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

cnzakii avatar Sep 08 '24 17:09 cnzakii