semaphore icon indicating copy to clipboard operation
semaphore copied to clipboard

Add configuration to support free concurrent tasks and optimize git library update performance issues

Open gaoyue1989 opened this issue 1 year ago • 2 comments

In the process of frequently creating tasks in large batches, several problems were discovered:

  1. Concurrent tasks of the same template are not supported
  2. Taskpool is slow to deliver tasks every 5 seconds.
  3. The git library will need to be updated every time a task is executed, but in most cases this is not necessary.

This is controlled through the incremental scheme, and is switched by configuring GitCacheTime, TaskFreeConcurrencyMode, and TaskPoolTickerSec. When not configured, it remains the same as the previous code.

Consider that inventory is created through taskid when using git. It is actually created every time. Currently, GitCacheTime does not control this.

gaoyue1989 avatar Oct 15 '24 09:10 gaoyue1989

Hi @gaoyue1989

Did you test the PR in real tasks?

fiftin avatar Oct 15 '24 10:10 fiftin

Caching doesn't resolve the conflict issue and introduces unexpected behavior. I think we need to check for new commits (without pulling) on each task run. In free concurrency mode, new tasks fail if a new commit appears. Or ignore new commits if active tasks exist. We can add numeric option for this: max_skip_attempts_for_new_commits (name must be changed :D).

fiftin avatar Oct 15 '24 10:10 fiftin