taskiq icon indicating copy to clipboard operation
taskiq copied to clipboard

Potential Conflict Between `max-tasks-per-child` and Concurrent Task Processing?

Open jiangbaihe opened this issue 6 months ago • 0 comments

Background: I'm exploring ways to prevent memory leaks in TaskIQ workers and noticed two relevant parameters:

  1. max-async-tasks (default: 100) - Sets the maximum concurrent tasks per worker
  2. max-tasks-per-child - Recycles workers after completing N tasks  Hypothetical Scenario: If I set max-tasks-per-child=1 (to aggressively recycle workers) while keeping the default max-async-tasks=100, I'm concerned this might create an edge case:
  3. A worker starts processing multiple tasks concurrently (up to 100)
  4. When any single task completes, the worker might meet the recycling condition
  5. This could potentially terminate other in-progress tasks  Questions for Clarification:
  6. Is this understanding correct, or does TaskIQ have safeguards against this?
  7. How does the system count completed tasks when dealing with concurrent execution?
  • Does it wait for all concurrent tasks to finish before counting?
  • Or does completion of any single task trigger the count?
  1. If this is a valid concern, would you recommend:
  • Setting max-async-tasks=1 when using max-tasks-per-child?
  • Or is there another preferred pattern for memory leak prevention?

jiangbaihe avatar Jun 03 '25 00:06 jiangbaihe