taskiq
taskiq copied to clipboard
Potential Conflict Between `max-tasks-per-child` and Concurrent Task Processing?
Background: I'm exploring ways to prevent memory leaks in TaskIQ workers and noticed two relevant parameters:
max-async-tasks(default: 100) - Sets the maximum concurrent tasks per workermax-tasks-per-child- Recycles workers after completing N tasks Hypothetical Scenario: If I setmax-tasks-per-child=1(to aggressively recycle workers) while keeping the defaultmax-async-tasks=100, I'm concerned this might create an edge case:- A worker starts processing multiple tasks concurrently (up to 100)
- When any single task completes, the worker might meet the recycling condition
- This could potentially terminate other in-progress tasks Questions for Clarification:
- Is this understanding correct, or does TaskIQ have safeguards against this?
- How does the system count completed tasks when dealing with concurrent execution?
- Does it wait for all concurrent tasks to finish before counting?
- Or does completion of any single task trigger the count?
- If this is a valid concern, would you recommend:
- Setting
max-async-tasks=1when usingmax-tasks-per-child? - Or is there another preferred pattern for memory leak prevention?