queick icon indicating copy to clipboard operation
queick copied to clipboard

cannot limit the workers

Open jmriego opened this issue 4 years ago • 1 comments

Hi! sorry if I'm misunderstanding this completely but I'm trying to create a way to queue jobs in my Raspberry Pi and only have them running sequentially. My test code is like this:

# test.py
from queick import JobQueue
from jobfunc import function
from time import time

q = JobQueue()
for i in range(1,50):
    q.enqueue(function, args=(f"hello{i}",), priority=i, max_workers=1)
# jobfunc.py
def function(arg):
    time.sleep(5)
    print(arg)

I was expecting this code would show me one hello{i} every 5 seconds as I'm only passing max_workers as 1 and all jobs have a different priority. Instead, after 5 seconds all jobs finish at the same time. Any idea of what am I doing wrong? Thanks!

jmriego avatar Dec 06 '21 19:12 jmriego

I think I found the issue in the code. The QueueManager is creating a new executor for every job received so each job has its own executor with as many cores as the Raspberry Pi has. The ThreadPoolExecutor should only be created once before starting the watch loop

jmriego avatar Dec 07 '21 12:12 jmriego