The queue is growing as big as the memory!

[selfinpireself]

I have created a producer and several comsumer threads. Producer add work to the threads using thread_add_work function. Consumers threadpool threads do their job.

Problem is: if number of thread is 5 and producer is 100 times faster than the consumer, then queue size is just increasing.

[Pithikos]

I am not sure if this should be solved from the thread pool side. It is possible but it would make everything too complex for no good reason imo:

• the user would need to specify max jobs when creating a thread pool, which defeats the aim to keep things simple and easy
• if this is implemented with add_work blocking, it makes it counter-intuitive. Many (new) users would find it odd that this function just "hangs" there some times.
• this can be implemented with returning QUEUE_FULL when the user tries to add_work on an already full queue. The problem with this is that the user will have then to loop somehow, checking until the queue is not full. The user can however already do that without the added complexity to the thread pool. (And that brings to my suggested solution below.)

I think a simpler solution is to simply check how much work there already is in the queue before adding new tasks.

Pseudocode:

thpool = thpool_init(4)
..
while (thpool.jobqueue.len <10) {
    sleep(1)
}
thpool.add_work(a)

[dvhh]

While assisting someone online with the same issue, I noticed that thpool is an opaque pointer, so the suggested solution is no longer viable.

the library should provide a function that return the job queue length

[Pithikos]

Just for posterity, here is a reproducing example:

int main(int argc, char *argv[]){
        threadpool thpool = thpool_init(4);
        printf("%d\n", thpool->jobqueue_p.len );
        return 0;
}

[Pithikos]

Probable solution: http://stackoverflow.com/questions/38714429/out-of-memory-in-a-thread-pool-in-c-linux