stopping pool

[OmarZaki96]

I am using ProcessPool with 12 workers and 50 tasks; the pool is being used over and over to perform long tasks with timeout.

I want to be able to stop the pool at any time by stopping the processes involved; However, I am facing a weird situation:

1. When the pool is starting (started less that 12 processes), triggering ProcessPool.stop() actually stops the processes involved.
2. When the pool already started (12 processes started), triggering ProcessPool.stop() does nothing, and the program is stuck at ProcessPool.join()

This is the running Pool code:

import pebble
pool = pebble.ProcessPool(12)
def evaluate(x,output):
    raw_result = pool.map(partial(objective, **kwargs), x,
                                      timeout=300,
                                      chunksize=1)
    iterator = raw_result.result()
    outcomes = []
    while True:
        try:
            result = next(iterator)
            outcomes.append(result)
        except StopIteration:
            break
        except: #in case of timeout, just send None
            outcomes.append(None)
            output[0] = outcomes

objective is just a long function that evaluates a results and sends it back; kwargs is defined key words for objective function evaluation; x is the iterator of the inputs (of size 50)

This is the function I use for interrupting the pool:

def stop_pool(pool):
    pool.close()
    pool.stop()
    pool.join()

Also, tried with x of size 12 or lower, works perfectly; for any size higher than 12, not working and gets stuck

[noxdafox]

Please provide a full working example which reproduces the above. It is not clear from the provided code how the issue arises.

It is not clear, for example, when the stop_pool is called. Is it happening in a separate thread? Is it happening within the same thread?

[OmarZaki96]

I apologize for not providing enough information on the issue, and for not responding to your comment. I really appreciate your help on this matter.

I know the issue is very old, but I am facing it again.

What I need is: How can I immediately terminate a pool after calling Pool.map as it can be done in multiprocessing using Pool.terminate(), but in Pebble as ProcessPool.terminate function does not exist

[noxdafox]

As the documentation states, the stop method will terminate abruptly all ongoing tasks. You need to call join to ensure all resources are cleaned up correctly afterwards.

The difference between ProcessPool.close and ProcessPool.stop is the former will let all enqueued jobs to finish whereas the latter will interrupt immediately their execution. In both cases, you need to call ProcessPool.join afterwards.

Calling both as you are doing in your example is redundant.

[OmarZaki96]

import pebble
import time

def task(parameter):
    while True:
        pass

if __name__ == '__main__':
    pool = pebble.ProcessPool(4)
    PoolProcess = pool.map(task, range(100))

    print(1)
    time.sleep(10)
    print(2)
    pool.stop()
    pool.join()
    print(3)

The previous code can reproduce the problem; the code never prints "3" as it never reaches it; my environment is Windows 10, Python 3.8.10 and Pebble 4.6.3

[noxdafox]

When I first tested your code I made a mistake myself. I can now reproduce the issue. I will try to understand what's going on.

[noxdafox]

Commit https://github.com/noxdafox/pebble/commit/7b1264298ec1fb4eb99841288abdd11d3ddcefca fixes the issue. It will be released in the next version v5.0.0.

[OmarZaki96]

Thanks a lot for your effort. I really appreciate it.

[noxdafox]

Issue resolved with release 5.0.0.

Thanks for reporting it!