How stop pool with KeyboardInterrupt?

[vitidev]

Does not work

  try:
         results = executor.map(say_hello, range(50))
  except KeyboardInterrupt:
         executor.shutdown()

When ctrl-c I see

Hello from 11044 with arg 7
  File "c:\python\lib\concurrent\futures\_base.py", line 598, in result_iterator
    yield fs.pop().result()
  File "c:\python\lib\concurrent\futures\_base.py", line 435, in result
    return self.__get_result()
  File "c:\python\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 25, in <module>
    n_workers = len(set(results))
  File "c:\python\lib\site-packages\loky\process_executor.py", line 794, in _chain_from_iterable_of_lists
    for element in iterable:
  File "c:\python\lib\concurrent\futures\_base.py", line 603, in result_iterator
    future.cancel()
  File "c:\python\lib\concurrent\futures\_base.py", line 354, in cancel
    with self._condition:
  File "c:\python\lib\threading.py", line 241, in __enter__
    return self._lock.__enter__()
KeyboardInterrupt
Hello from 25208 with arg 8
Hello from 16140 with arg 9

and execution continues

[tomMoral]

Hello,

You need to use the argument kill_workers=True in shutdown to immediately stop the workers on shutdown.

In [1]: from loky import get_reusable_executor 
   ...: executor = get_reusable_executor(2)
   ...:
   ...:def say_hello(i): 
   ...:     import time 
   ...:     time.sleep(2) 
   ...:     print("hello {i}") 
   ...:   
   ...: try: 
   ...:      results = list(executor.map(say_hello, range(50))) 
   ...: except KeyboardInterrupt: 
   ...:      executor.shutdown(kill_workers=True) 
   ...:                                                                                                                                                                                                                                      
hello {i}
hello {i}
^C
In [2]:

[vitidev]

@tomMoral

kill_workers=True in shutdown to immediately stop the workers on shutdown.

I do not want to kill workers. I want to stop gracefully

It doesn't work anyway. Execution doesn't even go into except block and shutdown never called.

Win 10 x64. Python 3.7.4

[tomMoral]

I am not sure what behaviour you expect. could you please share a reproducible example? In your first snippet, you call executor.map which is non-blocking in the try/catch block, so the KeyboardInterupt is not catch in this block.

The kill_workers argument stop the workers immediately without waiting for task completion. If you do not use this, you will need to wait for the completion of any task which is already in queued.

[vitidev]

 results = list(...) 
 +
 executor.shutdown(kill_workers=True) 

works for hard interruption. Thanks.

But what if I want to gracefully interrupt - let me finish the current executing tasks (not queued) and exit.

I don’t understand what executor.shutdown doing without kill_workers=True.

[tomMoral]

I think to be able to interrupt only the work that were not schedule, you need to use the cancel method from Future. It seems that by the time Ctrl+C is pressed, 18 tasks have been scheduled. I think improving the cancellation policy would prove valuable and it would be a good new feature.

In [29]: 
    ...: def say_hello(i): 
    ...:     import time 
    ...:     time.sleep(1) 
    ...:     print(f"hello {i}") 
    ...:     return i 
    ...:  
    ...: from loky import get_reusable_executor 
    ...: from loky import CancelledError 
    ...: executor = get_reusable_executor(2) 
    ...: try: 
    ...:      futures = [executor.submit(say_hello, i) for i in range(50)] 
    ...:      results = [r.result() for r in futures] 
    ...: except KeyboardInterrupt: 
    ...:     print([r.cancel() for r in futures]) 
    ...:     executor.shutdown(wait=True) 
    ...:     results = [] 
    ...:     for r in futures: 
    ...:         try: 
    ...:             results.append(r.result()) 
    ...:         except (CancelledError, KeyboardInterrupt): 
    ...:             results.append('cancel') 
    ...: print(results)                                                                                                                                                                                                                       
hello 0
hello 1
^C[False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]
hello 4
hello 5
hello 6
hello 7
hello 8
hello 9
hello 10
hello 11
hello 12
hello 13
hello 14
hello 15
hello 16
hello 17
hello 18
[0, 1, 'cancel', 'cancel', 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel', 'cancel']

One mysterious think that I find buggy is the fact that tasks 2/3 are marked as cancelled. It seems that they receive the KeyboardInterrupt signal, which feels like a bug.

[vitidev]

I think improving the cancellation policy would prove valuable and it would be a good new feature.

if it is possible.

I am develop on C#. There are "thread abort" is so deprecated, that in the new dotnetcore it is not even implemented. Cancellation pattern is preferred. But I understand the difference between threads and processes and that this feature can increase overhead.

[ogrisel]

One mysterious think that I find buggy is the fact that tasks 2/3 are marked as cancelled. It seems that they receive the KeyboardInterrupt signal, which feels like a bug.

Indeed, it would be great to turn your example into a non-regression failing test as a WIP PR.