ipython
`concurrent.futures.ProcessPoolExecutor ` miss print
Hi, I have a jupyter notebook cell, codes show below.
from concurrent.futures import ProcessPoolExecutor, wait
import time
executor = ProcessPoolExecutor(4)
def is_odd(ns, i):
print('get', ns, i)
# time.sleep(10)
return [bool(n & 1) for n in ns]
and another cell
futures = []
i = 0
batch = []
for x in [1,3,5,6,7,8,9,4,12,13]:
batch.append(x)
if len(batch) == 3:
futures.append(executor.submit(is_odd, batch, i))
i += 1
batch = []
wait(futures)
print('done')
I run cell#1 first, then cell#2. Things go on well. However when I run cell#2 again, there is only 'done' printed.
If I merge these 2 cells (or put executor = ProcessPoolExecutor(4) in cell#2), each cell run will print out info in is_odd and everything is right. I uncomment the time.sleep line to check the program has execute the is_odd function.
Mac 10.14/pyenv installed python 3.7.1/ipykernel 5.1.0/jupyter 1.0.0
Run in command line program ipython is ok.
Actually should not work, because it is not designed to. If in jupyter notebook you could do following:
Cell1:
fn="test.py"
with open(fn, "w") as f: f.write(
"""from concurrent.futures import ProcessPoolExecutor, wait
import time
executor = ProcessPoolExecutor(4)
def is_odd(ns, i):
print('get', ns, i)
# time.sleep(10)
return [bool(n & 1) for n in ns]
def main():
futures = []
batch = []
for i, x in enumerate([1,3,5,6,7,8,9,4,12,13]):
batch.append(x)
if len(batch) == 3:
futures.append(executor.submit( is_odd, batch, i))
batch = []
wait(futures)
for f in futures:
print(f.result())
print('done')
if __name__ == '__main__':
main()"""
)
Cell2:
!python $fn
Output:
[True, True, True] [False, True, False] [True, False, False] done get [6, 7, 8] 5 get [1, 3, 5] 2 get [9, 4, 12] 8
