Speed (bounded_pool_executor slower)

[SaschaHeyer]

Hi

do you have any idea why BoundedThreadPoolExecutor is slower as ThreadPoolExecutor? Machine has 4 CPUs ThreadPoolExecutor executor is using by default 5*CPU = 20 which equals the BoundedThreadPoolExecutor max_workers.

Before

with concurrent.futures.ThreadPoolExecutor() as executor:
        future_to_feature = {executor.submit(extract_features, filename): filename for filename in files}
        for future in concurrent.futures.as_completed(future_to_feature):
        ...

After

with BoundedThreadPoolExecutor(max_workers=20) as worker:
    for file in files:
    ...

[GabrielBigardi]

Hello, i'm not the owner of this repository but this is totally normal, what happens is that unlike ThreadPoolExecutor, BoundedPoolExecutor will only put a net worker when another one is finished, this avoid the memory leaking problem that ThreadPoolExecutor still has, so you wan't to make sure that all the workers finish as soon as possible, if you have some kind of timeout in your function for example make sure to put it as low as you can.