nalepae
Hangs on Completion When nb_workers is Too High
Debian 9.9 pandarallel 1.4.5 pyhton 3.7.5
I'm applying in parallel some string comparisons:
track_matches = isrcs_to_match_by_title["title_cleaned"].parallel_apply(
lambda title_cleaned: tracks.index[tracks["name_cleaned"].values == title_cleaned])
It's not always reproducible. In some runs it will work and others it won't.
Setting progress_bar=True or False doesn't seem to affect it.
The higher the number of processes, the less likely it seems to complete. When I set nb_workers=8, it always completes. 24 sometimes completes, 96 never completes.
On completion the processes all die out (none of them are being used), and the program never continues.
I'm actually no longer certain it's caused by nb_workers > 8. When it was reproducing for me it was clearly happening with high number of workers and not at low number of workers. Now, on the same machine just minutes later with the same data and code, it's not reproducing regardless of number of workers.
I assume this is no longer an issue. If not, feel free to reopen with a complete minimal example that reproduces the problem.
