occurrence uses all the cores

[wangjiawen2013]

trafficstars

Hi, when I ran sq.gr.co_occurrence, it used all the 128 cores of my server, even the n_jobs were set to n_jobs=30, how to limit the number of cores it uses ?

[michalk8]

Hi @wangjiawen2013 , sorry for such a late reply. I will investigate, but I suppose this is due to implicit OMP parallelism and not because of the process parallelism we use explicitly.

[giovp]

hi @wangjiawen2013 ,

is this still an issue? It's probably not dependent on our joblib implementatioon.

[wangjiawen2013]

Yes, and it breaks as "segment fault" when treating some datasets. But when setting numba_parallel=False and n_jobs=1, the problem was solved, and faster than using all the cores.

[giovp]

Thanks for the feedback, if you set numba_parallel=False yet set n_jobs=2, does it use 2 cores correctly?

[wangjiawen2013]

it also worked when numba_parallel=False and n_jobs=2, why ?

[wangjiawen2013]

Sorry, I mean sq.gr.ligrec can also use all cores. But when setting numba_parallel=False and n_jobs=1, the problem was solved, and faster than using all the cores. In the case of sq.gr.co_occurrence, I cannot set numba_paralle, becuase this parameter is not in sq.gr.co_occurrence(). It still used all of the CPU cores.

[giovp]

thanks for trying it out, I will check again with the co_occurrence? Shall we set numba_parallell = False by default @michalk8 ?

[giovp]

hi @wangjiawen2013 , I'll close this issue as it seems it's a matter of setting defaults and am not super sure tbh. thanks for raising this, let's continue conversation if it raises again.