added processes arg to executor call

[stevenmikes]

Using a distributed job submission platform to submit the function evaluation calls to a cluster. It seemed that the number of evaluations run in parallel was lower than what I set it to with the "batch" parameter. Reading through the Python doc for multiprocessing.Pool, the default for the processes argument is the number of cores on the local machine. Setting processes=batch solved the problem, allowing the number of evaluations set to run in parallel.

[paulknysh]

So, I was thinking about your change - here is the problem. executor has parameter processes only if it's Pool (default executor). But code in general case accepts any executor from user, including custom ones. Those executors might not have processes parameter (or it would be called something else).. Do you agree?

Do you think it would be possible to refactor your change such that it only affects default executor and not all executors? Perhaps by adding processes argument into get_default_executor() function.

[stevenmikes]

Yes I see what you mean. A silly fix would be to check if the executor is the Pool class before trying to instantiate it with the processes argument. Alternately you could decree that executors must accept a processes argument. Maybe you can think of a better solution?

On Mon, Jun 8, 2020 at 9:29 PM Paul Knysh [email protected] wrote:

So, I was thinking about your change - here is the problem. executor has parameter processes only if it's Pool (default executor). But code in general case accepts any executor from user, including custom ones. Those executors might not have processes parameter (or it would be called something else).. Do you agree?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/paulknysh/blackbox/pull/22#issuecomment-640974269, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHYBAORJZNRVTBRE3AXSSDRVWGBLANCNFSM4NYUHLTA .