Multiple/concurrent runs at once

[nelinnemann]

Hi

First off thanks for a simple to use optimization library.

Is it possible to add multiple/concurrent runs at the same time? I do a lot of CAD-CFD optimization and the runs tend to be quite long. Not computational intensive as everything is run on a HPC queueing system (SGE).

I would like to instantiate X number of concurrent runs to allow for some speedup of the solution. You could also say that I want to call the optimization problem/function multiple times with different input at the same time.

[Wbec]

Although I have not yet used it, the PoolEvaluator should do this. There is an example for it [here].(https://github.com/Project-Platypus/Platypus/blob/87e744e09618bda46ee80f0e117337b6e582c588/examples/simple_mpi.py) It looks like there are a few different evaluators in platypus/evaluator.py which support different types of multiprocessing/asynchronous behavior.

[whhxp]

I use Platypus to optimize electric motors by using GetDP, a FEM solver. Now I run the FEM solver on a Cloud computing platform (QingCloud) in Hong Kong with 10 instances. I just simply use the Cloud Platform's API to get the IP and port, then call Paramiko to send and receive files and execute commands. A zeroMQ based resource allocator in Python is needed to prevent this 'producer–consumer problem' from deadlock.

[khoopes]

I have followed the pattern here ant it seems to work very well:

https://github.com/Project-Platypus/Platypus/blob/87e744e09618bda46ee80f0e117337b6e582c588/examples/simple_parallel.py

it is limited in that it can only execute on the local machine platypus is running. At least I think so. Someone correct me if I am wrong as that would be amazing if it could somehow connect to remote hosts to execute.

[dhadka]

Hi @khoopes, that example uses the ProcessPoolEvaluator which only distributes the work locally. To use remote hosts, you would first need to find a distributed computing library that supports remote hosts (e.g., see https://wiki.python.org/moin/ParallelProcessing) and setup the remote nodes.

Ideally, the library chosen supports one of the submission patterns supported by one of Platypus' evaluators. For example, MapEvaluator takes a map-like function as an argument and will call that map function to distribute the evaluations. You could use the Python map function, concurrent.futures map, or another map-like function provided by a third-party library. We also have SubmitEvaluator, ApplyEvaluator, and PoolEvaluator that support other patterns.

As an example, https://github.com/Project-Platypus/Platypus/blob/87e744e09618bda46ee80f0e117337b6e582c588/examples/simple_mpi.py demonstrates using PoolEvaluator with a pool object provided by mpi4py.

[khoopes]

@dhadka, Thank you for your reply. I will look into using a distributed library. That could really speed up the project I am working on.

[github-actions[bot]]

This issue is stale and will be closed soon. If you feel this issue is still relevant, please comment to keep it active. Please also consider working on a fix and submitting a PR.