Error in multiprocessing on Windows.

[Brunohermans]

Dear developers,

Recently I tried to integrate an agentpy model with the EMA workbench, the example that was on the documentation was helpful, and I was able to make that run (this example: https://agentpy.readthedocs.io/en/latest/guide_ema.html?highlight=ema ).

However, when I tried to run the model on multiple cores, using the MultiprocessingEvaluater (see https://emaworkbench.readthedocs.io/en/latest/ema_documentation/em_framework/evaluators.html ). I was not able to run my model anymore. The code that I used to run the my model, as well as the example model was:

from ema_workbench import (MultiprocessingEvaluator, ema_logging)
from ema_problem_definitions import ema_problem

model = ema_problem(1)
with MultiprocessingEvaluator(model) as evaluator:
    experiments, outcomes = evaluator.perform_experiments(scenarios=5)

Wherin ema_problem(1) is a function to define the agenpty model as a function. The function is implemented as follows:

from ema_workbench import (Model, RealParameter,
                           ScalarOutcome,  
                           Constant, IntegerParameter)
from model import EtmEVsModel

def ema_problem(problem):
    
    # convert model to function
    EtmEVs = EtmEVsModel.as_function()

    
    model = Model('EtmEVsModel', function=EtmEVs)

I left out the specification of the uncertainties, constants and levers as my post would be rather long if I were to include that. This code resulted in the following error: Can't pickle local object 'Model.as_function.<locals>.agentpy_model_as_function

This error does not occur when I try to run the multiprocessing evaluator with a basic example model from the EMA workbench. However, it does occur with my custom model and the example model from the documentation. Also, it is noteworthy to mention is that I've tested this code on two different windows 11 machines, which both had the same error. However, a Linux machine that we have tried worked and was able to execute the code.

*I don't really have much experience writing an issue, this is my first attempt in my career as a coder. So if something is wrong in the way I write this issue, please let me know.

[jofmi]

Hi @Brunohermans, Python multiprocessing often leads to issues on windows. Have you tried the ipyparallel evaluator?

The multiprocessing feature of agentpy uses joblib, which works quite well (see here). Maybe @quaquel could consider a new evaluator for ema_workbench based on that library.

[Brunohermans]

Hi @JoelForamitti! Thanks for your useful suggestion to use ipyparallel. After configuring the ipyparallel, which requires additional setup, we've got the same error:

Can't pickle local object 'Model.as_function.<locals>.agentpy_model_as_function

It seems like this does not depend upon the type of multiprocessing evaluator one uses in the EMA workbench. We did not try the ipyparallel evaluator in Linux, as we just assume that it works like the multiprocessing evaluator.

However, I can confirm that the multiprocessing feature of Agentpy works in both ubuntu and windows 11. However, if one uses this feature, it's harder to integrate with the EMA workbench.

[jofmi]

Hmm ok, thanks for checking. I am not sure how to solve this. One option would be to to open an issue at EMA workbench, asking if it is possible to add another evaluator based on joblib. The ipyparallel evaluator has just 40 lines of code, so it should be quite doable to add another (see here).

[quaquel]

Both multiprocessing and ipyparallel rely on pickle for interprocess communciation. Most likely something in your agentpy model is not pickleable at the moment of parallelization.

The workbench largely gets arround this by seperating model initialization into two steps, with the second being called only after parallelization.

[jofmi]

Thanks for the input @quaquel! Do you think it would be possible to add another evaluator based on joblib?

[quaquel]

I am not familiar with joblib so there would be a bit of a learning curve.

I actually think that a simpler solution is possible here by not using the as_function approach, but wrapping the model slightly differently through subclassing one of the Model classes that come with the workbench. This is how I generally implement connectors to other modeling/simulation tools (e.g., Vensim, Excell, Simio, NetLogo).

[jofmi]

Ok, thanks for the suggestions! I will look into both options