EMAworkbench
EMAworkbench copied to clipboard
quaquel
MESA example
Dear all, I am trying to connect the EMA workbench framework to my MESA model. I have followed the tutorial for python. Yet, I encounter some errors and I cannot make it run ('Simple model' object is not subscriptable- when running perform_experiment). Is there any chance you could upload an example with a known MESA model (like the MESA version of Wolf Sheep https://github.com/projectmesa/mesa/tree/main/examples/wolf_sheep ). Since I am still a beginner in Python I am pretty sure it is a very stupid mistake that I am doing to not run the model. Thanks in advance A
are you trying to extend an AbstractModel class, or are you simply wrapping the mesa model run within a python function?
if you have some code to share, I am happy to take a look.
In fact, I have been meaning to write some documentation on how to write your own connectors by using MESA as an example.
Hi, thanks for the answer. Actually I only tried to apply the simplest example to my model mesa object: but of course it does not work so I don' really know what to do. An example to see how to build your connector with MESA would be great. Thanks
Hi Alessandro, Our files from SocSimFest are available here: https://github.com/BROSE-Uninc/SSF2021 and include an example with the EMA Workbench and a Mesa model. Or are you trying to do something else?
@quaquel I can add this as an MVP example for EMA and mesa if you would like, connected just through Python.
Hi, thanks, that is great! Is there any chance i could find the recording somewhere? it would be very helpful
Unfortunately, there were some technical issues on ESSA's side, so the recording is not available.
Okay thanks. I managed to connect my model to EMA. Yet it is very slow, I remembered there was an option for multiprocessing, is there any example of how to implement that?
Also, which SA (that in in EMA workbench), would you recommend for a non-ergodic model? (2 final possible equilibria)
Thanks in advance!
depends a bit on whether your model needs access to the file system and how you made it work with MESA.
If done correctly, all you need to do is
from ema_workbench import MultiprocessingEvaluator
with MultiprocessingEvaluator(model) as evaluator:
experiments, outcomes = evaluator.perform_experiments(100)
Thanks. I actually did adapt my model to be the same as the tutorial:
This is my code:
def model_EEG_ABM(F1=5,
F2 =25,
H =350 ,
B =1 ,
T = 2,
S = 0,
seed = 4,
steps=200):
import model
EEG_ABM = model.KSModel(F1=50,
F2 =250,
H =3500 ,
B =1 ,
T = 0.02 ,
S =0,
seed = 4)
EEG_ABM.run_model(steps)
outcomes = EEG_ABM.datacollector.get_model_vars_dataframe()
return {'TIME' : list(range(steps + 1)),
"Real GDP coastal" : outcomes["Real GDP coastal"].tolist(),
"Real GDP internal" : outcomes["Real GDP internal"].tolist(),
"Unemployment rate coastal" : outcomes["Unemployment rate coastal"].tolist(),
"Unemployment rate internal" : outcomes["Unemployment rate internal"].tolist()}
Importantly, if i test it like this it works fine:
F1=50
F2 =250
H =3500
B = 1
T = 2
S =0
seed = 4
steps = 10
model_EEG_ABM(F1= F1,
F2 =F2,
H =H ,
B = B ,
T = T ,
S =S ,
seed = seed,
steps= steps)
Outputs: Out[18]: {'TIME': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'Real GDP coastal': [125, 0, 500, 1000, 1770, 2898, 3008, 3010, 3000, 3000, 2750], 'Real GDP internal': [125, 0, 500, 1000, 1772, 2846, 2750, 2750, 2500, 2500, 2127], 'Unemployment rate coastal': [0.0, 1.0, 0.57, 0.17, 0.0, 0.11, 0.14, 0.14, 0.14, 0.14, 0.21], 'Unemployment rate internal': [0.0, 1.0, 0.57, 0.19, 0.0, 0.11, 0.21, 0.21, 0.29, 0.29, 0.0]}
However I get some issues when I run the experiments:
model = ReplicatorModel('KSModel', function=model_EEG_ABM)
model.uncertainties = [RealParameter("T", 0.01, 0.1),
RealParameter("S", 0.02, 0.15)]
model.constants = [Constant('F1', F1),
Constant('F2', F2),
Constant('H', H),
Constant('B', B),
Constant('seed', seed),
Constant('steps', steps)]
model.outcomes = [TimeSeriesOutcome('TIME'),
TimeSeriesOutcome('Real GDP coastal'),
TimeSeriesOutcome('Unemployment rate coastal')]
model.replications = 1
results = perform_experiments(model, 10, uncertainty_sampling='sobol')
I get this error:
File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 230, in _store_outcomes self.results[outcome][case_id, ] = outcome_res
KeyError: 'TIME'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "
File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/evaluators.py", line 468, in perform_experiments evaluator.evaluate_experiments(scenarios, policies, callback)
File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/evaluators.py", line 230, in evaluate_experiments callback(experiment, outcomes)
File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 265, in call self._store_outcomes(experiment.experiment_id, outcomes)
File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 244, in _store_outcomes self.results[outcome][:] = np.nan
ValueError: cannot convert float NaN to integer
Any clue? Thanks in advance for the support I am almost there.
which version of the workbench are you using? I recently addressed a bug related to this.
the problem is presently TIME is a range of integers. If this were a range of floats, the problem would be gone.
more fundamentally, this is a bug that needs fixing on the workbench side.
I have changed time in: 'TIME' : [float(i) for i in range(steps +1)] But still I get the same error. I have also tried to remove 'TIME' so using only the other variables but still I get the same error.
Then I changed also the other variables into floats and it works. So the bottom line is: only outcomes with float. thanks!
and the fact that outcomes need to be floats is indeed the bug that needs fixing
I see, perfect! Thanks. I still got problems with parallel runs but I guess is something on my end that I got to fix.
regarding the parallel running a few questions because the code seems fine.
- what OS are you using?
- Are you in a .py file or a notebook?
- What is the nature of the error?
A fix is being tested for the other bug, so hopefully it will be more permanently fixed in the future.
At the bottom of the code I have added:
from ema_workbench import MultiprocessingEvaluator
with MultiprocessingEvaluator(model) as evaluator:
experiments, outcomes = evaluator.perform_experiments(10)
- I am using macOS Big Sur 11.3.1
- I am in a .py file
- The error is that I keep receiving this error in a loop:
[MainProcess/INFO] pool started [MainProcess/INFO] performing 10 scenarios * 1 policies * 1 model(s) = 10 experiments /usr/local/lib/python3.9/site-packages/ema_workbench/em_framework/optimization.py:48: ImportWarning: platypus based optimization not available
Traceback (most recent call last):
File "
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
And then it starts again...
Traceback (most recent call last):
File "
....and so on
hmmmm.... You are using brew for your python distribution if I am not mistaken. And python 3.9 as well.
I am on a Mac as well, Big Sur also, but anaconda distribution and python 3.8. I have not seen your error before.
Interesting. I think it might be something wrong with brew. I have to test Thanks for the help!
Hi guys, just a follow-up. I made the general ema workbench working (I had a problem with brew and conda directory mixed up). However I can't make it work with the multiprocessor. It does not give any error yet it nevers finish the run. (it basically says "n experiments running sequentially". Anyone noticed that before (the code is the same as before)? Cheers
if it says running sequentially, you are not using multiprocessing. Can you try one of the pure python examples that come with the workbench (e.g., the lake problem)?
I am sorry, it says 'pool started' not running sequentially. I tried to copy and paste the sobol lake problem, it stayed like this the for 3 hours: (I am running in spyder, in a server that has 40 CPUs so it is usually very fast)
C:\Users\TabernaA\Anaconda3\lib\site-packages\ema_workbench\em_framework\optimization.py:48: ImportWarning: platypus based optimization not available warnings.warn("platypus based optimization not available", ImportWarning) [MainProcess/INFO] pool started [MainProcess/INFO] performing 12000 scenarios * 1 policies * 1 model(s) = 12000 experiments
you really need to provide much more context. For example, are you using notebooks, a normal script, can you show the exact code? Also why not try to make it work on your Mac first.
Hey thanks for the quick reply. As I said I am using Spyder, the code is the same as the one I posted before so:
def model_EEG_ABM(F1=5, F2 =25, H =350 , B =1 , T = 2, S = 0, seed = 4, steps=200):
import model
EEG_ABM = model.KSModel(F1=50,
F2 =250,
H =3500 ,
B =1 ,
T = 0.02 ,
S =0,
seed = 4)
EEG_ABM.run_model(steps)
outcomes = EEG_ABM.datacollector.get_model_vars_dataframe()
return {'TIME' : list(range(steps + 1)),
"Real GDP coastal" : outcomes["Real GDP coastal"].tolist(),
"Real GDP internal" : outcomes["Real GDP internal"].tolist(),
"Unemployment rate coastal" : outcomes["Unemployment rate coastal"].tolist(),
"Unemployment rate internal" : outcomes["Unemployment rate internal"].tolist()}
F1=50
F2 =250
H =3500
B = 1
T = 2
S =0
seed = 4
steps = 10
model_EEG_ABM(F1= F1,
F2 =F2,
H =H ,
B = B ,
T = T ,
S =S ,
seed = seed,
steps= steps)
Outputs: Out[18]: {'TIME': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'Real GDP coastal': [125, 0, 500, 1000, 1770, 2898, 3008, 3010, 3000, 3000, 2750], 'Real GDP internal': [125, 0, 500, 1000, 1772, 2846, 2750, 2750, 2500, 2500, 2127], 'Unemployment rate coastal': [0.0, 1.0, 0.57, 0.17, 0.0, 0.11, 0.14, 0.14, 0.14, 0.14, 0.21], 'Unemployment rate internal': [0.0, 1.0, 0.57, 0.19, 0.0, 0.11, 0.21, 0.21, 0.29, 0.29, 0.0]}
model = ReplicatorModel('KSModel', function=model_EEG_ABM)
model.uncertainties = [RealParameter("T", 0.01, 0.1),
RealParameter("S", 0.02, 0.15)]
model.constants = [Constant('F1', F1),
Constant('F2', F2),
Constant('H', H),
Constant('B', B),
Constant('seed', seed),
Constant('steps', steps)]
model.outcomes = [TimeSeriesOutcome('TIME'),
TimeSeriesOutcome('Real GDP coastal'),
TimeSeriesOutcome('Unemployment rate coastal')]
model.replications = 1
If I do individual run it works:
results = perform_experiments(model, 10, uncertainty_sampling='sobol')
While if I do:
from ema_workbench import MultiprocessingEvaluator
with MultiprocessingEvaluator(model) as evaluator:
evaluator.perform_experiments(100, uncertainty_sampling='sobol'))
I get:
In[4]
with MultiprocessingEvaluator(model) as evaluator:
experiments, outcomes = evaluator.perform_experiments(100, uncertainty_sampling='sobol')
[MainProcess/INFO] pool started
[MainProcess/INFO] performing 600 scenarios * 1 policies * 1 model(s) = 600 experiments
But it stays like this forever, without never finishing...I would like to run in the server because with MESA batch run I can 40 in parallel while in my mac only 12 and the server can run overnight without problems.
Also if I try with the lake model (the sobol one), just copy and paste in a .py file in spyder, I get the same problem.
please show the full code, not snippets. My hunch is that you miss
if __name__ == '__main__':
# model definition and run code needs to go here
@quaquel I think we should try not to rescope this one again, but get it in 2.5.
If not a tutorial on connectors, just a simple example.
Don't have a computer at hand so just writing this thought down: Can we make the EMAworkbench leverage or interface with the Mesa batchrunner?
I'm unsure what that would mean or why you would want to do that. The batch runner, in my understanding, specifies a full factorial over the provided parameter values. The workbench offers you much more flexibility.