mesa
mesa copied to clipboard
DataCollector: store datasets internally as structured arrays
At the moment DataCollector
keeps a dictionary for each of:
- model_vars
- agent_vars
- tables
These currently store values that are lists of tuples, lists of lists of tuples, or dictionaries of lists. This works fine for small models, but performance in using these data or making pandas.DataFrame
s from them doesn't scale up well to thousands of agents over thousands of steps.
One option is to use structured arrays. These require knowing in advance the size of the array you need to make them worthwhile, but even for models in which agents appear and disappear they could still be used effectively at least at the level of individual steps.