mesa icon indicating copy to clipboard operation
mesa copied to clipboard

DataCollector: store datasets internally as structured arrays

Open dotsdl opened this issue 8 years ago • 2 comments

At the moment DataCollector keeps a dictionary for each of:

  • model_vars
  • agent_vars
  • tables

These currently store values that are lists of tuples, lists of lists of tuples, or dictionaries of lists. This works fine for small models, but performance in using these data or making pandas.DataFrames from them doesn't scale up well to thousands of agents over thousands of steps.

One option is to use structured arrays. These require knowing in advance the size of the array you need to make them worthwhile, but even for models in which agents appear and disappear they could still be used effectively at least at the level of individual steps.

dotsdl avatar Aug 18 '15 05:08 dotsdl