activitysim icon indicating copy to clipboard operation
activitysim copied to clipboard

potential pipelining issues/improvements

Open bstabler opened this issue 8 years ago • 2 comments
trafficstars

As we build out more of the model system, we plan to watch out for some potential issues with the pipelining:

  • the file size might be too big
  • one big file is difficult to copy, can get corrupted, etc.
  • what random numbers were used (for debugging, etc.)
  • too much performance degradation

Some potential solutions:

  • can we only write at select milestones (such as after key long calculations)?
  • can we only write the additional columns added to each table after a submodel as opposed to the whole table?
  • can we split it into separate files to avoid potential issues with one giant file?
  • can we add random numbers for each model for each chooser to the relevant tables?

We don't need to do any of these things right now, but let's be mindful of them moving forward.

bstabler avatar Apr 14 '17 20:04 bstabler

We did this one - can we only write at select milestones (such as after key long calculations) - by allowing the user to skip persisting tables by adding underscore to a model step name in the models list.

bstabler avatar Jul 29 '20 01:07 bstabler

#397

bstabler avatar Mar 23 '21 00:03 bstabler