Allow writing data during `run!`

[kavir1698]

Adds an option to run! so that collected data can be written to file during the simulation. It is useful when collected data are too large to fit the memory by the end of the simulation.

[codecov-commenter]

Codecov Report

Merging #630 (1f2241d) into master (dbddf76) will increase coverage by 0.17%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #630      +/-   ##
==========================================
+ Coverage   91.04%   91.22%   +0.17%     
==========================================
  Files          24       24              
  Lines        1631     1664      +33     
==========================================
+ Hits         1485     1518      +33     
  Misses        146      146              

Impacted Files	Coverage Δ
src/Agents.jl	100.00% <ø> (ø)
src/simulations/collect.jl	100.00% <100.00%> (ø)

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[Datseris]

@kavir1698 thanks for the PR! This is a good feature, however I believe we should not complicate the run! function more, it already has a lot of options. I would recommend to make a new function that does this functionality. I suggest as name offline_run!, but feel free to suggest better ones. Given that the function is supposed to operate when final data are too large to fit in memory, then the function also shouldn't return anything.

[kavir1698]

I don't mind creating a new function. But that would create a lot of duplicate code because ensemblerun! may also support writing outputs during runs.

[Datseris]

But then ensemble run would also need extra code because it would need to somehow clarify the extra column. While it is sad that we will have code duplication, I think it will lead to simpler development in the long run if we have smaller functions instead of really large functions that do many things in once with a bunch of if statements inside.

[Datseris]

Apologies @kavir1698 , I need to release 5.4, so this will go in in the next minor release, since we need some time to think what is the best interface for this feature.

[kavir1698]

Apologies @kavir1698 , I need to release 5.4, so this will go in in the next minor release, since we need some time to think what is the best interface for this feature.

Sure. It would take me some time to rewrite this PR.

[Datseris]

@kavir1698 I'll be releasing v5.5 sometime soon, perhaps you want to get this in there?

[kavir1698]

I'll work on it! :)

[fbanning]

Hey @kavir1698, are you still planning to continue work on this PR? Recently there has been some interest in writing data to disk during simulation runs (specifically for paramscans but that's just more of the same). It might be worthwhile to pick this PR up again, fix the minor doubts from @Datseris and update it for the current Agents version. If you don't have any time for this, I might pick it up where you left it and try to finish it.

[kavir1698]

Hi @fbanning I am currently distracted because of the situation in my country, Iran. I can't promise to finish it soon. If you have time, by all means, finish this PR please.

[fbanning]

I am currently distracted because of the situation in my country, Iran.

✊🏻

If you have time, by all means, finish this PR please.

I'll talk to the person over on Slack who is interested in this topic and see if they want to pick it up, otherwise I'll do it.

[mastrof]

I would need this and can try to get this done soon-ish. Did something relevant change since the PR was initiated?

[Datseris]

no

[Datseris]

Closing in favor of #815