Resume Japonicus after crash/interruption?

[rcoenen]

My EC2 (Amazon Web Service) Gekko/Japonicus instance became unresponsive and crashed. I suspect it ran out of memory or something since I am using a t2.micro free tier instance.

Anyhow, this happened at 4AM while at least a few hundred out of 3000 epoch runs had been analyzed. I just restarted the server, gekko and japonicus and... everything starts from epoch 0/3000 again 😞

Is there a way for Japonicus to pick up from where it left off? Can it store any progress made so far to disk somehow?

[rcoenen]

Just FYI - I have experimented with /japonicus/evaluation/gekko/backtest.py and under backtestResultExporter I have set writeToDisk to True (default: False). However this resulted in a lot of backtest results being logged directly in the /gekko/ directory. When I restarted japonicus, it seemed to disregard these cached files and just new ones were being generated. Unsure what the use is of this feature.

[Gab0]

Hey bro, backtestResultExporter is a gekko option, so gekko writes the log on its own directory and they won't show meaningful stuff about the GA. There is no way to resume a session... it would be cool to have that function yeah, those things needs some care to implement, otherwise the bugs rise.

But we can somewhat resume a session, at japonicus/log the best parameters are saved periodically, so if the session crashes you can start another after changing the strategy settings to match those good parameters. That's kinda laborious but it's what we have now... Also, from the logs you can see if scores kept rising even at epoch 3k, which I don't think was the case based on my runs...