Optim.jl Checkpointing in Optim

I've come across the scenario of using an optimization algorithm, but did not have the time in the rest of the day to let the optimizer complete and find a solution. I was curious if you thought about putting a checkpointing system in Optim to allow users to stop their algorithms at an arbitrary point and continue off at that point at another time (i.e. Save the progress that the optimizer has already made)

Dec 07 '20 19:12 umbriquse

How would you stop it? With SIGINT ?

Dec 17 '20 11:12 pkofod

Yes, but also in the situation of either power failure, an error occurring in the user code after some iterations, or some segfault/ bug in julia that hasn't been corrected. Mostly with SIGINT though.

Dec 17 '20 16:12 umbriquse

This idea comes from another package that I am using called DFTK. They rely on using JLD2, and save a snapshot of the current state so that the progress made isn't lost and can be continued at another time. If you're curious it's in the jld2io.jl folder.

Dec 17 '20 17:12 umbriquse

Okay okay, I think I may understand. If you're that worried about sudden problems I would suggest you use a callback to save the current state. Does that make sense?

Dec 17 '20 21:12 pkofod

Yes, I thought it wasn't possible to save the progress of the state and then start an optimizer in at that saved state when using any callback function. Thank you, I'll do that.

Dec 18 '20 19:12 umbriquse

It's not super easy to start it again, but I can help you set it up if you need something like this.

Dec 19 '20 13:12 pkofod

Thank you for offering, but I think I can work it out. I am curious if you would like the function that I would make though? So others could add that function into the callback and save the state of their optimizers.

Dec 23 '20 20:12 umbriquse

Yeah, please show it. If nothing else we can put it in the docs :)

Dec 23 '20 22:12 pkofod