criu
criu copied to clipboard
checkpoint-restore
Checkpoint restore for startup acceleration, but with non-destructive restore and many uses for the same checkpoint
I believe I've identified an excellent use case for criu, but it appears that there are a handful of options that I may need in order to meet my requirements. We have an application with relatively complicated state that takes several minutes to startup, but this state will be the same across users and in many cases between two vastly different uses. It's very much in line with the 'Slow-boot services speed up' usage scenario listed on criu.org, though there are a couple of features I would like to have that either aren't currently present or I simply haven't read the correct doc, or perhaps I haven't read the doc correctly.
It seems criu is very good at restoring a process exactly as it was. But what if I needed to restore it, say, 98% as it was? Like with a different user, pid, and log output, and without rendering the checkpoint unusable for subsequent usage.
As I have experience in interfacing with linux, I'm happy to perform the modifications needed myself, but I don't want to overlap with existing features. Here are the requirements. Note this is a non-containerized application and there aren't plans to containerize it, though in practice there is no external resource that should change between different instances and different users, at least not until the point of being fully initialized.
I need to restore the same checkpoint for several users: some issues I've run into or believe I will run into regarding this: -I need each restore to write output to a different log file by overwriting the original fd with the fd for a new log file. This is probably mostly a scripting thing, though I'm not sure if this overwrite is possible. -Don't want to restore with the same pid. This should as simple as cloning without specifying one. -Need to specify a new uid for the process to belong to. Yes, there could be permission issues, but in our case every file should have roughly the same permissions for all users. I think the responsible implementation would strictly enforce that the new user has all of the required permissions. -non-destructive restore regarding nfs silly rename.
Also, anything else one might need to restore from the same checkpoint image 1..n times on the same machine. The restored processes must be orthogonal to one another. Luckily, due to no changing external state during bootup and no remaining IPC at checkpoint time, this probably shouldn't be as complicated as it sounds. Famous last words?
Is there a hard roadblock to this that I'm not seeing? What feature overlap, if any, is there with criu at the moment?
Did you look into changing the checkpoint images before restore with the help of crit? Assuming you have full control of your application and can handle those not 100% restores crit should solve most of your problems.
The PID problem probably not as you would have to select a certain PID before restoring. For the PID, even if it is not containerized, you could still use PID namespaces on restore. Restore each process into a new PID namespace.
Maybe namespaces would also work for the log file renaming. Use mount namespaces and bind mount different log files to the same log file inside the mount namespace.
At this point it feels like you could solve your questions with namespaces and checkpoint image files editing.
Would any of this work for you?
Good point- we are planning on running criu as root, so I shouldn't have any trouble setting up and tearing down some namespaces. And should I run into any other issues, they would likely be the answer as well.
Except perhaps the silly nfs rename issue where a restore deletes the hardlinks holding it the renamed file in place. I can imagine a couple of ways to deal with without changing criu this that involve copying the hardlinks, but it may be more seamless to add a --non-destructive option to restore for this. Copying the hardlinks then replacing them could introduce race conditions- perhaps a bind mount and namespace solution exists here too. I will experiment.
I cannot say anything about NFS. @mihalicyn can you say anything about the NFS questions?
In my own experimentation I added an option which prevents criu restore from deleting link-remaps when specified. This has worked for my use case, though it's possible that there's already a good way to accomplish this.
It should be ok to change pid if we are dumping one process. But if a target process has many threads, can we change their tid-s? I am almost sure that it will trigger some issues.
I think one major issue with changing pids and tids is that these are often explicitly held as program state and tracking that down and changing it would be a real pain. Pid namespaces works for me here.
Thanks for answering my questions. Pending company approval I plan to contribute some changes regarding link-remaps.