gvisor
gvisor copied to clipboard
google
Checkpoint & Restore between machines with different CPU features
Description
Currently, a checkpoint image created on a physical machine with a newer CPU may encounter restoration failures due to missing CPU flags when restored on a physical machine with an older CPU[1].
This has increased the complexity of using Checkpoint/Restore technology to accelerate container startup (one image, multiple containers). We either have to find a machine(or choose a vm) that has a feature set as the maximum subset to create the checkpoint image, or we must create separate checkpoint images for each type of machine and distribute them according to the machine type.
Additionally, you may observe in application logs your Function being memory snapshots multiple times during its first few invocations. This happens because memory snapshots are compatible with the underlying worker type that created them, and Modal Functions run across a handful of worker types.
Modal has encountered similar issues, which has led them to create multiple images[2].
- https://github.com/google/gvisor/blob/release-20250217.0/pkg/sentry/kernel/kernel.go#L800
- https://modal.com/docs/guide/memory-snapshot
Is this feature related to a specific bug?
No response
Do you have a specific solution in mind?
Thanks to the capability of gVisor's cpuid emulation, we can control the CPU features exposed to the user application (i.e., the maximum feature subset of all CPUs in the cluster), which allows us to create only one checkpoint image. This has been widely used internally, and we hope to merge this feature into the mainline.
Currently, we use an annotation dev.gvisor.internal.cpufeatures inside config.json to pass the CPU features exposed to the user application, and we also hope the gVisor community can give some input to see what approach would be more general.
Yeah we need to add support CPU feature leveling - i.e. allow users to specify the CPU feature set the application can use. If runsc users want to checkpoint/restore across hosts, then they would define that as the lowest common denominator between the two CPUs.
PRs are much appreciated! What you have described seems acceptable. The annotation approach is good.
PR https://github.com/google/gvisor/pull/11498 has been submitted; please review it when you have time :)