Live snapshotting in tart

[jerrymarino]

Good morning!

I was looking to do some "live" snapshotting macOS VMs with tart, where memory and state is suspended and then continuously re-serialized

Mainly I'd do something like this:

tart run foo
.. open some programs, write some memory
tart clone foo bar

When running bar, I'd like to have the programs running resume from where they last were at: e.g. counters, ram, etc

tart run bar

Is this one on your radar or a reasonable feature to upstream?

[fkorotkov]

There is this pause method that we haven't investigated yet. The documentation is pretty vague and I'm not sure how it can preserve RAM (haven't seen any reference to anything like this). I know in Anka version 2 suspending itself is pretty challenging since your VM was launched and bunch of stuff like networking was already initialized so resuming of such suspended VM is not trivial.

If that pause method does what you expect then I think we can add Puase button to the UI. Otherwise we'll need to wait for some kind of support from the Virtualizaion.Framework.

[edigaryev]

Unfortunately, the Virtualization.Framework that the Tart uses does not support VM state serialization yet.

Regarding the pause method, it doesn't seem to cover the use-case described as it only stops the vCPU from executing more instructions.

Hopefully this functionality will be added in the future, but for now we are limited to what Virtualization.Framework has to offer in terms of low-level functionality.

[edigaryev]

Will keep this open as it seems that other not possible atm issues are open too.

[jerrymarino]

Cool thanks for the heads up on this. Yeah it seems like it'd probably need some finessing to say the least 🙃 . For context, I have a number of features that I'd like to add to tart or file an issue for. I am attempting to use tart this to spin up throwaway test environments; booting up macOS to run a single test and throwing it away. I might have 1-2 other ones to file around this use case

Regarding the pause method, it doesn't seem to cover the use-case described as it only stops the vCPU from executing more instructions.

Since this method is on a public interface for an Apple VM system, maybe it'd be prudent on my end to feedbacks to ensure it works. I will also give this a a try too.

With some of the open source hypervisor / VM tech out there, this works to some extent but with caveats: e.g. disconnecting virtio devices, and taking other steps to making sure it'd handle the resume

[edigaryev]

Snapshotting support has landed in macOS 14.0 (Sonoma) in the form of saveMachineStateTo() and restoreMachineStateFrom ().

I've checked and it does indeed work, however, it's not possible to snapshot VMs with the following devices enabled (validateSaveRestoreSupport() fails):

• audio: VZVirtioSoundDeviceConfiguration() and friends
• entropy: VZVirtioEntropyDeviceConfiguration()
• keyboard: VZUSBKeyboardConfiguration()
  • can be worked by using VZMacKeyboardConfiguration() instead
• pointing devices: VZUSBScreenCoordinatePointingDeviceConfiguration()
  • can be worked around by only using VZMacTrackpadConfiguration()

Catching and ignoring the validateSaveRestoreSupport() exception results in an app crash, so the only option to proceed would be to provide flags like --no-audio and --no-entropy for tart run.

And even after this, the resulting functionality is pretty limited and fragile, for example, snapshotting a VM with one networking device (e.g. --net-softnet) and then restoring it with another (e.g. the default one) results in (1) unusable network and (2) macOS settings going haywire: The application "System Settings" can't be opened.

Preserving the device state is the holy grail of VM snapshotting, and in the current state of VM snapshotting in Virtualization.Framework it doesn't stack so well with all of the options available in tart run.

It would be nice to see solid examples on how this feature would benefit the users compared to just starting the VM from scratch (which is a pretty fast process), otherwise we risk shipping a half-working and unpredictable functionality.

[Jon889]

I would think there would be many usecases for not changing the configuration of a VM while it's saved. For example I use Tartelet to run GitHub action runners, they terminate and restart automatically so there's no way to change configuration in between.

For that use I'd want saving and restoring so I could keep the iOS simulators running. VMs might be quick to turn on and off, but the software that running on them can take more time.

Even if running them manually, I wouldn't expect changing configuration to work, it would be like ripped out the network chip while a non WM Mac is running.

[fkorotkov]

I guess what we can potentially explore is an option to save the state only for local VM and not push it with tart push. In either case the saved state is locally encrypted and doesn't work on a different device.

So in case you run a VM locally and close a window or use tart stop, we can try to dump the state locally so the next time you tart run this VM or a cloned one, then the state will be restored. But if you did shutdown macOS from within the VM then there will be no state.

There might be one potential caveat though. When Tart clones a local VM, it generates a new MAC address for it so you can run both VMs in parallel without network conflicts. We'll need to change if the VM state can survive restoring for a different MAC address.

[timsutton]

I may have read through this issue too fast and missed discussion of this, but there's an additional blocker if anyone wanted to archive and transfer saved VM state from one Mac to another: supposedly it includes some unique material from the SEP, so saved machine states are designed to not work on any Mac besides the one on which the save was originally taken.

[fkorotkov]

Tart 1.8.0 release is building now and will be shortly available. It brings support for suspendable macOS virtual machines. Note that both host and guest should be on Sonoma and you need to pass --suspendable argument to tart run:

tart clone ghcr.io/cirruslabs/macos-sonoma-vanilla:latest sonoma-vanilla
tart run --suspendable sonoma-vanilla

[fkorotkov]

@Jon889 Tart 1.8.1 will work with Tartlet out of the box. Just configure the local VM with tart run --suspendable and when Tartlet will call tart run the VM will be resumed.

But note that you can only run a single suspendable VM at a time. You can't resume two suspendable VMs cloned from the same base VM since MAC address collision.

[fkorotkov]

We also just tweeted out a demo of how snapshotting is working.

https://twitter.com/cirrus_labs/status/1677308360385765382

[Jon889]

@fkorotkov I was finally able to update to Sonoma, I get this error: Failed to suspend the VM: Invalid virtual machine configuration. Unsupported console device in configuration

even when creating a new VM with: tart create sonoma --disk-size=80 --from-ipsw=https://updates.cdn-apple.com/2023FallFCS/fullrestores/042-54934/0E101AD6-3117-4B63-9BF1-143B6DB9270A/UniversalMac_14.0_23A344_Restore.ipsw

and then running it with: tart run --suspendable son

Do you know how to change the configuration to make it work?