vagrant
vagrant copied to clipboard
hashicorp
Provider for UTM (VM for M1 processors)
On new Mac's with M1 processors users can't use Virtualbox and VMware, but UTM works: https://mac.getutm.app/
Could you please add provider for UTM? That will save a lot of developer life :) Thanks!
UTM doesn't expose an API AFAIK so I don't think it would be possible to build a provider for it.
However UTM is simply a GUI wrapper around QEMU. An official QEMU provider would indeed be awesome to see for Vagrant, it would provide very wide platform support including Linux, Mac (Intel and arm) and Windows.
There is already a libvirt provider for Vagrant (https://github.com/vagrant-libvirt/vagrant-libvirt) which is able to interface with QEMU via the libvirt API but I haven't had much luck getting it working outside Linux. The libvirt API only exposes a subset of what you can do with QEMU directly so a QEMU provider would still be a great addition IMO.
I am a developer for UTM and we are interested in bringing Vagrant support. Is there a high level list of things that is needed to bring support? We support both QEMU and Apple Virtualization.framework so there's consideration in what features we can support that will work well with both.
Hi, I also help to develop UTM, and have looked at the custom provider docs for vagrant.
If I understand it correctly, the bare minimum needed for a provider is to:
- Define a box format
- Implement
vagrant upto start the VM - Implement
vagrant ssh, which would require some way to get the VM's IP address (since UTM can use Shared or Bridged networking)
Is there anything else that would be needed (at least initially)?
Is implementing vagrant status necessary for a valid provider? The VM state is already stored, we would just need to figure out some way to expose it externally.
Edit: Looking at some example Vagrantfiles using the Virtualbox provider, it seems like configuration of the VM is done in the Vagrantfile (for example CPU cores, memory, networking). This isn't really compatible with UTM because it stores a config.plist in the VM bundle.
Is exposing VM configuration to the Vagrantfile necessary?
Looking at some example Vagrantfiles using the Virtualbox provider, it seems like configuration of the VM is done in the Vagrantfile (for example CPU cores, memory, networking). This isn't really compatible with UTM because it stores a config.plist in the VM bundle.
Is exposing VM configuration to the Vagrantfile necessary?
Not a Vagrant expert, just a long time user, and a recent Apple Silicon user, so I would welcome UTM support!
Vagrant users expect that the virtualization application becomes invisible; you don't want to need to touch it at all. You script the VM(s) you need using the Vagrantfile, then Vagrant and the VM provider create the desired VM and give you access (via SSH, port forwardings, custom network config, and shared filesystems). When you're done, you have Vagrant destroy the VM(s) again. It is therefor necessary to have Vagrant being able to specify relevant desired VM parameters, mainly the amount of RAM and the number CPU cores, and network parameters. I have had more complicated setups where I also added additional disks through Vagrant. And while it's possible to use Vagrant to create a VM with a GUI, I think the overwhelming use case is to have a headless server.
Think of Vagrant as a mini IaaS cloud on you laptop.
@osy @ktprograms I know it's frowned upon to write "me too" messages but I do want to thank you for your work with UTM and really hope you can bring a provider to Vagrant!
Edit: Looking at some example Vagrantfiles using the Virtualbox provider, it seems like configuration of the VM is done in the Vagrantfile (for example CPU cores, memory, networking). This isn't really compatible with UTM because it stores a config.plist in the VM bundle. Is exposing VM configuration to the Vagrantfile necessary?
As for this question -- I don't know the specifics (I'm just a user of Vagrant, not a developer of Vagrant) but as I understand it the idea is that the configuration of the machine is written in the Vagrantfile, and the comnand-line UTM program should accept command-line parameters in order to create the machine with the right config (# core, memory, networking). Once it's created then of course you can save those configs in your internal plist file.
@osy @ktprograms you're best looking at the parallels provider as an example. However some finer details
I can foresee the following:
- a vagrant plugin will need to be built that adds
utmas a provider - vagrant files and plugins are written in Ruby
- this plugin will interface with
utm, Virtualbox does this via calls tovboxmanageand other CLI, Parallels has an equivalent, UTM will likely need the same
As for the Vagrant file, as a writer of a vagrant file I specify the machines I want and some options as documented by the vagrant docs. This is a declarative API, so I never interact with the VM itself, rather I place an order with Vagrant who goes away and cooks up an environment for me. Then I can run provisioners scripts/commands in those environments once they've booted up. The vagrant plugin would be responsible for taking that and issuing the necessary instructions to UTM.
The other factor is how your box will be formatted, I couldn't comment on the details of that.
- Here's the parallels provider: https://github.com/Parallels/vagrant-parallels
- Vagrant plugin basics: https://www.vagrantup.com/docs/plugins/development-basics
- Here's vagrant provider docs: https://www.vagrantup.com/docs/plugins/providers
- Packaging and distributing plugins: https://www.vagrantup.com/docs/plugins/packaging
I believe with this you could get a very simple level of support operational quickly depending on UTM's APIs, and I'd suggest encouraging open-source contributions with docs. The finer points of shared/mounted folders and networking options may take longer to build, though provisioners should be quicker I would expect.
The biggest obstacle I can see right now is that there is no documented API for UTM or CLI application advertised. If this existed someone would probably have built a basic vagrant plugin by now, but instead it looks like a pure GUI Mac application
What's the advantage of having UTM be the provider instead of QEMU? Is it that folks want to manage the VM from both Vagrant as well as the UTM UI?
On my M1, I can cat ~/Library/Containers/com.utmapp.UTM/Data/Documents/ArchLinux\ ARM.utm/config.plist and see the configuration for the VM. Is Vagrant unable to store changes to this file if a user launched a VM from Vagrant?
If there's a problem reading/writing to config.plist, would it be easier to implement the QEMU provider?
BTW, UTM is awesome and was instrumental in me getting virtualization up and running on the M1, and it's inspired me to dig deeper and learn more lower in the stack. Thanks for implementing it and being helpful with all of my support questions.
One of the reasons that I installed UTM is that the experience with X11 is pretty miserable if going through, say a Docker Desktop to XQuartz (the color tables are completely off and screen capturing doesn't get it right either), whereas UTM's display is quite nice (better than I remember VirtualBox being). I've tried out several linux variants and the Gnome Desktop and they're all very responsive (when virtualized), and any work with docker inside the VM can just mount the /tmp/.X11-unix dir for accessing the XServer as good as it gets through the unix socket.
Ultimately, I'd like to see proper support for UTM if the formats/configs can be negotiated/shared to make that so, but in the meantime, before I try it out, does the qemu provider have a UI facility at all? What does it use to display?
QEMU uses either a WebKitGTK window or VNC, to the best of my knowledge. You can check out the QEMU docs for the different display types you can pass to the -display argument.
I am not sure how good they are though. So far I have only used them on console only VMs.
One huge advantage to run vagrant on UTM instead of vanilla QEMU is vm.networking entitlement. Current select()/poll() from glib on macOS 11+ is broken and will corrupt external network packets (example).
Right now we either have to run vde-2 with vmnet as root, or run the entire qemu (ver 7.1+) as root with vmnet netdev. I wouldn't consider those solutions are secure and viable when scaled.
I don't know if this helps, but you may want to look at what Red Hat did with podman (a CLI which can run containers, a fully compatible yet complete replacement of Docker). On Mac:
podman machine init
podman machine start
will spin up a VM using qemu. This VM contains a Linux environment which runs all the containers started by podman .
The options they use to spin up the VM are here for amd64: https://github.com/containers/podman/blob/668deb2ff1cb918ec787e4c1de3a7c321b0def0b/pkg/machine/qemu/options_darwin_amd64.go
and here for arm64: https://github.com/containers/podman/blob/668deb2ff1cb918ec787e4c1de3a7c321b0def0b/pkg/machine/qemu/options_darwin_arm64.go
Maybe if Vagrant has a provider that just uses the bare minimum of qemu to spin up a VM, similar to how podman spins up a VM in qemu, that would be enough?
I think @baude and @jonpspri have worked on this.
To add even more justification for UTM support... I use Vagrant for all my VM provisioning, but when it comes to port management then I do it ad-hoc through the UI so that I don't have to stop / start the VM. Like many others here, I'm transitioning from VirtualBox on an Intel Mac to UTM/QEMU on Apple Silicon because there is no VB solution. QEMU is a beast to deal with directly which is why UTM is my solution of choice. At this point if I'm forced to chose between UTM and Vagrant then I'll pick UTM. Most of my stuff runs in Docker anyway so it's not like VM setup is a huge hassle. I'm not eager to ditch Vagrant, but I get less operational hassle if I stick with UTM
I'd love to see this happen! Do you guys need help on either side? I'm happy to throw in a few hours to see this happen. Feel free to DM me.
@neonihil I believe that the open questions at the moment are from the UTM side, e.g. you could create a vagrant provider for UTM but knowing how to configure UTM to spin up a basic box without trigger user interface actions may be difficult.
Correct me please if I'm mistaken but:
- Some people have mentioned plists for configuration, a basic example configuration and an equivalent vagrant file that should map to each other would be helpful
- There may have been some hesitancy with the new
vagrant-goand v2.3 release not being out, but this should now be resolved with the recent release - I believe there is no machine/CLI/programmable UTM interface yet? But that one was planned? If this hasn't been done then this would be the primary blocker to a vagrant UTM provider, but if it has been done or it's in progress then collating that information would be valuable research for someone speccing out how a UTM provider would work. E.g. there are questions such as:
- how would you turn the environment on and off? Destroy it? Create it?
- given a libvirt box how would you give that to UTM?
- how would you retrieve the status of the VM from UTM?
As for wether it will be officially supported rather than as a plugin, that's something Hashicorp would need to comment on, we can make it easier by helping with the technical details, but there's strong precedent for community written providers doing the work of adding support.
knowing how to configure UTM to spin up a basic box without trigger user interface actions may be difficult.
In UTM 4 (still in beta), there's support for removing all displays, so the VM runs headless. I'm not sure if the main UTM app will still show up, and as mentioned there isn't any CLI interface so you'd have to use the URL scheme to start VMs.
I don't want to derail the conversation about UTM but I want to mention that Tart VM for Apple Silicon supports now all needed technical requirements (Linux guests, API, shared folders etc.) for Vagrant with new macOS Ventura.
They just need help for creating the Vagrant provider for it:
https://github.com/hashicorp/vagrant/issues/12760#issuecomment-1228698988
Unfortunately Tart is out of the picture for me, because HyperKit does not support USB device passthorugh, and it is critical to embedded/firmware engineers to have an external debugger (mostly over USB) attached to the developing environment. I used to work with docker-machine but they deprecated the project last year, and therefore I am forced to work with UTM with physical USB device passthorugh.
So many developers are using MacOS, and so many developers are using virtual machines. Many of those developers are now upgrading to Apple Silicon. What is Apple themselves recommending these developers to use? Many developers use for example Vargant + VirtualBox. You would think that Apple would have suggested an alternative for those who upgrade to M1/M2. Does anyone know if there is any kind of official recommendation or statement from Apple for these people?
I don’t believe Apple has ever made such a recommendation arm or Intel. The closest you’ll get is their hypervisor APIs.
There are equivalents on Arm but only parallels has built a consumer facing vagrant provider plugin. You can’t just drop in a replacement vendor.
There is now a scripting interface for UTM implementing the OSA for macOS apps: https://github.com/utmapp/UTM/blob/main/Scripting/UTM.sdef
We would like some feedback about any additional functionality that would make it easier to run Vagrant.
Note that there are already plans to implement some mechanism to talk with the QEMU Guest Agent although this is not currently implemented. That means right now there's no way to get any information out of a guest (such as SSH address).
Would this be the interface/api a vagrant provider for UTM essentially interact with? (Sorry new to all of this but very interested to learn more to know how I can help).
https://developer.apple.com/library/archive/documentation/AppleScript/Conceptual/AppleScriptX/Concepts/osa.html
It's an interface for other apps to interact with your app. I need to know what to provide in this interface for apps such as vagrant to function.
I created a basic list of things that are needed for Tart integration in:
https://github.com/hashicorp/vagrant/issues/12760#issuecomment-1303559526
The same things/questions apply to UTM, I've been doing exploratory work with regards to a Tart provider, UTM support via AppleScript should be suitable, any Applescript APIs can be worked with via the osascript command.
At a minimum we would need to figure out what the applescript/osascript commands are to do these actions in UTM, much like hashicorp did for vboxmanage etc
Looking at the list, here's the things that seem to be needed on the UTM side:
- [ ] Getting VM IP address (will be possible with QEMU guest agent). After that SSH and running single commands should be possible.
- [ ] Setting up shared directories via apple events (I'm not sure if this is already possible).
- [ ] ID mapped mounts for 9pfs (Linux kernel side), this is needed since default UIDs on macOS don't match the 1000:1000 on Linux (it's 501:20 on macOS IIRC). I'm planning to work on this but not sure how easy it will be. If needed, the old WebDAV shares are still available I think. Alternatively, would SSHfs work on a macOS host without FUSE?
@bernsel
you would think that Apple would have suggested an alternative for those who upgrade to M1/M2.
Noone would think that, it's Apple .. they could not care less and you know it )))
