trellis
trellis copied to clipboard
roots
Trellis and Apple Silicon
- [x] I've read the guidelines for Contributing to Roots Projects
- [x] This request isn't a duplicate of an existing request
- [x] This is not a personal support request that should be posted on the Roots Discourse community
Summary
Apple Silicon is here, VirtualBox support is far away (if ever) – what is the road going forward?
Motivation
Apple have started to ship their first arm based macs to customers and as many of us developers are sitting on macOS systems I'm curious if you've considered how the road forward will look? As I understand VirtualBox support seem far away, Docker is more probable. I love working with the Roots stack when doing WP work, would be nice to be able to do so in the future.
Some links with discussion around M1 based macs. https://forums.virtualbox.org/viewtopic.php?f=8&t=98742 https://www.docker.com/blog/apple-silicon-m1-chips-and-docker/ https://localwp.com/community/t/local-on-apple-silicon-macs/22834/14
I haven't looked into this enough yet, but my interpretation of the current state of things is:
- it seems like virtualization support won't come until the next gen M2 chips (and that's just people's hopes)
- VirtualBox likely won't ever work on M* chips until/if the above happens
- Trellis is quite tied to x86 right now
I think the key point is that Trellis is coupled to x86 more so than VirtualBox itself. Trellis already supports other Vagrant providers which some people use.
There's two biggest questions:
- will it become more common place to run ARM on production servers?
- or are people willing to run ARM in development and then deploy to x86 in production?
I personally don't really know the implications of running difference architectures in dev/prod yet. So even if Trellis found a way to migrate to Docker... there's still a lot of potential problems and big open questions.
My stance for now is nothing will change for a while. Anyone running M1 chips is unfortunately out of luck running for Trellis for at least the short-term and almost certainly for the medium-term too.
1 sounds a bit farther away, but 2 looks like a necessity already for anyone with ARM macs.
@swalkinshaw Scott, as for short-term, do you have weeks or rather months in mind?
If you want to run ARM in development now then I'd try and find another Vagrant provider that supports it. VirtualBox isn't the only choice so Trellis doesn't really need to do anything to support that (as far as I know).
A bit over my head, but I would like to try it. Would you suggest replacing virtualbox with docker? I remember seeing a couple of years ago a docker feed on discourse, but not sure if it gave any solid fruits in the end.
From what I can see, there is no working provider at the moment. Docker is on standby waiting for upstream updates from Go and others with no ETA, Parallels is still in development with no ETA either, VMware is committed as well, but completely silent about roadmap. So there is no possibility to try out the 2nd option for the time being. But it would be wise to at least leave a note for any silicon users stating that the Trellis framework is not compatible with Apple M1 laptops until at least some of the Vagrant providers start rolling out arm tailored versions.
Ah, yeah sorry I wasn't sure what the status of Parallels and VMware was yet. I had just read they had betas or were working on it.
But yes we definitely need to document this limitation somewhere.
Seems like great progress is being made!
https://www.xda-developers.com/apple-silicon-mac-boot-windows-10-and-linux-virtualization/
So developers who bought new Macbook Pro with M1 are currently not supported by Trellis? Docker seems to be working on M1, but Trellis is not compatible with Docker. Maby Trellis should really consider Docker as other provider than VirtualBox?
@Pls I think Docker is a bit different solution to the question as it basically replaces trellis instead of just creating a vm. But you can try out docker approach. You can find more information here: https://roots.io/docs/bedrock/master/local-development/#additional-resources.
I thought about Rosetta 2 since it allows some apps not made for the M1 chip to actually run, but unfortunately they don't support virtual machines.
What can’t Rosetta 2 translate? Rosetta cannot translate kernel extensions or Virtual Machine apps that virtualize x86_64 computer platforms. Developers should be aware that Rosetta is also unable to translate AVX, AVX2, and AVX512 vector instructions.
Leaving it as a reference for anyone stumbling here.
Just got a new iMac with the M1 chip, would love to know what the gameplan is for making it compatible with the Roots suite. Is it just a waiting game at this point?
Has anyone attempted Trellis on ARM in development on the M1? Parallels should support that. https://www.parallels.com/blogs/parallels-desktop-m1/
Can we get support for m1 macs through docker? It looks like virtualbox and even vmware are dragging their feet on support the macs and some people got vagrant working on docker: https://dev.to/taybenlor/running-vagrant-on-an-m1-apple-silicon-using-docker-3fh4
It would be great if we could flip a flag and spin up a docker image instead of a virtualbox machine, not sure how hard this would be in practice though.
-- @Digital-Nomad https://github.com/roots/trellis/issues/1302#issue-930245243
I've been looking into this a bit more lately, and here's what I've found.
New M1 chips
The new M1 Pro and M1 Max CPUs don't change anything virtualization wise unfortunately. They are the same as the original M1 in that regard.
Virtualbox
There has been absolutely no movement on Virtualbox offering ARM support and I highly doubt there ever will be. It defeats the purpose of VirtualBox which is very tied, by design, to x86.
Other virtualization solutions
While Trellis defaults to VirtualBox (since it's open source and free), it's always supported other providers like VMWare and Parallels.
Parallels
As mentioned above, Parallels does have support for M1 macs but only for ARM based operating systems.
VMware
Likewise, VMware Fusion now supports ARM-based Linux distributions just like Parallels.
Others
- lima-vm - promises all combinations of VMs?
- UTM - QEMU wrapper with GUI
- libvirt - lower-level QEMU wrapper
- docker
One issue with all of these is integrating them with Vagrant. docker has by far the most mature Vagrant integration with a 1st-party provider. Lima and UTM have none so far. vagrant-libvirt exists but has a lot of issues on macOS
Parity between Intel & ARM
The best bet at this point is running Virtualbox on Intel Macs and some other ARM-based virtualization solution on M1 chips. This isn't 100% ideal since it's likely your production server will be running x86 for the foreseeable future; however, I'm not sure it matters in reality. While there are difference between x86 and ARM, it likely does not matter for the purposes of running a web server with WordPress. PHP should be PHP; Nginx should be Nginx, etc.
It's a small trade-off we'll have to make and be okay with in the end.
Conclusion
This space is changing every day and it's hard to keep track. There's a lot of solutions which are like 50-80% of the way there, but not 100% unfortunately. Right now the most promising solution is:
- stick with Vagrant as the main way to create development VMs
- continue to use Virtualbox for Intel-based Macs
- decide on the best option for running Ubuntu ARM VMs for M1 Macs
- default to Docker currently because it has the most mature Vagrant driver
Of course Docker's default design is not compatible with Trellis because docker images/containers don't like to run multiple services and lack an initd system. This has previously been discussed, but I did find a useful base docker image which solves most of these problems: https://medium.com/nerd-for-tech/developing-on-apple-m1-silicon-with-virtual-environments-4f5f0765fd2f
I've been testing this out on an x86 Mac, and I've run into a few problems (NFS and networking related), but are hopefully solvable.
If anyone wants to help test out any of the other Vagrant providers or VM solutions, please let me know or post your results. It's also helpful for people to post updates about these various projects if they come across them.
Further reading:
- https://github.com/Varying-Vagrant-Vagrants/VVV/issues/2468
- https://medium.com/@lizrice/linux-vms-on-an-m1-based-mac-with-vscode-and-utm-d73e7cb06133
- https://github.com/hashicorp/vagrant/issues/12518
- https://gist.github.com/aserhat/91c1d5633d395d45dc8e5ab12c6b4767
Help wanted
If you have an M1 Mac and have Parallels or VMware, you can help by trying out an ARM based Ubuntu box.
Since Trellis already supports Parallels and VMware fusion, it's possible to try using an ARM-based Ubuntu vagrant image and get Trellis working "as is".
The main Ubuntu vagrant box we use is bento/ubuntu-20-04 which doesn't yet have support for ARM. However, https://github.com/chef/bento/pull/1374 looks like it has most of it ready. I think you'd need to packer to actually create that box/image and use it locally.
Ok! I got this to work with parallels on an M1 Pro macbook pro. I'm not the smartest person here so I'm going to lay out some steps in hopes of helping others like me. The machine provisions without errors, but I have not extensively tested beyond this.
**note I believe parallels pro is needed which is what my trial is for this test. The basic version does not provide the cli tools. https://www.parallels.com/products/desktop/pro/#compare
Following swalkinshaw's extensive legwork, using the chef/bento vagrant box on parallels is the only one I could get working. I tried using VMware fusion but I think more work needs to be done to get that working. Jeffnoxon already has a vagrant box compatible with parallels up on the vagrant cloud (https://app.vagrantup.com/jeffnoxon/boxes/ubuntu-20.04-arm64). I'm not sure if it's safe to just reference this box or if something more formal needs to be done.
I added that box to the vagrant.default.yml
vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64'
vagrant_box_version: '>= 1.0.0'
I also needed to make a slight change in the maria-db role. roles > mariadb > defaults > main.yml -- change the arch from amd64 to arm64
mariadb_ppa: "deb [arch=arm64] http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.5/ubuntu {{ ansible_distribution_release }} main"
after that, vagrant up using the parallels provider (make sure you install the parallels vagrant provider plugin (https://parallels.github.io/vagrant-parallels/docs/installation/)
vagrant up --provider=parallels
And success! https://prnt.sc/1xujta1. Loading the site at example.test works and everything appears good. Obviously this will only work for a development environment. I have not tested provisioning a staging / production server on arm64... one step at a time. I can now at least continue developing in glorious real all day battery land and I'll just copy the files to an x86 based machine for the rest of the workflow until I can get more experience with the rest of it.
I'll follow up here with any issues as I find them.
@jgarib awesome, thank you so much! I'm not that surprised that this "just" worked. Ubuntu should be Ubuntu, so it makes sense it worked once you got that up and running.
re the jeffnoxon/ubuntu-20.04-arm64 box: the base Vagrant box shouldn't matter that much, as long as it's a standard Ubuntu image 👍
I'll grep the source again, but assuming MariaDB is the only place with a hardcoded architecture, we can use Ansible's ansible_architecture to support both (it just needs a mapping).
@jgarib did Nginx not fail with the same issue? It has deb [arch=amd64]. Either way, I'll fix both 👍
No issue with nginx, though you're right.. you'd think it would fail the same as mariadb. Here is the output of the nginx section of the provisioning with arch=amd64 still set. I can run a quick test with the same change to verify it still works if you'd like?
https://prnt.sc/1xv8e7c
I'm pretty sure we can just remove the deb arch options entirely. It defaults to the arch that dpkg supports. Running dpkg --print-architecture on my x86 VM just outputs amd64 as you'd expect. I'm assuming that if you run it you'll get back arm64. This makes sense to me since you shouldn't have to explicitly tell apt what your architecture is; it should just default to using your current arch.
This is working on the x86 side for me and I'm assuming it will work for M1s. https://github.com/roots/trellis/pull/1318
Confirmed working on ARM side. So once that is merged the only change needed to use trellis in a development environment on an M1 mac is:
- parallels pro
- reference the arm based vagrant box mentioned above (
jeffnoxon/ubuntu-20.04-arm64) in vagrant.default.yml or whatever change you make to support both! - ???
- profit
Thanks much Scott!
🎉 yep, but I can probably update the Vagrantfile to use that box automatically as well so ideally there's no manual steps needed.
Just chiming in to say that I've got Trellis working on an M1 Mac Mini by following these instructions with Parallels Pro.
Is it possible to set a default provider in the vagrant.default.yml file or anywhere else? It's not ideal to have to type the provider flag every time I provision a machine, though not the end of the world.
Trellis’s Vagrantfile tries to auto select a provider in this order:
- virtualbox
- vmware_fusion
- vmware_workstation
- parallels
- hyperv
if multiple providers are installed, first one will be used.
Alternatively, you can overrides it via the VAGRANT_DEFAULT_PROVIDER environmental variable.
See: https://www.vagrantup.com/docs/providers/basic_usage#default-provider
Yeah if you only have Parallels installed it should be the default? Is that not what's happening?
If you have an M1 mac then you definitely shouldn't have VirtualBox installed since it won't work anyway.
That was it. I had VirtualBox installed somehow from when i was first getting started on this M1 machine.
Also posting here to say that this discussion led to a resolution for me. I'm running Parallels on a trial right now and all systems go. I moved Parallels entry up in Vagrantfile so I wouldn't have to specify a provider. I don't have VirtualBox installed but it was still defaulting to it.
I'm running the full stack (Trellis/Bedrock/Sage) so I'd also like to reference the discussion on Sage's repo resolving an issue with Node-Sass while building with newer versions of Nodejs since previous versions aren't compatible with Apple Silicon. https://github.com/roots/sage/issues/2411
moved Parallels entry up in Vagrantfile so I wouldn't have to specify a provider. I don't have VirtualBox installed but it was still defaulting to it.
What is the error message when you vagrant up without --provider and without re-ordering Vagrantfile?
(Please test it on a brand new VM because vagrant "remembers" which provider was used on first vagrant up)