local: virtualize all the things

[bacongobbler]

A number of issues have cropped up using the local installer, mostly related to local system setup, permissions-based issues, and zombie processes causing indeterminate behaviour on future demo runs.

A short summary of these issues can be found here:

• https://github.com/fermyon/spin/pull/560#issuecomment-1154266103
• https://github.com/fermyon/installer/issues/43
• https://github.com/fermyon/installer/issues/36
• https://github.com/fermyon/nomad-local-demo/pull/13

Additionally, some engineers have questioned whether "tire kickers" would be comfortable installing a large number of tools on their system just to run a demo.

These issues have led to a very frustrating experience. In comparison, the aws installer is a breeze to debug and clean up because it all runs on a single EC2 instance. When the EC2 instance is torn down, the entire demo is cleaned up. Re-running the demo spins up a new EC2 instance. Orphaned processes from previous demo runs never collide.

IMO these class of issues would be moot if we virtualized the local installer. Users would only require the VM driver (xhyve for MacOS, WSl 2 for Windows, and QEMU for linux) and the spin CLI. When the VM is torn down, the entire demo is cleaned up, ready for a fresh run.

[bacongobbler]

@adamreese pointed to https://github.com/linuxkit/linuxkit/blob/master/docs/platform-hyperkit.md which may be a good jump-off point to spike this idea.

[itowlson]

Could we Dockerise it? Or do we need a higher degree of virtualisation?

[bacongobbler]

I recall someone mentioning nomad doesn’t run well in Docker. Whether it’s because of Docker-in-Docker issues or something else, I don’t know. But it’s worth exploring further.

[itowlson]

That sounds plausible. This chap (https://github.com/vancluever/docker-nomad) claims to have got it working but it seems to come with lengthy instructions and caveats - Hashi not releasing an official image might be a hint that it's not optimal...

[SamiKouatli]

Hello all,

I have tried to do something similar with Vagrant (with VMware on Windows) and some modifications of the local installer script + port forwarding.

So far, with a simple "vagrant up" command (and few command lines, because I have not finished yet) I can :

• Install Rust / Spin / Traefik / Consul / Nomad / Bindle / Hippo / Fermyon
• Launch Fermyon
• Access Consul, Nomad and Traefik on my host computer on 127.0.0.1 with different ports

But I have some issues on :

• Bindle server access, the URL is not accessible from my host machine.
• Hippo is accessible but HTTP requests (like sign-up, login...) fail with error 404.

I chose Vagrant to have a full VM, this way I can have a full dev/testing environment (with local DB and other local software interacting with my Spin project) which I can kill and re-deploy very quickly between several computers and with no further setup.