feat: Development Server

[adityahase]

Our engineers have two options for their development environment.

Local VMs

Run VMs locally with Vagrant + Libvirt (KVM). This is only supported on Linux AFAIK (I'm the primary user). You must also build a good base image with packer.

You can fast-track this with our Vagrantfile and the bootstrap script.

Few downsides

1. VMs are memory-hungry (Elastic, MariaDB, etc). So you need a lot of RAM.
2. Running a whole cluster slows down Chrome significantly, not that it needs any help.
3. Setup is often bottlenecked by Internet speeds.
4. Eats a ~50 GB ~disk space.
5. Starting from scratch takes an hour (without any intervention).

Public Cloud

Run VMs on the Public Cloud. Most engineers use Hetzner for now.

Few downsides

1. Some infra features, provision, resize, etc are only supported on AWS and OCI. So engineers have to set that up now and then.
2. There are no base images to start quickly. Everybody has to start from scratch. Run Setup Server playbooks etc.
3. Almost every engineer has a broken setup. Mostly parts that need to run but they shouldn't be worrying about.
4. Almost nobody runs auxiliaries, Prometheus, ElasticSearch, etc.

My hypothesis is this high friction interface is slowing development here and there e.g. Analytics, Monitoring, Log features.

There's always the YOLO option to develop without any test infra.

---

This is an attempt to provide a VPC+EC2+EBS equivalent interface. I'm trying to solve two specifics

Quick Startup for Engineers

Most UI/Platform engineers aren't concerned about the specifics of OS, packages, etc.

So, provide a working set of images that can be used to spawn a full cluster in under a minute.

You can start directly by creating benches, sites, etc. No need for the first few steps e.g. create servers, run the "Setup Server" playbook, etc. Everything should work the way it works on production.

Faster Bootstrap for Onboarding

For first-time engineers or those who occasionally dabble in FC development, create even better "prepared" VM images + backups of Press.

Should have a few benches, sites, and related documents (domains, backups, migrations, etc), ready to go. So the features on top have as little setup costs as possible.

And this is how I attempt to solve those

Solution Specs

1. VMs run on a Baremetal server on Scaleway. These are cheaper and faster than VMs.
2. Agent endpoints as a wrapper on Vagrant. You don't need to have Vagrant running locally.
3. Wireguard tunnel that extends the VM network to the engineer's machine. The current architecture assumes every VM has its IP, but Scaleway won't give us many for free. IPv6 can solve this problem, but it doesn't work everywhere.
4. Every engineer gets a namespace to avoid name, and IP collisions.

[codecov[bot]]

Codecov Report

Attention: 27 lines in your changes are missing coverage. Please review.

Comparison is base (2f8d65c) 44.02% compared to head (1c38a24) 43.98%. Report is 5 commits behind head on master.

Files	Patch %	Lines
...e/doctype/development_server/development_server.py	0.00%	24 Missing :warning:
backbone/setup.py	0.00%	3 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1462      +/-   ##
==========================================
- Coverage   44.02%   43.98%   -0.04%     
==========================================
  Files         315      316       +1     
  Lines       21020    21044      +24     
==========================================
+ Hits         9254     9257       +3     
- Misses      11766    11787      +21     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.