Upgrading Debian on our VMs

[foolip]

Most of our VMs are running Debian 9.13 according to /etc/debian_version. We should upgrade to Debian 10. The process I would use for this is:

• take a snapshot of the running VM, making sure that nothing is deployed to it while the snapshot is taken
• create a new VM from that snapshot and run fsck or something to ensure no filesystem corruption
• upgrade that copy of the VM
• verify it's good by modifying /etc/hosts locally to point to the upgraded VM
• make it live by changing DNS or redirecting the floating IP in the cases we use that

Unfortunately this process isn't repeatable, and will lead to our setup scripts getting out of sync and unable to set up a new server from scratch. If we wanted to address that, we should probably make docker images instead, and keep filesystem and database outside of those images. This is however a much larger undertaking.

@annevk asked me to file this and cc @domenic + @sideshowbarker