buildmaster needs to leverage object storage

[kallisti5]

Constraints:

• 6 volume mounts per node (OS + 6)
  • https://docs.digitalocean.com/products/volumes/details/limits/
• Pods that share volumes must schedule on the same node
  • https://docs.digitalocean.com/products/volumes/details/limits/

Considerations:

• Persistent volumes are expensive. Object storage is cheap

Volumes today:

• buildmaster-sources (shared by every buildmaster)
• buildmaster-packages (shared by every buildmaster and frontend)
• buildmaster-x86-gcc2
• buildmaster-x86-64

We might be able to combine the buildmaster work volumes into one big shared pool of disk... but it really starts to shift this stuff back to a big pile of spaghetti. (however, this doesn't solve the high cost of direct attach storage either)