Kubernetes and hosting revamp

[curtismchale]

Source Curtis

We should revamp Kubernetes (K8S) hosting with the following updates. Some of these will become big projects in their own right.

• better dashboard so we can see a site that's being slow
  • see traffic numbers
  • see "stuck" sites
  • reports on WP DB queries that are taking way too long
  • basic traffic reports so we can see if a site using resources goes with a traffic spike
• dev debug modes on live sites
  • so devs can interact with a site in debugging without users seeing it
  • better variables and logging when a site is pushed to debug mode
  • should be an easy command to enable this
• full auto-deploy based on branch without numerous commands in terminal
  • so staging to test sites
  • live branch to prod sites
  • all from a CI pipeline that does our unit tests and pushes to the expected site
  • should be able to push to a single site, or exclude a site as needed
  • pre-deploy automated backup
• offline backup somewhere regularly
  • could be weekly but it should be off our regular infrastructure and automated in some fashion
• single command rollback
  • platform wide rollback with single command
  • site specific rollback with single command
  • site specific rollback to backup with single command
• dev locally on full infrastructure from hosting
  • the same k8s setup used on live/test should be easily available for local dev
  • one line pull of client sites full file and DB
• faster code sync
• currently we wait 8 - 10 minutes for a code change manually made to show up for testing and this should be instant
• tested system for migration to other hosts
  • if Google has a big issue, can we easily migrate to Linode/Digital Ocean/other?
  • this should be fully tested documented and working if there is a long term issue with Google
• Ability to load plugins only on the sites they're used on
  • how do we check if they need updates though and make sure we stay on the current versions of the plugins? Currently site specific plugins are buried in site specific yaml and unless the plugin is installed on beta I'll never see that it needs updates. ELGL and their business directory is an example of this.
• look at helm or kustomize to roll out k8s package updates
• when moving a site from test to prod we need better checking to make sure that all values have been changed
  • see https://github.com/proudcity/wp-proudcity/issues/2310
• If we haven't got to it yet then add: https://github.com/proudcity/wp-proudcity/issues/2311
• This is a nice dashboard at MainWP
  • at least I like what it does and we should have most of these features built into our dashboard
• we should have a periodic (daily) check that all domains listed in a customer site are still pointing at our IP
  • if a customer has moved the domain then our SSL system won't run and it will take down all the sites with an invalid SSL
• we should have a notification for failed SSL provisioning that highlights the reason if it can be determined
  • specifically if there is an error message, what was it and what domain does it have to do with?