Test Feasibility of Citus GKE Node Pool Auto Upgrade

[jnels124]

Problem

we need to tests for stability of a running system experiencing a GKE upgrade

Solution

1. Create a cluster with a version behind the latest (disable auto upgrade)
2. Deploy common and mirror charts
3. Run 10k testing to start ingesting data
4. Manually trigger node pool upgrades for citus pools
5. Monitor logs during upgrade to observe behavior
6. Observe logs post upgrade to verify system recovers (importer may end up being in a hash mismatch state due to the time it takes to upgrade and may require a restart before becoming functional)

Alternatives

No response