Proposal for 2.0 - Move to proper CM tool to allow management of long-lived clusters

[dansimone]

Overview:

In order to apply some of the knowledge and lessons learned from managing Kubernetes clusters over time, as well as to resolve some long-standing bugs (e.g. #2, #24, #40, #41, #132), we would like to provide a mechanism for managing multiple cluster "environments" and move from a strictly Terraform approach to a Terraform + Ansible approach where Terraform is used for provisioning the OCI infrastructure and Ansible is used for software configuration management on the compute nodes.

Separating all software configuration into Ansible provides a more tenable strategy for rolling out changes to live clusters over time. It also accelerates feature development - for example, software changes made in Ansible can be easily tested in isolation without having to destroy and re-create compute instances.

Install process:

Instead of invoking Terraform only, the "installation" of a cluster will consist of calling Terraform, generating some data to pass to Ansible, then calling Ansible. We are proposing a CLI driver script to tie these steps together into a single call.

CLI Driver Script

The proposed CLI driver script would do the following:

• Provide a single interface that exposes all (Terraform and Ansible) configuration options (e.g. —k8s-version=1.8.4, —k8s_master_oci_lb_enabled=true, etc.).
• Create an environment directory for each cluster that contains the variables and state file for that cluster / environment
• Call terraform with the proper state and variables.
• Generate the Ansible host inventory, and other artifacts that are needed to software configuration via Ansible (SSH keys, k8s certs, etc)
• Call ansible playbooks against the above Ansible inventory.

After installation, a user can still call Terraform or Ansible directly to make updates (with extra care needed, as always, for Terraform updates). Also, it's always possible to bypass the CLI Driver script in the first place (if one wants to do that), by calling Terraform or Ansible directly.

Repository organization:

The common Terraform and Ansible configurations would each live in their own top-level directories. These configuration files are the cluster "blueprints" that change over time as the repository evolves. For each cluster that is provisioned, a new directory is automatically created by the helper CLI tool that contains the Terraform terraform.tfstate, terraform.tfvars, as well as the Ansible inventory files for that cluster. Additionally, each cluster directory would contain a reference to the commit ID or tag that the repository was at when that cluster was provisioned. Cluster directories are essentially analogous to "instances" of the configuration that enable the user to maintain a number of cluster over time - possibly at different versions.