Need ability to pass a cluster name to GKE plugin from a previous drone step

[brian-lavery]

The problem: Hard coding cluster names into a GKE Drone step is not possible when dealing with Cloud Composer deployments.

We currently use a Drone Cloud Composer plugin to spin up a new Cloud Composer environment. The cluster name is generated dynamically by Google and is NOT predictable (e.g. us-central1-prd-1c7de959-gke). This is where the complication arises. The next step in our drone file uses the drone-gke plugin to deploy a pod running a Cloud SQL Proxy onto the newly created GKE cluster (the proxy is needed to allow secure communication with Cloud SQL DBs talked to by Airflow). The problem is that the GKE step to deploy the proxy pod doesn’t know the name of the just created Cloud Composer cluster because it can't be hard coded into the Drone file ahead of time.

What we'd like to do to address the problem: We'd like to modify the GKE plugin to allow the passing of a GKE cluster name from a previous drone step in the same pipeline. The only method for communication between Drone steps in the same pipeline run seems to be a shared volume that any step can read and write to. Here's the sequence of events for the solution we're thinking about: Drone step #1: create the Cloud Composer environment Drone step #2: capture the name of the cluster via a gcloud CLI call and save the name to a file on the shared volume. Drone step #3: The GKE plugin step examines the cluster name and if it starts with the prefix 'file-path:' (e.g. cluster_name: file-path: tmp/cluster_name.txt), then the plugin reads the cluster name from the file on the shared volume. If the prefix is not present then it uses the hard coded cluster name as is.

Alternatives we've considered: We could modify the Cloud Composer plugin to add the optional deployment of a SQL proxy onto the cluster but that would duplicate what the GKE plugin already does.

[msuterski]

When drone is running a build it checks out the repo to the workspace and executes the builds steps. Even if the build steps perform different tasks, the workspace (and the checkouted repo content do not change).

What if, instead of making changed to the plugin, we modified the cluster's name in the .kube.yml file in-between the composer and GKE steps? Something like:

1. Run composer plugin 2a. Read the new cluster name 2b. Run sed on the .kube.yml file with the new cluster's name
2. Run the GKE plugin

It would not require changes to the existing plugins, just one extra builds step.

[oliver-nyt]

The cluster name is configured via the drone-gke plugin in the .drone.yml file, not in the kube.yml file.

[msuterski]

Ah, that's true. It's the gcloud that needs the cluster name. 👍

How about we set the cluster's name through the gcloud config?

https://cloud.google.com/sdk/gcloud/reference/config/set

[tonglil]

Note that default values set from file (e.g. FilePath) take precedence over default values set from the environment (e.g. EnvVar). -- https://github.com/urfave/cli#values-from-files

This seems easy enough of a change to me: https://github.com/nytimes/drone-gke/blob/6ac83b93954ef35b0ff2d797b3f58e2d978aa975/main.go#L110-L113