ol-infrastructure icon indicating copy to clipboard operation
ol-infrastructure copied to clipboard

Create a module for managing values in Consul key/value data

Open blarghmatey opened this issue 3 years ago • 0 comments

As we are migrating to use consul-template for our runtime configuration it introduces some complexity around how to manage the values that are used by the templates. To date we have used Pulumi to populate the key/value data in Consul but that introduces numerous decision points about which project to associate the data with. It also forces us to add arbitrary values to the stack configuration and additional resources that serve only to propagate that data.

In order to simplify the process of writing templates and their related values per environment we will add a module in the bridge package similar to what was done in the secrets module to manage SOPS secrets. This will be a simple function signature to allow reading data from the appropriate file so that it can be used in the ol_infrastructure or bilder packages. In addition, we will write a Concourse resource to allow us to populate the contents of the environment specific files into the appropriate Consul cluster so that the templates will have the proper values to consume.

For example, to populate the values that are used by the edxapp templates that render to lms.yml and studio.yml we would have YAML files in this module named mitxonline.qa.yaml, mitxonline.production.yaml, etc. corresponding to the appropriate deployment environment. The contents of that YAML file would be along the lines of:

edxapp/lms_domain: courses-qa.mitxonline.mit.edu
forum/domain: forum-qa.mitxonline.mit.edu
...

Those values would then be written out to edxapp/lms_domain, etc. in the Consul cluster running in the MITx Online QA VPC.

blarghmatey avatar Nov 04 '21 14:11 blarghmatey