Add variables support

[tleguijt]

MACH should support variables in the configuration to support the following use-cases:

• Use CI secrets in your configuration
• Use 1 configuration file which can be used over various environments (for example 1 file for staging + production)
• Load in secret values from other sources:
  • Load variables from a SOPS encrypted file
  • Load variables from an AWS secrets manager
  • Load variables from an Azure KeyVault
• Use an Terraform output of one component as input for another
• Use another value from a Site definition (or child object) as input for a component

A couple of implementation ideas:

• [x] Support environment variables
• [x] Support variables from a regular JSON/YAML file
• [x] Support variables from a SOPS encrypted JSON/YAML file
• [x] Reference a component's Terraform output value
• [ ] Reference a MACH-managed resource/output (should be pre-defined and documented)
• [x] Determine syntax. A good example could be Serverless variables support: https://www.serverless.com/framework/docs/providers/aws/guide/variables

[pimvernooij]

Let's be cautious about this; because there is multiple 'types' of secrets that are managed through MACH composer, that might not be very clear to 'new' users;

• Secrets that are required at deploy-time of mach composer, i.e. your commercetools or sentry tokens)
• Secrets that are required at runtime of your microservice, i.e. the component-specific API client for commercetools

One could think that when encrypting the runtime secrets using SOPS, and then injecting it as an environment variable, would be secure. However that is not the case, as the secret would be decrypted at deploy-time, and injected in plain text as an environment variable.

It should be clear that those types of secrets, should be added as a 'secret' parameter for a component, so that MACH composer takes care of storing that secret in the secrets store (AWS secrets manager or Azure Key Vault at this time).

[tleguijt]

Can be solved with the correct syntax (in some cases).

If secrets are loaded in via a SOPS encrypted file, or via Secret Manager, we can expose them with ${secret.<var-name} for example. Once we detect usage outside of the secrets block (so within variables for example) we can output a warning for example.

[pimvernooij]

For providing this feature: Use 1 configuration file which can be used over various environments (for example 1 file for staging + production)

We could look in to using .tfvars files for providing this, and see if we can encrypt those using SOPS. Variables in those files could then be referenced in the YAML file?

Through one of these solutions:

• https://github.com/carlpett/terraform-provider-sops
• terraform apply -var-file=<(sops -d secret.tfvars.json) (source)

This might be easy to achieve? We could generate the required terraform code for this (as in the data source definitions) and then a JSON file per environment should exist.

[tleguijt]

Proposal for variable syntax: ${...} We could have 4 flavours for now;

• env. environment variables
• var. JSON/YAML file (with or without SOPS - can be auto-detected)
• component.<name>. Component output
• site. or global. MACH managed resource

Some examples:

env. environment variables

components:
  api-extension:
    variables:
      ORDER_PREFIX: ${env.FOO_VALUE}

Usage:

FOO_VALUE=some-value mach apply

var. JSON/YAML file (with or without SOPS - can be auto-detected)

commercetools:
  client_secret: ${var.${site.identifier}.commercetools.client_secret}

with:

mach apply -var-file=site-variables.yml

This can auto-detect if the variables file is encrypted with SOPS or not. If encrypted with SOPS it will use https://github.com/carlpett/terraform-provider-sops to reference the encrypted values (data.sops_file.site-variables.data["commercetools.client_secret"])

component.<name>. Component output

components:
  order-notifier:
    variables:
      MAIL_QUEUE_ARN: ${component.email-component.sqs_queue_arn}

site. or global. MACH managed resource

components:
  order-notifier:
    variables:
      MAIL_SUBJECT: New order on ${site.identifier}

or

components:
  api-extension:
    variables:
      REGION: ${global.azure.region}

---

To clarify the usage of ${var.} references a bit more;

If you have the following configuration for your component:

components:
    - api-extensions:
      variables:
        ORDER_PREFIX: ${var.order_prefix}

and var file variables.yml:

---
order_prefix: my-prefix-

mach apply -var-file=variables.yml will result in the Terraform output:

module "api-extensions" {
    variables = {
        ORDER_PREFIX = "my-prefix-"
    }
}

if variables.yml is encrypted with SOPS, MACH composer can auto-detect that and render the following:

data "local_file" "variables" {
  filename = "variables.yml"
}

data "sops_external" "variables" {
  source     = data.variables.yaml.content
  input_type = "yaml"
}

module "api-extensions" {
    variables = {
        ORDER_PREFIX = data.sops_external.variables.data.order_prefix
    }
}