Sharing variable definitions between modules

[sudoforge]

Current Terraform Version

1.2.2

Use-cases

This may be a bit of a unique beast, so bear with me; I'll try to make this as clear as possible.

When composing a large monorepo of modules, one often ends up with two forms of modules: those which are "internal", and those which compose various "internal" modules into a larger public interface. To provide a concrete example, let's assume we have the following modules:

• An internal label module, which is used to generate some formatted, well-known outputs (e.g. id, default_tags)
• An internal aws-s3-bucket module, which is used to create an S3 bucket within AWS with some sane defaults
• An internal aws-cloudfront-distribution module, which is used to create a Cloudfront Distribution with some sane defaults
• An internal aws-route53 module, which is used to create some R53 resources
• A public aws-static-app module, which composes the above internal modules into a single "mega-module" which is exposed to end users

When composing the "internal" modules into the aws-static-app module, you'd need to re-define the variables that each of the modules defines. This becomes tedious to maintain; to solve this, you might want to symlink the file defining variables for the internal modules into the aws-static-app module's directory. Of course, there are some variables which you don't want users of the aws-static-app module to change, so instead, you split those out into another file which you do not symlink: variables.internal.tf. This causes users of the aws-static-app mega-module to not have those variables defined (or required), allowing the mega-module's implementation to take care of managing the values passed into the internal modules.

On disk, this might look something like the following:

.
└── lib
    └── terraform
        ├── aws-cloudfront-distribution
        │   ├── main.tf
        │   ├── output.tf
        │   ├── variables.internal.tf
        │   ├── variables.label.tf -> ../label/variables.tf
        │   └── variables.tf
        ├── aws-route53
        │   ├── main.tf
        │   ├── output.tf
        │   ├── variables.internal.tf
        │   ├── variables.label.tf -> ../label/variables.tf
        │   └── variables.tf
        ├── aws-s3-bucket
        │   ├── main.tf
        │   ├── output.tf
        │   ├── variables.internal.tf
        │   ├── variables.label.tf -> ../label/variables.tf
        │   └── variables.tf
        ├── aws-static-app
        │   ├── main.tf
        │   ├── output.tf
        │   ├── variables.aws-cloudfront-distribution.tf -> ../aws-cloudfront-distribution/variables.tf
        │   ├── variables.aws-route53.tf -> ../aws-route53/variables.tf
        │   ├── variables.aws-s3-bucket.tf -> ../aws-s3-bucket/variables.tf
        │   ├── variables.label.tf -> ../label/variables.tf
        │   └── variables.tf
        └── label
            ├── main.tf
            ├── output.tf
            └── variables.tf

Where the files above serve the following purposes:

• main.tf: Where the bulk of the module's logic and resource/data blocks are defined
• variables.tf: Where each module's "public variables" are defined
• variables.internal.tf Where each module's "private variables" are defined (for consumers using this module directly, like the mega-module in our example)
• output.tf: Where each module defines its output variables

This works pretty beautifully. Some of the HCL in lib/terraform/aws-static-app/main.tf might look like this:

module "label" {
  source = "../label"

  # These variable definitions come in from the `variables.label.tf` symlink
  name        = var.name
  environment = var.environment
  team        = var.team
  tags        = var.tags
}

module "route53" {
  source = "../route53"

  # This module also consumes the label module, so we pass in the same variables
  name        = var.name
  environment = var.environment
  team        = var.team
  tags        = var.tags

  # The variables used below are defined in the `variables.aws-route53.tf` symlink
  r53_zone_name    = var.r53_zone_name
  r53_zone_records = var.r53_zone_records
  ...
}

The issue with this approach comes when there's a "mega-module" which has a unique requirement to avoid exposing a variable which is typically "public" (that is, in an internal module's variables.tf file as opposed to its variables.internal.tf file). Let's say that I wanted to create another "mega-module" which wanted to have a static value for var.environment, and not expose that variable to the user: this is where this approach falls short, because the new "mega-module" would be symlinking lib/terraform/label/variables.tf into lib/terraform/some-mega-module/variables.label.tf, which would include the variable "environment" {} block.

The solution would be to define a local variable and use that instead:

locals {
  environment = "some-static-value"
}

module "label" {
  source = "../label"

  # These variable definitions come in from the `variables.label.tf` symlink, but we use a local variable for `environment`
  name        = var.name
  environment = local.environment
  team        = var.team
  tags        = var.tags
}

However, this doesn't remove the definition of the environment variable and still exposes that to the user of this new "mega-module", even though it isn't used. I can provide a non-git-ignored override file, but that doesn't really solve the problem either.

Proposal

A great way to be able to handle this would be to allow for unsetting a variable; something like:

unset "environment" {}

... although that feels a bit clunky. Perhaps another property on the variable block could be added, akin to:

variable "environment" {
  ignore = true
}

which could be set in an included override file, or somewhere else in the "mega module" that wanted to explicitly ignore a variable.

---

A third, and probably more cleaner and more robust solution, would be to provide a method for including the body of a variable from another module, such that the mega-module doesn't use a symlink farm to include internal module variables, but rather, for each variable that it was to re-define, does something like:

variable "environment" {
  source = "../label"
  <overrides here>
}

[crw]

Thanks for this detailed enhancement request!

[theogravity]

I think this issue is the same as mine:

• I have a main module I work from
• I have private modules I created that exposes variables (variables.tf) for the main module to access
• I end up having to copy the variables.tf defs of those private modules into my main module
• I'd like to have an option to pass-through the variable defs to my main module so I don't have to define them again:

# main.ts in my main module, which includes a sub-module
module "application" {
  source     = "../modules/application"
  # new module option to auto-expose the variables to this main module
  # var.<variable def> in this main module should now be accessible
  inherit_variables = true
}

[sudoforge]

@theogravity that would be a great way to support this sort of workflow, but perhaps that's a map instead of a boolean, so that this particular feature request can be supported. Something like...

module "application" {
  source = {
    path = "../modules/application"

    variables = {
      # maybe this is a bool to include or exclude all
      include = true

      # and allow excluding by name here
      exclude = [
        application_foo_var,
        application_bar_var,
        application_baz_var,
      ],
    }
  }
}

This feels much cleaner to me than my proposals listed above. What are your thoughts?

[sudoforge]

That does have the downside of dealing with a module that is sourced twice, though.

[theogravity]

It's a good idea - I forgot that I do tend to omit certain variables.

Definitely for inclusion/exclusion config.

[sudoforge]

@theogravity I forgot to mention that it sounds like the issue you originally commented about could be solved by symlinking the variables.tf files from your private modules to the mega-module(s), like I described in the original comment. This approach works well until you run into the case of wanting to exclude a particular variable that is (typically) included.

[redzioch]

That does have the downside of dealing with a module that is sourced twice, though.

Yes... but creating third module which can be use inside other is working and the most simple solution.

I use similar approach to share values between not only different modules, but also different projects i.e. common values for dev and prod environments.

[sudoforge]

@redzioch can you expand what you mean?

From reading your comment, I'm assuming you use some context module that might look like this:

module "context" {
  source = "../path/to/module"

  some_variable    = "foo"
  another_variable = "bar"
  ...
}

which is then passed into another module:

module "app" {
  source = "../path/to/app"

  context = module.context
  ...
}

This doesn't solve the problem described in my original comment on this thread.