Passing complex data structures through modules to providers without manually duplicating their type constraints in input variables

[engineertdog]

Terraform Version

Terraform v1.3.2
on windows_amd64

Use Cases

A use case I'm working through now is the ability to dynamically control terraform state through JSON files. This approach works well in most cases, except for one specific area: dynamic types.

I'll utilize an AWS resource as an example, but this would be true for any provider where a similar process would be desired. Let's say you're working with identity management and you want to dynamically define the permission policy in JSON and supply that to a terraform module. An example input variable would be such:

variable "configs" {
  description = "Policy Configurations"
  type = map(object({
    Name         = optional(string)
    NameOverride = optional(string)
    Description  = optional(string)
    Path         = optional(string)
    Policy       = dynamic
    Tags         = optional(map(string))
  }))
  default = {}
}

The item to pay attention to is Policy. This is the configuration that could be dynamic from file to file. Two policy file examples:

{
  "Name": "IAM_POLICY-MANAGE_INFRASTRUCTURE",
  "Description": "Allow infrastructure management",
  "Path": "/zone/env/app",
  "Policy": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "Tags",
        "Effect": "Allow",
        "Action": [
          "ec2:DeleteTags",
          "ec2:CreateTags",
          "ec2:DescribeTags"
        ],
        "Resource": "*"
      },
      {
        "Sid": "KeyPair",
        "Effect": "Allow",
        "Action": [
          "ec2:CreateKeyPair",
          "ec2:DeleteKeyPair",
          "ec2:ImportKeyPair",
          "ec2:DescribeKeyPairs"
        ],
        "Resource": "*"
      },
      {
        "Sid": "FSx",
        "Effect": "Allow",
        "Action": [
          "fsx:TagResource",
          "fsx:ListTagsForResource",
          "fsx:UntagResource",
          "fsx:DescribeFileSystems",
          "fsx:CreateFileSystem",
          "fsx:UpdateFileSystem",
          "fsx:DeleteFileSystem",
          "ds:DescribeDirectories"
        ],
        "Resource": "*"
      },
      {
        "Sid": "ACM",
        "Action": [
          "acm:RequestCertificate",
          "acm:DescribeCertificate",
          "acm:DeleteCertificate",
          "acm:ImportCertificate",
          "acm:ListCertificates",
          "acm:ListTagsForCertificate",
          "acm:AddTagsToCertificate",
          "acm:RemoveTagsFromCertificate",
          "acm:UpdateCertificateOptions"
        ],
        "Effect": "Allow",
        "Resource": "*"
      }
    ]
  }
}

{
  "Name": "IAM_POLICY-GET_SECRET",
  "Description": "Allows retrieving secrets from SecretsManager",
  "Path": "/zone/env/app",
  "Policy": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "SecretsManager",
        "Action": [
          "secretsmanager:GetSecretValue"
        ],
        "Effect": "Allow",
        "Resource": "*"
      }
    ]
  }
}

Attempted Solutions

With the current available types, any would not suffice as it tries to find a common type for the two policies outlined below. This results in the following error message:

Unsuitable value for var.configs set using the TF_VAR_configs environment variable: cannot find a common base type for all elements.

I could pass in the whole file as a string, but then the rest of the properties lose their validation. This also causes issues with some nested module dependencies.

Proposal

I feel that there are scenarios where you truly want a dynamic type. For the use case I mentioned above, the provider is going to perform the validation on the value of Policy when supplied to the provider. In this scenario, terraform is the limiting factor in being able to make the module work as desired.

My proposal is introducing a new type dynamic to terraform that, unlike any, does not try to perform any type casting on the values provided in order to prevent the error message shown above.

References

No response

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!

[apparentlymart]

Thanks for sharing this use-case, @engineertdog!

With Terraform as currently designed the "best" solution would be to actually describe the structure of IAM policies using Terraform's type system.

Of course, that would be non-trivial and a lot of overhead to need to redefine in every place where an IAM policy might appear. It would be similar to trying to reproduce the full schema of the aws_iam_policy_document data source from the hashicorp/aws provider: possible in principle, but annoying when the provider has already done that.

If it were more reasonable to define the actual type here though, I assume that would be preferable to leaving it completely dynamic because then Terraform would be able to type-check it, and fail early if the given data structure isn't a valid IAM policy.

If the language included some way to specify that you want a statically-typed IAM policy document at this location, would you agree that this would be preferable to a totally-dynamic attribute in this situation, or are there some details about your use-case that I'm not considering.

(I'm not yet proposing any particular solution to this problem of how to concisely describe the static type constraints for an IAM policy document; I'm curious to evaluate whether that even seems like a viable strategy before spending more time digging into the details.)

Thanks again!

[engineertdog]

For simplicity, I was fine with Terraform allowing a true dynamic value and allowing the provider to throw an error should there be a validation issue. As long as the provider's error message is brought through to the end user when running Terraform, I didn't see a need to bring the provider's types into Terraform.

It wouldn't be a terrible idea to have some provider types like aws_iam_policy.policy or aws_iam_role.assume_role_policy in Terraform and validated before being sent to the provider. But to your point, that would likely be a lot of overhead. That's why I thought letting the provider handle that validation wouldn't be a bad idea.

[apparentlymart]

Thanks for confirming, @engineertdog!

Given that, I think there is another potential design for this use-case:

Today Terraform only supports built-in types in its type constraints, but it would be possible in principle for providers to export as part of their schema a set of named type aliases that can then be used in the type constraints for variables in any module that requires a particular provider.

// INVALID: This is a hypothetical configuration that does not
// work in any existing version of Terraform.

terraform {
  required_providers {
    # This declaration is what gives meaning to the "aws."
    # prefix in the type constraint shown later.
    aws = {
      source = "hashicorp/aws"
    }
  }
}

variable "thing_with_policy" {
  type = map(object({
    # ...

    # Any leaf type constraint which includes a single
    # dot is understood as a type constraint alias
    # defined by the corresponding provider.
    policy = aws.iam_policy_document
  }))
}

A provider would presumably also declare some resource type arguments that expect values of the same type constraint, so that it would be possible to assign them directly without any further type conversions. However, I'm imagining this as something entirely separate from the idea of resource types and their attributes: any correlation between resource type schemas and exported type constraint aliases would be, from Terraform's perspective, just a convenient coincidence.

This allows propagating static type information offered by a provider out to the module's interface, so that it can be checked on entry to the module rather than on entry to the resource.

However, it also for the first time makes the full definition of a module's API dependent on the exact provider version selected. Today it's possible to generate module documentation or CDK for Terraform wrappers only with access to the source code of that particular module.

Presumably with this change CDK for Terraform would need to generate module bindings that in turn depend on the generated bindings for any providers a module uses, so that the parameters that correspond with the input variables can be given type constraints that will follow the definition of that type in whatever version of the provider the caller is currently using.

Both the CDK for Terraform team and the Terraform Language Server team both always advocate for more static type information, so I assume that if we were able to find a suitable resolution to the above problem of dealing with indirect provider dependencies in CDK for Terraform that both of these teams would be more in favor of exposing static type information indirectly from the provider rather than just making it totally dynamic.

---

I'm not meaning to assert that this other design is necessarily better than just allowing a dynamic-typing escape hatch, but I think we should evaluate the pros and cons of each before deciding either way.

And of course there is the third option of doing nothing and requiring passing policy documents in as JSON strings; you already described some downsides of that so I don't think we need to investigate that path any further but we should still weigh those downsides against the downsides of the two possible changes we could make since it may still end up being the best compromise, all things considered.

I don't expect we'll be able to do anything with this immediately since either way this will require some deeper consideration of the two possible solutions, but we'll keep this here for now to represent the use-case. (I'm also going to rename the issue title so that for now it describes just the problem to be solved and not any particular solution to that problem.)

[jamiet-msm]

I visited this repo today to request something very similar to what is being discussed in this thread so I'm going to add my twopenneth-worth here.

We frequently use google_service_account data source in our terraform configurations and we pass those data sources to various modules. When we do so we have to define the module variables like so:

variable "my_service_account" {
  type = object({
    email        = string
    name         = string
    display_name = string
    member       = string
    }
  )
}

(admittedly we never have to name all of the attributes because we don't refer to all of them, but we always need a subset of them).

In my opinion terraform could make this much easier by allowing us to define a type in terms of the resource/data source its expecting to receive. Something like this:

variable "my_service_account" {
  type = type(data.google_service_account)
}

or

variable "my_service_accounts" {
  type = map(type(data.google_service_account))
}

Learning how to craft complex data types isn't particularly easy, especially for newcomers to terraform. Allowing what I've proposed here would make it easier, convey more useful information to the reader, and (I think) be forward-compatible. it would also mean people wouldn't resort to

variable "my_service_account" {
  type = any
}

or

variable "my_service_accounts" {
  type = map(any)
}

or, even worse

variable "my_service_accounts" {}

which is what I see frequently.

[apparentlymart]

Hi @jamiet-msm,

My proposal above was more limited than what you've described here: I was proposing to allow providers to export specific named types that would not necessarily be full resource types, and would more likely be the types only for specific nested arguments such as an IAM policy like we've been discussing here.

While it might be possible in principle to echo out an entire resource type, I'm not sure it would be useful in practice because each time something new is added to the provider your module would appear to support that new attribute in its API but it would have no logic internally for actually making use of those new attributes, and so the caller setting them would do absolutely nothing until you also update your module.

If your module is exporting the entire API surface of a resource type then it isn't really adding anything: the caller could just write the same resource block out directly and not endure the extra complexity of a module without the corresponding benefit of raising the abstraction level.

The problem you described here (it being unclear how to declare an object type equivalent to a subset of the arguments to a resource type) is a real problem but I think that particular thing would be better solved with development-time tooling to help generate a fixed signature that will be saved as part of the module's own source code. It would start as a snapshot of some subset of the provider schema at the current selected version and would be frozen like that until the module author intentionally extends it, presumably at the same time as adding support for a new feature of the provider elsewhere in the module.

There are some community tools which already do some things like this, but they tend to just expose the entire resource API rather than only some relevant subset. It would be interesting to consider a tool which can take a variable whose type is currently any and analyze the rest of the module to see how it is being used and to propose a more specific type that matches that use, which would hopefully notice that e.g. the module is expecting a small number of specific attributes and so transform the type constraint into an object type describing only those attributes. It would not be perfect for all edge cases but presumably it only needs to make a good enough starting point that the author can see how to complete it.

[jamiet-msm]

Hello @apparentlymart , thank you as always for your very considered response and apologies to @engineertdog for hijacking their thread. I take your point, Martin, regarding unused attributes being declared in the variable and for that reason agree with you that this probably isn't the right solution. Your point about dev-time tooling is interesting, I'll harbour hopes that a tool will provide that one day (preferably, for me, that tool would be Hashicorp's terraform extension for VSCode).

[daveskywalkersilva]

I would like to reiterate the usefulness of this feature!

I am working currently on a use case for dynamic deployment of policies (to be more precise their definitions still) on Azure Cloud. My goal was to just split the arm_template you can retrieve from the cloud itself unto 3 jsons, one for each of the requested terraform parameters (meaning parameters, policyRule and metadata):

Policy ARM template:

{ 
  "displayName": "Azure Machine Learning Computes should have local authentication methods disabled",
  "policyType": "BuiltIn",
  "mode": "All",
  "description": "Disabling local authentication methods improves security by ensuring that Machine Learning Computes require Azure Active Directory identities exclusively for authentication. Learn more at: https://aka.ms/azure-ml-aad-policy.",
  "**metadata**": {
    "version": "2.1.0",
    "category": "Machine Learning"
  },
  "version": "2.1.0",
  "parameters": {
    "effect": {
      "type": "String",
      "metadata": {
        "displayName": "Effect",
        "description": "Enable or disable the execution of the policy"
      },
      "allowedValues": [
        "Audit",
        "Deny",
        "Disabled"
      ],
      "defaultValue": "Audit"
    }
  },
  "policyRule": {
    "if": {
      "field": "type",
      "equals": "Microsoft.MachineLearningServices/workspaces/computes"
    },
    "then": {
      "effect": "[parameters('effect')]"
    }
  },
  "directory_id": "/providers/Microsoft.Management/managementGroups/Test",
  "scope_id": "" 
}

Input Variable:

variable "policies" {
  description = "Map of policies to be created, including their definitions and, optionally, its scope."
  type = map(object({
    displayName   = string
    policyType    = optional(string, "Custom")
    mode          = optional(string, "All")
    description   = optional(string, "")
    metadata      = optional(any)
    parameters    = object({
      effect = object({
        type            = string
        metadata        = optional(any)
        allowedValues   = list(string)
        defaultValue    = optional(string)
      })
    })
    policyRule = **dynamic**
    #These need to be added aside from the ARM template:
    directory_id  = string #where the definition should be created
    scope_id      = optional(string, null) #the scope of it's assignment, if null there is no assignment
  }))
  default = {}
}

I just want to keep the json really inside the policyRule really, since the only action is to use it as it is on the deployment. Already tried with "any" also, but since sometimes the policyRule changes the fields, this breaks

[Iamwalrus1]

Providing another example using the AWS SNS Topic Subscription 'filter' argument. The argument requires a json string, however the structure is always unknown. Since the structure is always unknown, not sure how else to resolve except to use the suggested dynamic feature request.

main.tf:

resource "aws_sns_topic_subscription" "sns_test" {
  for_each = {for sqs in var.sns_test_topic_subscription: sqs.name => sqs}

  topic_arn     = "arn:aws:sns:us-east-2:1234567890:sns-test"
  protocol      = "sqs"
  endpoint      = each.value.endpoint
  filter_policy = jsonencode(each.value.filter)
}

variable "sns_test_topic_subscription" {
  type = set(object({
    name     = string
    filter   = optional(any, {})
    endpoint = string
  }))
}

.tfvars:

sns_test_topic_subscription = [
  {
    name     = "test1"
    endpoint = "arn:aws:sqs:us-east-2:1234567890:sqs-test1"
    filter = {
      "key1" : ["value1"],
    }
  },
  {
    name     = "test2"
    endpoint = "arn:aws:sqs:us-east-2:1234567890:sqs-test2"
    filter = {
      "key2" : [{ "key3" : ["value3"] }]
    }
  }
]

Error:

Error: Invalid value for input variable
│
│   on terraform.tfvars line 68:
│   68: sns_test_topic_subscription = [
│   69:   {
│   70:     name     = "test1"
│   71:     endpoint = "arn:aws:sqs:us-east-2:1234567890:sqs-test1"
│   72:     filter = {
│   73:       "key1" : ["value1"],
│   74:     }
│   75:   },
│   76:   {
│   77:     name     = "test2"
│   78:     endpoint = "arn:aws:sqs:us-east-2:1234567890:sqs-test2"
│   79:     filter = {
│   80:       "key2" : [{ "key3" : ["value3"] }]
│   81:     }
│   82:   }
│   83: ]
│
│ The given value is not suitable for var.sns_test_topic_subscription declared at sns.tf:10,1-39: element types must all match for conversion to set.