Testing—Mechanism for identifying breaking changes (e.g.: stateful resource replacement)

[bgshacklett]

Terraform Version

Terraform v1.7.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/archive v2.4.2
+ provider registry.terraform.io/hashicorp/aws v5.37.0
+ provider registry.terraform.io/hashicorp/http v3.4.1
+ provider registry.terraform.io/hashicorp/random v3.6.0
+ provider registry.terraform.io/hashicorp/time v0.10.0

Use Cases

As a platform engineer, I want to be able to detect breaking changes, such as replacement of stateful resources, across versions of terraform modules, so that I can reduce such changes, or update the major version of the module appropriately if they cannot be avoided.

Attempted Solutions

n/a

Proposal

Some potential solutions:

• Pre/post-run hooks Enable execution of actions, such as a git checkout, prior to and after run blocks.

  This could look something like:

  run "test_previous" {
    before = {
      exec   = "git checkout \"$(git describe --tags --abbrev=0)\""
    }
    after = {
      exec   = "git checkout \"$CI_GIT_SHA\""
    }
  
    run "test_current" {
      ...
    }
    ...
  }
  

• Partial execution, allowing control to be returned to the CI tool/shell

  run "test_previous" {
    pause = true
    # could be a number of different keywords; break, halt, etc.
  }
  
  run "test_current" {
    ...
  }
  

  #!/bin/sh
  git checkout "$(git describe --tags --abbrev=0)"
  terraform test
  git checkout "$CI_GIT_SHA"
  terraform test --continue
  ...
  

References

• #34356

[liamcervante]

Hi @bgshacklett, thanks for filing this!

We have another issue https://github.com/hashicorp/terraform/issues/34500 tracking a feature that will add the ability to validate the status of a particular resource during a plan (such as checking if it is being created/updated/deleted/replaced). Would that suit your needs here?

I don't really understand the connection between the issue title and your proposed solutions. Could you elaborate a bit more on how being able to execute git checkout commands will help you identify breaking changes? Is it as simple as wanting to update the configuration being tested between different module versions?

We do currently support loading alternate modules for executions into run blocks, so maybe a better solution here would be for us to find a way to allow run blocks from different modules to share the same state file. Then maybe you could write tests like this:

run "previous_version" {
  module {
    source  = "hashicorp/my-module"
    version = "0.1.0" # previous version
  }
}

run "current_version" {
  module {
    source  = "hashicorp/my-module"
    version = "0.2.0" # current version
  }

  # Introduce some way to validate none of the resources
  # are being updated.
  assert {
    # ...
    # not possible yet, tracked in #34500.
    # ...
  }
}

Hopefully my idea makes sense! Do you think that would enable you to test what you need to?

Thanks!

[bgshacklett]

My need largely boils down to identifying: "do the changes in this PR result in significant effort for those who would potentially consume the newer version of the module?". I left the title broad, because I didn't want to limit the discussion to a particular implementation.

In the past, to handle this using a custom test framework, I would:

1. check out the previous tag (not a specific tag, but just the most recent)
2. run an apply
3. check out the current SHA (merged result or branch)
4. run a plan
5. identify any potential problems
   1. errors generating the plan, such as incompatible inputs
   2. recreation of stateful resources or resources which could otherwise cause problems
6. run an apply
7. identify any potential problems...
8. report

That's one potential workflow (which TF doesn't currently support), but I suspect there are some other good ways to handle it. The example in your response looks like a good potential solution, with the benefit of being written in a declarative fashion. As written, however, there are two things that could stand to be improved:

• The module source must be specified. This isn't a big problem, but could lead to failures in the event that a repository is restructured.
• The version is specified statically. This is more problematic. It would require everyone to change the version of the previous module in the test for each pull request. This is easy to forget, and would also require that PRs are updated every time a new tag is added. That can be a significant amount of effort and is a very easy step to forget.

[bgshacklett]

I just ran into another issue with the above suggestion. It does not allow us to apply *_override.tf files against the previous version of the module, which we require for mocking/stubbing certain behaviors. At present, there are a number of capabilities that are lacking fro the built-in mock_provider and override_* capabilities, and we continue to rely on *_override.tf files to overcome those limitations in the meantime.

[brettcurtis]

We'd like to see something for this as well. I mentioned it in the last paragraph of this issue comment. For us, the problem is the loss of remote state other tools, like kitchen-terraform, had.