Test Terraform plan

[bitfield]

Firstly, thanks for Terratest—it's fantastic.

I've already used it successfully to write some integration tests for Terraform modules. However, these are very slow. That's not Terratest's fault; it just takes a while to create cloud resources, wait for them to be available, and destroy them again.

I'd like to also write some unit tests that I can run very quickly; all I need to do is get the Terraform plan, and verify one or two things about it. I can run terraform.InitAndPlan(), and get an exit code that indicates whether or not the plan is a no-op. This is very useful, but I'd like to do more. I'd like to get the list of target resources, for example, and check some of their attributes.

Here's a simple example. If my Terraform code creates an Azure VM named 'testvm', I'd like to test that:

• [x] The code plans without errors
• [x] The plan is not a no-op
• [ ] The plan contains one target
• [ ] The target is an azurerm_virtual_machine named testvm
• [ ] The target's tags include the string terraform

(I've ticked the things I can already do with Terratest.)

Now, I can call terraform.GetExitCodeForTerraformCommandE myself with the required arguments to save a plan file, open the plan file and parse it myself using the Terraform library, and so on. But a way to do this directly in Terratest would be a big timesaver.

For example, I'd like to be able to write something like this:

terraform.InitAndPlan(t, terraformOptions)
p := terraform.GetPlan()
if len(p.Targets) != 1 {
	t.Fatalf(...)
}
if p.Targets[0] != "azurerm_virtual_machine.testvm" {
	t.Fatalf(...)
}
...

[brikis98]

This is a great idea 👍 A PR to add this is very welcome.

There is a gotcha: the format of plan output changes quite often, and the folks at HashiCorp have said that they will not be committing to a stable format in the next few releases. However, there are a few possible solutions:

1. Open source libraries: https://github.com/palantir/tfjson and https://github.com/lifeomic/terraform-plan-parser look interesting.
2. terraform show: Once Terraform 12 is out, the terraform show command will be able to parse plan files (i.e., the output of terraform plan -out=<FILE>) and return JSON. Read more here: https://www.terraform.io/docs/internals/json-format.html

I suspect the second option will be the way to go: run terraform plan -out=<FILE>, run terraform show -json <FILE>, parse the resulting JSON, and return it as a Go struct.

[bitfield]

In previous programs I've used the terraform library itself to open and modify plans. I agree that parsing the output of any command is too fragile to be worth considering.

I'll try and do a proof of concept for a PR. Thanks for the feedback.

[bitfield]

Currently this is a blocker: https://github.com/hashicorp/terraform/issues/21121

[bitfield]

Some interesting discussion on that issue. The Terraform project has an unusual attitude to their Go library; they don't support it! In other words, it's not intended to be a stable public API.

Terraform 0.12 will add a terraform show -json PLANFILE command, which is intended to be the publicly consumable interface to plan files. I think any facility in Terratest which talks to plan files would be wise to use show -JSON if available, but fall back to terraform.ReadPlan() otherwise.

[nmiodice]

I went down a very long road trying to use the new Terraform plan package that is included with Terraform 12. Long story short, it is a complete nightmare to work with and is not really intended to be used outside of the core terraform package.

On a related note, I have been working on some unit testing capabilities. I'm not sure if you feel they would be leverage from within Terratest, or if they fit the style of Terratest, but it allows you to compare expected inputs/outputs in the plan using a subset match algorithm that leverages the output of terraform show -json

Here is the core library I am working on. I'd love to contribute this back to Terratest in some form. It does a bit of automation as well, which may or may not be relevant for terratest. Core library Core Library: Parsing Terraform Plan Core Library: Comparing Expected vs Actual Plan Usage example

[tiny-dancer]

EDIT: not a gruntwork/terratest team member, simply another happy community member :)

@nmiodice looks interesting, we're building out an opinionated terraform wrapper for development, testing and deployment of a "terraservices"-like approach and we're planning to incorporate very similar features (and ideally contribute into terratest), will hopefully get a chance to dig into your repo in the coming week(s).

The TL/DR of the wrapper if interested (open source ~1month):

Based on a dockerized folder convention automate the terraform execution and "glue code" code of one to many "terraservices" in a highly optimized way.

tracks/ # all directories are executed concurrently
├── iamsso # executes concurrently to `network`
│   ├── step1_awsiam # executes concurrently to `step1_onprem_adgroups` and `network.step1_vpc`
│   │   └── tests # executes after `step1_awsiam` (receives output variables as TF_VAR environment variables; go test files are compiled and layer cached in container build)
│   └── step1_onprem_adgroups # executes concurrently to `step1_awsiam` and `network.step1_vpc`
└── network # executed  concurrently to `iamsso`
    ├── step1_vpc
    └── step2_egress_proxy # executes after `step1_vpc` (receives output variables from `step1_vpc` as TF_VAR environment variables)

In regards to "unit tests", we're looking to include a source controlled expected assertion for each release that the wrapper can automatically verify prior to apply.

[bitfield]

@nmiodice infratests looks very cool indeed! Does it need to be part of the Cobalt project? It might be a nice idea to put it under its own GitHub project. That simplifies issue tracking and so on.

[hyp0th3rmi4]

@bitfield : Perhaps I misunderstand the issue here, but isn't the function: func InitAndPlanAndShowWithStruct(t testing.TestingT, options *Options) *PlanStruct sufficient for what you need to do?

You get a plan struct that you can navigate to check how many changes and planned variables are created. I use this to inspect and verify that the plan is configured as I expect based on the content of terraform.tfvars.

   // this is a custom function that ultimately invokes `InitAndPlanAndShowWithStruct` 
   // once we have resolved the `terraform.tfvars` associated to the given `fixturePath`
   plan, options := executePlanWithFixture(fixturePath) 

   // this is another function that navigates the generated plan and checks its structure 
   // against the settings defined in `terraform.Options`
   verifyPlan(plan, options) 

What I have found is that if you have many of these tests - and this may occur in case of multiple validation unit test cases - you end up running init, plan, and show for each test. This is a bit taxing on the CI/CD. What you really would like to do is wrap several of these test in a single test suite and run init once before all the tests, if your purpose is to verify only the plan without changing some of the conditions defined by init.

This would be easily achievable if the function https://github.com/gruntwork-io/terratest/blob/master/modules/terraform/plan_struct.go#L32 is exposed as a public method for the module. At present time this is internal to the package and cannot be invoked.

I imagine that given the volatility of the parsing and the possible continuous changes in the structure of the json plan, this may be not desirable.

[bitfield]

Perhaps I misunderstand the issue here, but isn't the function: InitAndPlanAndShowWithStructsufficient for what you need to do?

Indeed it is, and thanks for the tip! Many years after my original request, that function was added, in 0ee1f11a784a3bc492674f3526be7d368047d7e8. I'm happy to say that I no longer need it, having found other ways to solve the problem—but it may well be useful for someone else. Thanks to the maintainers for dealing with this.

[edify42]

@bitfield can probably close this as it's 2 years old and solved :)

I didn't like InitAndPlanAndShowWithStruct as it outputs the entire plan in an ugly manner. I ended up calling the InitAndPlan twice via:

terraform.InitAndPlan(t, terraformOptions)
// store plan in useable struct
plan := terraform.InitAndPlanAndShowWithStructNoLogTempPlanFile(t, terraformOptions)