Feature: Capture STDOUT of terraform.plan

[glarizza]

My use-case is that I'm building a Jenkins pipeline that is triggered via Github PRs, and that pipeline will perform basic syntax validation (fmt/validate) followed by a plan run. I want to capture the output of terraform plan (not the plan file itself) and return that as a PR Comment so that users with access to approve/merge PRs understand what will happen from a Terraform perspective if that PR is merged. I don't think there's a way currently to wrap the terraform.plan call and catch that output; I'd imagine something like returnStdout: true would be needed for the sh() invocation that happens in the plan method.

[mschuchard]

Current closest capability would be something along the lines of print readFile(<config_dir>/plan.tfplan) or terraform show -no-color <config_dir>/plan.tfplan.

I can look into something more robust than this though to implement within the library.

[glarizza]

The issue with that is that the plan file is binary encoded and not legible. Only the output from terraform plan is in plaintext, so that would need to be captured when the command is executed.

I did something here which works, but the downside is that the output doesn't show up in the job's console output, so I'll need to revise this method so I capture the output AND it is displayed in the job log: https://github.com/glarizza/jenkins-devops-libs/commit/87981163415e87e9c0f5660a1100fe9e53a1610e

[glarizza]

Followed by something like this to echo out plan output: https://github.com/glarizza/jenkins-devops-libs/commit/58e2d165c3b4691e8548d13b8e3abaf492c7a3a8

If you think this method works I can open a PR to start the conversation there?

[mschuchard]

That kind of concerns me because, according to the terraform show documentation, executing that command on the plan file is supposed to produce human readable output for the plan.

I will have to take that into consideration for the implementation.

Do not worry about a PR. I can add this to the task tracking for the project and knock it out within the week when I return to this project.

Thanks for testing this stuff out.

[glarizza]

Ahh, you're correct - I reacted to reading the Plan file directly and not to the terraform show command (which I would expect to work). That's certainly an option without modifying the code and should get what we need.

[glarizza]

One big difference with using terraform show is that the output doesn't match the output of the initial plan (it doesn't get you the nicely formatted diff). There is a -json output for machine readable/parsable output, which is useful for being able to scrape through the output and determine create and destroy actions, but would require its own formatting. Here's the output of both terraform show commands:

[Pipeline] echo
Terraform plan was successful.


[Pipeline] sh
+ terraform show -no-color /var/lib/jenkins/workspace/project_one_multibranch_PR-4/project/one/plan.tfplan
+ google_compute_address.external


[Pipeline] sh
+ terraform show -json -no-color /var/lib/jenkins/workspace/project_one_multibranch_PR-4/project/one/plan.tfplan
{"format_version":"0.1","terraform_version":"0.12.6","planned_values":{"root_module":{"resources":[{"address":"google_compute_address.external","mode":"managed","type":"google_compute_address","name":"external","provider_name":"google","schema_version":0,"values":{"address_type":"EXTERNAL","description":null,"name":"jenkins-testing","project":"test-project","region":"us-west1","timeouts":null}}]}},"resource_changes":[{"address":"google_compute_address.external","mode":"managed","type":"google_compute_address","name":"external","provider_name":"google","change":{"actions":["create"],"before":null,"after":{"address_type":"EXTERNAL","description":null,"name":"jenkins-testing","project":"test-project","region":"us-west1","timeouts":null},"after_unknown":{"address":true,"creation_timestamp":true,"id":true,"network_tier":true,"self_link":true,"subnetwork":true,"users":true}}}],"configuration":{"provider_config":{"google":{"name":"google","version_constraint":"~\u003e 2.1"},"google-beta":{"name":"google-beta","version_constraint":"~\u003e 2.1"}},"root_module":{"resources":[{"address":"google_compute_address.external","mode":"managed","type":"google_compute_address","name":"external","provider_config_key":"google","expressions":{"address_type":{"constant_value":"EXTERNAL"},"name":{"constant_value":"jenkins-testing"},"project":{"constant_value":"test-project"},"region":{"constant_value":"us-west1"}},"schema_version":0}]}}}
[Pipeline] sh

And here's the original output from terraform plan:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_compute_address.external will be created
  + resource "google_compute_address" "external" {
      + address            = (known after apply)
      + address_type       = "EXTERNAL"
      + creation_timestamp = (known after apply)
      + id                 = (known after apply)
      + name               = "jenkins-testing"
      + network_tier       = (known after apply)
      + project            = "test-project"
      + region             = "us-west1"
      + self_link          = (known after apply)
      + subnetwork         = (known after apply)
      + users              = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

I think there's benefit in being able to capture the output of the plan, but obviously I'm biased :) Wondering your thoughts?

[mschuchard]

Yeah that default terraform show seems to kind of suck, and doing JSON would grab everything, but then I would have to parse the output and write my own reporter which becomes overkill.

Will go the route of capturing plan output.

[mschuchard]

Ok this is added to the terraform.plan method with the parameter display set to false by default. Note relevant doc for method here.

I also had to update a few tests and quash several bugs along the way, so my objective to update all my tests, re-run them, and fix bugs that have showed up will probably be my task on this project for a while.

New functionality passed acceptance test on my end, so feel free to try it yourself and respond accordingly.

[glarizza]

Is there currently any way to access the contents of the plan_output variable from the location where we're calling terraform.plan {}? I need to access that variable so I can send a comment back to the PR. The way I tackled it in my fork was to return the contents of the plan so I could capture it like so:

def output = terraform.plan {
  dir = "${localWorkspacePath}/${directory}"
}

If you know of another way, though, I'd be open to that.

[glarizza]

An example of what I'm doing (with an AWFUL lot of noise) can be viewed here: https://github.com/openinfrastructure/jenkins-terraform-pipelines/pull/1

[mschuchard]

Leave this issue open and I will come back to this later to see what kind of functionality I could add for this feature without impacting existing code architecture.

[mschuchard]

Attempted in 967ac6a20607cda421f3d9802c0919ece8f0faf0. I feel like this implementation is basically what you originally suggested, so I am not sure why I did not implement it immediately? I must have had a concern I have forgotten since then, so hopefully this commit does not cause a conflict.

Let me know how this looks to you for functionality.

[glarizza]

Yep, that's what I was looking for! I've since finished the engagement where I was going to use this code, and they decided to implement it via shell code first until they understood how Jenkins libraries worked, so I'm sure I'll come back around to it. Thanks!

[mschuchard]

Assuming this feature is functioning, and closing issue.