Allow for local terraform plan output when Terraform Cloud Execution Mode is set to Remote

[git-benjamin]

Terraform Version

Terraform v1.3.3

Use Cases

Issue:

• Attempting "terraform plan -out" will return "The "remote" backend does not support saving the generated execution plan locally at this time."

Feature Use Case:

• Allowing an output of a plan or speculative plan will enable Terraform Cloud users with "remote" execution to better run compliance tests, unit tests, and without compromising on the state file management.

Context:

• We have a functioning CICD pipeline utilising GitHub Actions and Terraform Cloud to deploy Snowflake resources.
• We wish to add policy as code utilising Open Policy Agent (conftest) that will run on a push on GitHub actions.
• Conftest requires an output of the Terraform Plan to run conftest policies against.

Attempted Solutions

We have changed Terraform Cloud to "local" execution and output the state file in GitHub Actions for testing which works as expected. An extract of our GitHub actions/workflow below.

  - name: Terraform Plan
    id: plan
    run: terraform plan -out=temp_plan -lock=false

  - name: Terraform Show
    id: show
    run: terraform show -json ./temp_plan > ./temp_json.json

  - name: Compliance Checks
    uses: addnab/docker-run-action@v3
    with:
      image: openpolicyagent/conftest:latest
      options: -v ${{ github.workspace }}:/build
      run: |
        cd /build
        conftest test -o github temp_json.json

Proposal

No response

References

No response

[apparentlymart]

Hi @git-benjamin! Thanks for this feature request.

Terraform Cloud has an API endpoint which returns content equivalent to the output of terraform show -json PLANFILE:

https://developer.hashicorp.com/terraform/cloud-docs/api-docs/plans#retrieve-the-json-execution-plan

Would the result from that endpoint give you what you need to run the analysis steps you want to run?

Terraform Cloud intentionally doesn't expose the true saved plan file because it can contain sensitive information and many Terraform Cloud customers prefer to therefore keep it encapsulated inside Terraform Cloud as an implementation detail. Therefore I don't think it would be acceptable to export a plan file exactly the same as what a local run would generate, but it might be possible to generate a special sort of "stub" plan file which only contains enough information to request the plan JSON from the API in terraform show. I'm not sure if it's worth the extra complexity when that data is already available from the API, though; I'd be curious to hear what you think and then pass this feedback on to the Terraform Cloud teams (who would be the ones to make any final call on this, anyway.)

[git-benjamin]

Hi @apparentlymart - thanks for the detailed response! 😊

We're currently using Terraform via GitHub Actions as per below.

name: "Deploy"

on:
  push:
    branches:
      - main

jobs:
  snowflake-terraform-demo:
    name: "Terraform Tests & Apply"
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1
        with:
          cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
          terraform_wrapper: false

I made a change to my local configuration then ran terraform plan followed by curl --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/vnd.api+json" https://app.terraform.io/api/v2/workspaces/$WORKSPACE_ID/runs > runs.json.

It looks like there is no run with the correct datetime in the .JSON. I'm assuming the API endpoint can only retrieve the execution plan of plan-only runs made via the Terraform API and possibly the plans from Terraform CLI of applied states.

[apparentlymart]

Hi again @git-benjamin!

I'm reading between the lines of your answer here a bit: I assume you are trying to access this "runs" endpoint because your automation wouldn't otherwise know the exact run ID to use when building a URL like this:

https://app.terraform.io/api/v2/runs/$RUN_ID/plan/json-output

Indeed, that does seem to be a big snafu in what I was describing: this API endpoint is only useful if you were already doing all of the other steps using the Terraform Cloud API, rather than with Terraform CLI. If you're using Terraform CLI then the run ID is only known to the CLI process and isn't visible externally in any way that you can use for later scripting.

I think this essentially answers my question about whether it would be useful to allow Terraform CLI to generate a "stub" plan file which just refers to the remote run information from the API: that would be one way for Terraform CLI to "remember" the remote run ID or plan ID so that you could later ask terraform show -json planfile.

Since we already treat the format of saved plan files as an opaque implementation detail anyway, it seems plausible in principle to use a different file format for runs from Terraform Cloud. We could define a new opaque format that has a different header (so that terraform show -json planfile can distinguish the two) and that only contains the minimal metadata needed to call that API endpoint: the selected hostname (app.terraform.io for managed Terraform Cloud) and the run ID.

Then terraform show -json planfile would need to probe the given file to see which of the two formats it is in. If it's in the new "stub" format then it would essentially just request the runs/$RUN_ID/plan/json-output endpoint directly and copy the response verbatim onto stdout.

There are some remaining design wrinkles to figure out here, though:

• What should terraform show planfile (not in JSON mode) do if it's given a stub plan file? There isn't an API endpoint for retrieving the human-oriented plan output from Terraform Cloud, so it seems like in this case terraform show would effectively need to reinterpret the plan JSON back into the human-oriented format, which is not something Terraform CLI has ever needed to do before now.
• Should it be okay to run terraform apply planfile with a stub plan file? It seems like that should probably follow the same rules we already have about whether it's okay to run terraform apply with remote operations enabled, which I believe means you're allowed to do it unless you have a VCS integration enabled on your workspace. However, Terraform Cloud runs have a more complicated state machine than a standalone local plan; this operation would presumably need to fail if the designated run isn't currently in the "planned" (waiting for approval) state.
• What should happen if the stub plan file includes a hostname that doesn't match the current configuration? Should it just try sending a request to the hostname that's recorded and see what happens, or should it return an error saying that the current configuration doesn't match?

These all seem like answerable questions, but they will be for Terraform Cloud teams to answer rather than for me to answer, so I'm just going to leave this here as context and pass this feedback on to the Terraform Cloud teams to see what they think. (They may also have an entirely different idea for how to solve it than what I proposed above, which is also fine! :grinning:)

[git-benjamin]

Cheers mate @apparentlymart 😊

[jackmenzie]

Any update on this @apparentlymart? It's handy to have the JSON file whenever you're writing out new OPA tests.

[busser]

The moved block generation tool tfautomv also uses the JSON format of the plan. It doesn't work on Terraform Cloud workspaces with remote execution for the reasons stated above.

Any way to reliably get a plan's JSON format from the command-line would be a boon. The solution suggested by @apparentlymart would be ideal.

[yermulnik]

Same use case for https://github.com/dineshba/tf-summarize

[pazmariano]

Hi @git-benjamin I landed into this issue by facing the same scenario (run tests before merge code). In may 4 2022 they have released run tasks (https://www.hashicorp.com/blog/terraform-cloud-run-tasks-are-now-generally-available). I wonder if this would work in the scenario you described, considering there are tasks for pre-plan, post-plan, pre-apply, post-apply. If so, that would be an option. My two cents on this.