Implement `pipeform` for Terraform Operations

[osterman]

Describe the Feature

Improve the console DX of terraform operations by presenting the user with a friendly UI to show terrafom progress.

https://github.com/magodo/pipeform

Expected Behavior

Running atmos terraform plan and atmos terraform apply will launch the pipeform UI. Note, this does not require pipeform to be installed, as it will be used as a native Go module.

Use Case

On the command line, the output from terraform is overwhelming and doesn't convey the overall progress of the operation. Terraform will feel much friendlier if we present the user with a progress indicator.

Describe Ideal Solution

The UI would be enabled by default, but can be disabled via the atmos.yaml configuration.

# atmos.yaml
terraform:
  ui:
    enable: [plan, apply]

https://github.com/user-attachments/assets/701671cd-f361-4c4e-83a7-98687d331c28

Requirements:

• [ ] If no TTY is attached, the UI should be automatically disabled
• [ ] If the CI=true environment variable is set, then UI should be automatically disabled
• [ ] If the --no-ui flag is explicitly passed, then the UI should be disabled.

Alternatives Considered

No response

Additional Context

The pipeform library is licensed MPL2 and is not currently implemented as a Go module. We've contacted the @magodo (author) and he said he would accept a PR converting it to a library. As part of this implementation, we will need to first open a PR for that work.

[suzuki-shunsuke]

pipeform is interesting, but honestly I'm not sure if we should really use this by default. I tried pipeform a bit. When we use pipeform, the plan result doesn't output in the terminal.

$ terraform plan -json | pipeform
# No output

I think this is inconvenient. I'm not sure if we can convert the output of terraform plan -json to the conventional terraform output.

Furthermore, seems like we can't show the detail of changes. I expected that we can choose a resource and show its detail, but seems like we can't.

By the way, the final view of terraform plan -json | pipeform looks strange. I think we can contribute to pipeform.

pipeform shows nothing. This confused me. I found I needed to move page by h and l key.

[osterman]

pipeform is interesting, but honestly I'm not sure if we should really use this by default. I tried pipeform a bit. When we use pipeform, the plan result doesn't output in the terminal.

$ terraform plan -json | pipeform
# No output

This sounds like a bug in pipeform. Per the docs, it says it's supported.

the following terraform commands are supported:

terraform refresh -json
terraform plan -json
terraform apply -auto-approve -json

---

Furthermore, seems like we can't show the detail of changes. I expected that we can choose a resource and show its detail, but seems like we can't.

This would definitely be a welcome enhancement. It would likely benefit your tooling for terraform as well. We can sponsor this development in pipeform, if you're interested.

---

By the way, the final view of terraform plan -json | pipeform looks strange. I think we can contribute to pipeform.

Yes, this is not good. It should probably work the way you described above, allowing individual resources to be expanded. They shouldn't "disappear".

---

Ideal World

• When using pipeform, raw terraform logs are written to disk, allowing further inspection upon failure
• The UI would enable inspection of any resource
• A flag like --ui=false would disable the UI on a per-run basis.
• When a TTY is not detected, or we're in CI (env), then UI is disabled

[suzuki-shunsuke]

This sounds like a bug in pipeform. Per the docs, it says it's supported.

Really? I can't find the docs. In my envrionment, TUI is shown, but after TUI quits, no output is shown in the terminal.

[osterman]

@suzuki-shunsuke here is what I am referring to. Are we talking about the same thing?

[suzuki-shunsuke]

I think it means pipeform can provide TUI for these commands, but it doesn't mean pipeform can output Terraform's conventional outoputs to stdout.

[suzuki-shunsuke]

For pipeform to output the conventional Terraform command output, pipeform needs to convert the output of terraform plan -json to the conventional Terraform command output. This means the output of terraform plan -json needs to include all information of the conventional Terraform command output.

I checked the output of terraform plan -json, but seems like it doesn't include the information of resource attributes.

For instance, terraform plan -json outputs the data like this:

{
  "@level": "info",
  "@message": "github_issue_label.fooo: Plan to create",
  "@module": "terraform.ui",
  "@timestamp": "2025-01-15T22:33:03.343780+09:00",
  "change": {
    "resource": {
      "addr": "github_issue_label.fooo",
      "module": "",
      "resource": "github_issue_label.fooo",
      "implied_provider": "github",
      "resource_type": "github_issue_label",
      "resource_name": "fooo",
      "resource_key": null
    },
    "action": "create"
  },
  "type": "planned_change"
}

We can see the resource github_issue_label.fooo is created, but we can't see the attributes of the resource. We can't convert this output to the conventional output like this:

  # github_issue_label.fooo will be created
  + resource "github_issue_label" "fooo" {
      + color       = "FF0000"
      + description = "foo"
      + etag        = (known after apply)
      + id          = (known after apply)
      + name        = "fooo"
      + repository  = "hoge"
      + url         = (known after apply)
    }

[suzuki-shunsuke]

So unfortunately, to provide both TUI and the conventional output, we need to run terraform plan twice.

[suzuki-shunsuke]

to provide both TUI and the conventional output, we need to run terraform plan twice.

TUI has no meaning if you run terraform plan twice 😅

[osterman]

@suzuki-shunsuke if the objective is to also support the regular terraform output, I think that could still be accomplished.

  # github_issue_label.fooo will be created
  + resource "github_issue_label" "fooo" {
      + color       = "FF0000"
      + description = "foo"
      + etag        = (known after apply)
      + id          = (known after apply)
      + name        = "fooo"
      + repository  = "hoge"
      + url         = (known after apply)
    }

Isn't the workaround as simple as:

terraform plan -json -out planfile.out | pipeform
terraform show planfile.out

[suzuki-shunsuke]

Oh, I see. Sounds good.

[suzuki-shunsuke]

We need to improve pipeform.

1. Pipeform can't show the detail of resources from TUI
2. terraform plan -json | pipeform finishes at a last page which doesn't show any resources

We need to discuss these problems with pipeform's maintainers.

[osterman]

We need to discuss these problems with pipeform's maintainers.

I'll get the conversation started by opening an issue in the project and linking to this topic.

[osterman]

@suzuki-shunsuke the terraform plan handling was just resolved in:

• magodo/pipeform#10

[suzuki-shunsuke]

I'm working on it.

[suzuki-shunsuke]

• https://github.com/magodo/pipeform/issues/11

[suzuki-shunsuke]

When terraform plan finishes quickly, pipeform may annoy because users need to interact pipeform and need to exit it by Ctrl-C.

So I'm wondering if this should be enabled by default.

The UI would be enabled by default, but can be disabled via the atmos.yaml configuration.

[suzuki-shunsuke]

I've created a pull request.

• https://github.com/cloudposse/atmos/pull/981

This is still work in progress.

[suzuki-shunsuke]

How to install pipeform

There are several options:

1. Install pipeform automatically somehow
2. Disable pipeform by default, and let users install it by themselves
3. Embed pipeform as Go library

You would prefer 3.

About 3, this is what you mentioned at https://github.com/magodo/pipeform/issues/9 .

We would like to refactor portions of the code into a public package from internal/

A detailed issue describing the needs is welcome

[suzuki-shunsuke]

I tried to replace pipeform command with Go package.

• https://github.com/cloudposse/atmos/pull/983
• https://github.com/suzuki-shunsuke/pipeform/tree/feat-api

[suzuki-shunsuke]

https://github.com/cloudposse/atmos/issues/926#issuecomment-2594342392

This workaround is unavailable for terraform apply because it doesn't support -out option.

[suzuki-shunsuke]

1. What do you think? https://github.com/cloudposse/atmos/issues/926#issuecomment-2621481511 https://github.com/cloudposse/atmos/issues/926#issuecomment-2620348005
2. You can build and try https://github.com/cloudposse/atmos/pull/983 . I really appreciate it if you try it and give me your feedback, whether the experience meets your expectation.

I tried it:

go install .
cd examples/quick-start-simple
~/go/bin/atmos tf plan station -s dev

[osterman]

@suzuki-shunsuke Yes, I agree with you. Something is off with this from the DX for terraform plan. Let me think what we can do to improve this.

Instead, maybe we should strictly focus on apply for now. Also, the two-key press for Ctrl+C to exit the UI, is not user friendly. If we're contributing to pipeform, this might be one thing to consider adding an alternative key stroke of just q to quit.

[osterman]

Also, looks like a bug during plan, it says apply:

[osterman]

Aha, interesting, so navigating with the ← and → keys, you can traverse between the various phases of the plan. Instead, it appears that the summary is "broken" as it doesn't summarize anything.

[suzuki-shunsuke]

maybe we should strictly focus on apply for now.

About apply, we need to think about this. https://github.com/cloudposse/atmos/issues/926#issuecomment-2621481511

This workaround is unavailable for terraform apply because it doesn't support -out option.

[suzuki-shunsuke]

• https://github.com/cloudposse/atmos/pull/983#issuecomment-2621528772

Using pipeform as Go library makes atmos depend on CGO. In general, depending on CGO is undesirable. This is a trade-off.

[suzuki-shunsuke]

I'm concerned that UI of pipeform may confuse users who aren't familiar with it.

[suzuki-shunsuke]

Problems (Discussion topics)

There are some problems, so let's manage the list in this comment.

• [ ] Improve UX of pipeform
  • [ ] Ctrl-C isn't user-friendly
    • https://github.com/cloudposse/atmos/issues/926#issuecomment-2620348005
    • https://github.com/cloudposse/atmos/issues/926#issuecomment-2621898230
  • [ ] UI of pipeform may confuse users who aren't familiar with it
    • https://github.com/cloudposse/atmos/issues/926#issuecomment-2623348384
    • https://github.com/cloudposse/atmos/issues/926#issuecomment-2621908084
• [ ] apply can't show the conventional output
  • https://github.com/cloudposse/atmos/issues/926#issuecomment-2621481511
  • https://github.com/cloudposse/atmos/issues/926#issuecomment-2623392341
• [ ] Go library
  • [ ] Need to add public API to the upstream
    • https://github.com/cloudposse/atmos/issues/926#issuecomment-2621449586
  • [x] pipeform requires CGO https://github.com/cloudposse/atmos/issues/926#issuecomment-2623346758
    • [x] Build failed in CI https://github.com/cloudposse/atmos/pull/983#issuecomment-2621528772
    • Solved. https://github.com/cloudposse/atmos/issues/926#issuecomment-2623465765
• [ ] CLI
  • [ ] Need to install pipeform somehow https://github.com/cloudposse/atmos/issues/926#issuecomment-2620393705

[osterman]

• https://github.com/cloudposse/atmos/pull/983#issuecomment-2621528772

Using pipeform as Go library makes atmos depend on CGO. In general, depending on CGO is undesirable.

Ugh! Yes, I agree with the implications on long term maintainability for a noncore feature. I am surprised by this! I assumed it was just a thin charmbracelet veneer. What about it depends on CGO?

[osterman]

https://github.com/cloudposse/atmos/issues/926#issuecomment-2594342392

This workaround is unavailable for terraform apply because it doesn't support -out option.

Ah, that's unfortunate, I didn't think about that. 🤦‍♂️ so there are a few things though, and I am not sure this is a deal breaker by itself. In a CI context the UI would be disabled, and tools like tfcmt would work well.

You've raised some big concerns and the scope may be larger than initially anticipated, in which case will reevaluate what we do.

At this point let me do some more research and get back to you.