boundary-plugin-kms-transit.gz: no such file or directory

[jsmilani]

Terraform Version

Terraform v1.2.7 on linux_amd64

• provider registry.terraform.io/hashicorp/boundary v1.0.10

Affected Resource(s)

Please list the resources as a list, for example:

• provider "boundary"

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

provider "boundary" {
  addr             = "https://boundary-controller.service.consul:9200"
  recovery_kms_hcl = <<-EOF
kms "transit" {
  purpose            = "recovery"
  address            = "https://[REDACTED]:8200"
  token              = "[REDACTED]"
  key_name           = "boundary-recovery"
  mount_path         = "transit/"
}

EOF
}

Debug Output

╷
│ Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/2607004110/boundary-plugin-kms-transit.gz: no such file or directory
│ 
│   with provider["registry.terraform.io/hashicorp/boundary"],
│   on provider.tf line 1, in provider "boundary":
│    1: provider "boundary" {
│ 
╵

Expected Behavior

terraform plan should load the provider and attempt to connect to the boundary instance or give an actionable error to address.

Actual Behavior

The error about not finding /tmp/*/boundary-plugin-kms-transit.gz is printed and it is unclear what went wrong. I am unaware of other steps needed to use the kms transit but it looked like it was supposed to just work. I was able to use this recovery_kms_hcl as a file with the boundary cli so I assume it isn't malformed.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform plan

Important Factoids

Running in an alpine docker image on OSX.

[louisruch]

I am not able to reproduce this in OSX natively, I can look into this a bit on Monday on a Linux box. The TF provider will exec the needed plugins when invoked these are stored in tmp files and the error you show is similar to where we would expect the gz to exist. We cleanup these when we exit, as an example here are the processes that are running while I run terraform apply that uses transit locally:

$ ps -a | grep boundary
83753 ttys001    0:00.01 grep boundary

$ ps -a | grep boundary
83783 ttys000    0:00.40 .terraform/providers/localhost/providers/boundary/0.0.1/darwin_amd64/terraform-provider-boundary
83786 ttys000    0:00.05 /var/folders/b3/wfnhszxj29x8ths3qpr494gm0000gp/T/393474464/boundary-plugin-kms-transit-hz8y3
83788 ttys001    0:00.00 grep boundary

$ ps -a | grep boundary
83783 ttys000    0:00.40 .terraform/providers/localhost/providers/boundary/0.0.1/darwin_amd64/terraform-provider-boundary
83786 ttys000    0:00.05 /var/folders/b3/wfnhszxj29x8ths3qpr494gm0000gp/T/393474464/boundary-plugin-kms-transit-hz8y3
83793 ttys001    0:00.00 grep boundary

$ ps -a | grep boundary
83820 ttys001    0:00.01 grep boundary

Something might be deleting this file before we can exec or we lack permissions to put it there, though I would expect to see an error relating to the perm.

[jsmilani]

This is running in docker so it is technically a linux VM, not native OSX.

I checked the permissions for /tmp and my user has write permissions and I can manually create folders. I can't tell if the boundary plugin creates the folder nested under it because it doesn't persist after running the command. The same "no such file or directory" error is a common one for a few operations and can occur if the the destination folder it is trying to create the file in doesn't exist. I suspect something related to creating the /tmp/* folders is not working but only displays an error when attempting to create the file in it since it doesn't exist. Also the /tmp/* directory is randomly named so I cannot proactively create it to ensure it exists.

[louisruch]

@jsmilani Sorry for the delayed response, I was able to test this in Linux but did not have the any issue on my side. Have you tried again since? I believe there is a new version of the provider out that you could attempt

[giacomocariello]

I'm having the same problem with latest version (1.0.12).

[micchickenburger]

Also having this issue with Boundary terraform provider 1.0.12 in a gitlab runner (docker linux image terraform:1.2.1) on amd64.

Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/17469073/boundary-plugin-kms-awskms-eQQGf: no such file or directory
 
   with provider["registry.terraform.io/hashicorp/boundary"],
   on deploy.tf line 39, in provider "boundary":
   39: provider "boundary" {

[louisruch]

@giacomocariello @micchickenburger did this work as expected in 1.0.11 or a previous version of the provider?

In the mean time I am going to try and spin up a docker environment to try and reproduce the issue of the original poster. As mentioned I was not able to reproduce this on a native Linux or Darwin machine.

[micchickenburger]

@louisruch Yes, it worked with 1.0.5, which I just upgraded from.

[micchickenburger]

I stepped back versions one at a time and here are my results.

Error 1:  Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/17469073/boundary-plugin-kms-awskms-eQQGf: no such file or directory
Error 2:  Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: plugin is nil

1.0.12:  Error 1
1.0.11:  Error 1
1.0.10:  Error 1
1.0.9:   Error 1
1.0.8:   Error 1
1.0.7:   Error 2
1.0.6:   No error

[louisruch]

@micchickenburger Ok this makes sense, in version 1.0.7 Boundary core and the terraform provider moved to having the plugins external to boundary and exec'ed from a tmp file instead of being build into the binary. However, in 1.0.7 we never actually included the plugins hence the nil plugin error. This indicates that this has been an error since that move and helps me with narrowing down the issue. Thank you for doing that!

[louisruch]

I have not been able to reproduce this error yet, if anyone that is having this issue could help with some detailed steps to reproduce the issue, including docker images etc being used that might help. In Boundary we allow the user to define the directory the plugin will exec from, I am going to look at adding this support in the next TF release.

[louisruch]

https://registry.terraform.io/providers/hashicorp/boundary/1.1.0 includes a new config in the provider plugin_execution_dir I would be interested to see if this resolves the issue that is occurring in the tmp storage.

[jsmilani]

Here are some generic steps to reproduce that only require Docker to be installed. I believe these are all the latest versions of Alpine, Terraform and the Boundary provider.

1. Create these 2 files in the same directory:

Dockerfile :

FROM --platform=linux/amd64 alpine:3.16.2

ENV TERRAFORM_VERSION=1.3.1

RUN apk add --no-cache --virtual .build-deps curl && \
    TERRAFORM_URL=https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    curl -o terraform.zip "$TERRAFORM_URL" && \
    unzip terraform.zip && \
    chmod +x terraform && \
    mv terraform /usr/local/bin/ && \
    rm -f terraform.zip && \
    mkdir /tf-kms-test

WORKDIR /tf-kms-test

COPY main.tf .

RUN terraform init

main.tf :

terraform {
  required_version = ">= 0.13"
  required_providers {
    boundary = {
      source = "hashicorp/boundary"
      version = ">= 1.1.0"
    }
  }
}

provider "boundary" {
  addr             = "https://boundary-controller:9200"
  recovery_kms_hcl = <<-EOF
kms "transit" {
  purpose            = "recovery"
  address            = "https://vault:8200"
  token              = "xyz"
  key_name           = "boundary-recovery"
  mount_path         = "transit/"
}

EOF
}

resource "boundary_scope" "org" {
  scope_id    = "global"
  name        = "abc"
  description = "ABC Inc."

  auto_create_admin_role   = true
  auto_create_default_role = true
}

2. Open a bash shell CD'd into that directory
3. run docker build -t tf-kms-test .
4. Start interactive container by running docker run --rm -it tf-kms-test sh
5. In container run terraform plan

Error:

Error: error reading wrappers from "recovery_kms_hcl": Error configuring kms: error fetching kms plugin rpc client: fork/exec /tmp/2354344686/boundary-plugin-kms-transit-B19hR: no such file or directory

[louisruch]

@jsmilani Thank you for the steps to repro the issue! I am able to see the error you are seeing now, couple of interesting things to note:

1. I was using a very similar setup to your steps before, the main difference I was pulling the hashicorp/boundary as the FROM instead of alpine:3.16.2. If I take your exact files and just use hashicorp/boundary I do not get the error, link to how we build the boundary docker file
2. However, if I use the base of that boundary file docker.mirror.hashicorp.services/alpine:3.13 I still get the error
3. I also tested it with Ubuntu and similarly do not see the error
4. I decided to test boundary directly as we also make use of plugins in the same way the provider does, and interestingly boundary did not give any error in alpine while the provider did give the no such file error.
5. I used inotifywait to monitor the tmp directory and I can see the terraform provider creating the plugin and then cleaned up after the error occurs.

I will try and spend some time tomorrow digging into some build differences between the provider and boundary core to see if anything shows up there. But in the mean time using a different base for your docker image could unblock you.

[micchickenburger]

@louisruch If it’s helpful, I’m using the latest terraform docker image and experience the errors there as well.

[louisruch]

@micchickenburger I just repro'ed it using the terraform docker image as well. Before doing that I was trying to figure out the differences and I noticed that adding libc6-compat seems to resolve the issue. I have not dug into exactly why still, but thought that since there is activity on this issue right now I could post some initial findings. If you could add libc6-compat to any RUN apk add command you have and verify if that does indeed resolve this.

[micchickenburger]

@louisruch That certainly solved the issue I was seeing! But now I'm seeing a new one: The KMS recovery plugin configuration doesn't seem to work. Someone else already opened an issue for it: https://github.com/hashicorp/terraform-provider-boundary/issues/282