Unable to call packer validate with secrets passed as environment variables

[dalgibbard]

Overview of the Issue

With a HCL2 format packer file like the following:

variable "environment" {
  description = "Environment we're creating images in"
  default     = "test"
  type        = string
}
data "amazon-secretsmanager" "my_secret" {
  name = "my_secret_id_${var.environment}"
}
locals {
  my_secret_string = data.amazon-secretsmanager.my_secret.secret_string
}

source "amazon-ebs" "this" {
    ....
}

build {
    ....
    provisioner "shell" {
        ...
        environment_vars = ["MY_ENV_VAR=${local.my_secret_string}"]
    }
}

Running a packer build is successful; but, running packer validate fails with:

Error: Failed preparing provisioner-block "shell" ""

  on packer_build.pkr.hcl line 168:
  (source code not available)

1 error(s) occurred:

* Environment variable not in format 'key=value': <unknown>

Packer version

v1.7.9 packer-plugin-amazon: 1.0.8

Operating system and Environment details

Ubuntu 20.04LTS in WSL2 amd64. Building AWS AMIs with amazon-ebs source.

[LennyCastaneda]

What is the status on this issue? I'm encountering the same bug with Pack unable to interpolate secrets manager value into variable such as kms_id. Had to comment out all validate comments in CICD and only run build. When running packer build template.pkr.hcl it works but packer validate should work too. It might jus be that there is no function calling the Secrets Manager API to get value and verify upon interpolation.

[Glyphack]

I can spend some time on this.

@LennyCastaneda could you provide me the version of packer and amazon plugin you are using? since the issue is old it's good to confirm that it still happens with a recent versino.

[LennyCastaneda]

Hi @Glyphack

Packer version: 1.8.2 Amazon plugin version: 1.21

$ packer validate base-ami.pkr.hcl 
Error: 5 error(s) occurred:

* "<unknown>" is not a valid KMS Key Id.
* "<unknown>" is not a valid KMS Key Id.
* "<unknown>" is not a valid KMS Key Id.
* "<unknown>" is not a valid KMS Key Id.
* "<unknown>" is not a valid KMS Key Id.

  on base-ami.pkr.hcl line 108:
  (source code not available)

[LennyCastaneda]

Hi @Glyphack @SwampDragons just wanted to follow up on this issue with packer validate not able to call the secrets manager datasource and pull in the secrets to use for validation. However when running packer build it is successful.

Let me know if you need anything else to help identify root cause. Thank you much for your help.

[Glyphack]

Hey @LennyCastaneda, the problem is clear and I think there's enough data to reproduce & fix the issue.

I haven't had the time to work on this but you'll get notification if any development happens here. I'm not a Hashicorp employee, and I'm working on the project in my free time. So it's possible to be some delay between my initial message and submitting a PR. I have to find some free time, and I'll work on this as soon as I can in the following days.

[Glyphack]

Hey @LennyCastaneda @dalgibbard I reproduced the issue, after looking into the code there seems to be a flag for handling this:

This works on the example in the first message.

❯ packer validate -evaluate-datasources build-test.pkr.hcl
The configuration is valid.

Could you test & see if this works? I can document this if it's not clear from the docs to what to do.

[LennyCastaneda]

@Glyphack

lenny:~/base-ami $ packer -v 1.8.2

lenny:~/base-ami $ packer validate -evaluate-datasources base-ami.pkr.hcl flag provided but not defined: -evaluate-datasources Usage: packer validate [options] TEMPLATE

Checks the template is valid by parsing the template and also checking the configuration with the various builders, provisioners, etc.

If it is not valid, the errors will be shown and the command will exit with a non-zero exit status. If it is valid, it will exit with a zero exit status.

Options:

-syntax-only Only check syntax. Do not verify config of the template. -except=foo,bar,baz Validate all builds other than these. -machine-readable Produce machine-readable output. -only=foo,bar,baz Validate only these builds. -var 'key=value' Variable for templates, can be used multiple times. -var-file=path JSON or HCL2 file containing user variables.`

[Glyphack]

Interesting 🤔 this is the link to documents: https://developer.hashicorp.com/packer/docs/commands/validate#evaluate-datasources

I'm not sure how is this happening. I'll look more.

[Glyphack]

Just wanted to give an update. I was not able to reproduce this issue. So I'm not sure how to proceed.

[LennyCastaneda]

@Glyphack no worries... what I had to do was upgrade to 1.8.6 and running command with the flag worked.

These are the commands I ran to resolve the issue:

xcode-select --install brew tap hashicorp/tap brew install hashicorp/tap/packer

Updated Packer template to: packer { required_version = "1.8.6" }

$ packer validate -evaluate-datasources base-ami.pkr.hcl The configuration is valid.

[pluto00987]

To add.. I also found the option -evaluate-datasources to allow validate to work for me in the same situation. However, I still think the behavior reported is a bug. Case in point:

This validates just fine environment_vars = ["STUFF="]

But this fails with the error 'not in key=value format' environment_vars = ["STUFF=${unhydraded.datasource.variable}"]

In the worst case, the unpopulated value should evaluate to null. But instead something strange is occurring in this situation.

[Glyphack]

@pluto00987 I think the more reliable behavior is to raise an error so the user know that this is not set. Otherwise the problem of not setting the value shows up later in another place and makes it hard to track back to this.