terragrunt fails when TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE=true

[SerhiiSokolov]

I'm using terragrunt 0.38.9 hcl file has

dependency "tests" {
  config_path = "../../test/sources/"
  mock_outputs = {
    arn = "arn:aws:iam::1234567890:policy/mock_policy_id"
  }
  mock_outputs_allowed_terraform_commands = ["plan", "init"]
  mock_outputs_merge_strategy_with_state  = "shallow"
}

Dependency from the config has not been created yet.

If set TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE to true and run plan getting the error

DEBU[0001] Fetching outputs directly from s3://terraform-state-*****/****/test/sources/terraform.tfstate  prefix=[/*****/sources] 
ERRO[0002] NoSuchKey: The specified key does not exist.
        status code: 404, request id: ******, host id: ********** 
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 

[denis256]

Hi, the module was already applied? or it is first run?

[SerhiiSokolov]

Hi @denis256 It's the first run

[Udbv]

I encountered the same issue.

[Ido-DY]

@SerhiiSokolov @Udbv are you trying to apply changes only to a dependent object without its dependency? What Terragrunt command do you run (with args)?

[lev-ok]

Hi @SerhiiSokolov @Udbv,

Unfortunately I didn't manage to repro this issue. Could please provide your configs for the reproduction?

[simonholt]

I too have hit this problem (0.48.1, although I have confirmed it still occurs in 0.54.12). In my case it seems to be quite a specific (uncommon) combination of circumstances that trigger it:

We have a provider for the backend which looks something like this:

remote_state {
  backend = "s3"
  generate = {
    path      = "_backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket   = "my-state-bucket"
    key         = "${lower(replace(trimprefix(path_relative_to_include(), "../"), "/[0-9]{4}-/", ""))}.tfstate"
   ...

This simply uses the folder name as a basis for the state key, and uses the remote_state construct to get the dependency optimization working.

Folder "A" contains a terragrunt.hcl, which pulls in this backend, some other providers, and also a Terraform module

include "backend" {
  path = "${dirname(find_in_parent_folders("providers.hcl"))}/providers/backend_legacy.hcl"
}
terraform {
  source = "git::ssh://[email protected]/somethingsomething.git//my-module?ref=0.1.5"
}

Folder "B" has a dependency block like this (and similar backend/provider includes):

dependency "folder-a" {
  config_path = "../folder-a"
  mock_outputs = {
    role_arn = "arn:aws:iam::1234567890912:role/role-arn"
  }
  mock_outputs_allowed_terraform_commands = ["plan", "init", "validate", "fmt"]
  mock_outputs_merge_strategy_with_state  = "shallow"
}

Assume that these two folders are brand new, have never been run before, and don't have any corresponding state objects in S3.

Firstly, if I un-set TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE, Terragrunt works in all scenarios.

If I have TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE set to true, then:

• If I run terragrunt run-all plan against both folders, it plans fine, and uses mock outputs for the 2nd stack.
• If I run git clean -xfd, then run terragrunt plan on folder A, it plans fine.
• If I now run terragrunt plan against folder B, it plans fine.
• However if I run git clean -xfd, then run terragrunt plan on folder B, it fails with the "NoSuchKey: The specified key does not exist" message.
• If I apply folder A, then I can subsequently plan or apply folder B regardless of whether folder A was initialized (i.e. has a .terragrunt-cache folder)

The trigger seems to be that folder "A" pulls in a Terraform module, doesn't have a remote state object, and you're running a plan on folder "B" with state optimization enabled.
(For us, most of our Terraform code lives alongside terragrunt.hcl rather than pulling the module in like this, so we haven't seen this before)

[yiskaneto]

I recently came across this scenario, I usually always have TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE set to true, running "terragrunt run-all plan" always works, but the issue started to happen after we completely removed the state files , we did this because we restructured the project and after un-deploying all the resources we cleaned the remanent empty state files and locks, so this obviously will cause Terragrunt report back the key doesn't exists as the key itself is the name of the state file on S3.