terraform icon indicating copy to clipboard operation
terraform copied to clipboard
verified publisher icon hashicorp

panic when variable used in dynamic blocks was requested before it was provided

Open limakzi opened this issue 1 year ago • 4 comments

Terraform Version

$ terraform version
Terraform v1.9.7 
on linux_amd64

Terraform Configuration Files

  • File: root/github-lab/provider.tf 
terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "6.3.1"
    }
  }
}

provider "github" {
  owner = "<REDACTED>"
}
  • File: root/github-lab/teams/github_importer_lab/terraform.tf 
terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "6.3.1"
    }
  }
}
  • File: root/github-lab/teams/github_importer_lab/main.tf 
module "repositories" {
  for_each    = local.repositories
  source      = "../../../../modules/github-repository/"
  name        = each.key
  template    = lookup(each.value, "template", null)
  is_template = lookup(each.value, "is_template", false)
  ## [redacted]
}
  • File: modules/github-repository/terraform.tf 
terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "6.3.1"
    }
  }
}
  • File: modules/github-repository/variables.tf 
## [redacted]
variable "template" {
  default = null

  type = map(any)
}
## [redacted]
  • File: modules/github-repository/main.tf 
resource "github_repository" "this" {
  name        = var.name
  description = var.description

  visibility             = var.visibility
  has_issues             = var.has_issues
  has_projects           = var.has_projects
  has_wiki               = var.has_wiki
  has_downloads          = var.has_downloads
  auto_init              = var.auto_init
  is_template            = var.is_template

  ## [redacted]
  dynamic "template" {
    for_each = var.template[*]
    content {
      owner      = lookup(template.value, "owner", "<REDACTED>")
      repository = lookup(template.value, "repository", "github-repository-template")
    }
  }
  ## [redacted]
}

Debug Output

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

panic: value for module.team_github_lab.module.repositories["repo"].var.template was requested before it was provided
goroutine 259 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x5e
github.com/hashicorp/terraform/internal/logging.PanicHandler()
	github.com/hashicorp/terraform/internal/logging/panic.go:84 +0x18b
panic({0x2e7a540?, 0xc00e6e9a30?})
	runtime/panic.go:770 +0x132
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1.1()
	github.com/hashicorp/terraform/internal/terraform/graph.go:59 +0x4c5
panic({0x2e7a540?, 0xc00e6e9a30?})
	runtime/panic.go:770 +0x132
github.com/hashicorp/terraform/internal/namedvals.(*values[...]).GetExactResult(0x3e3a6e0, {{0xc006a71240, 0x2, 0x2}, {{}, {0xc008534618, 0x8}}})
	github.com/hashicorp/terraform/internal/namedvals/values.go:88 +0x219
github.com/hashicorp/terraform/internal/namedvals.(*State).GetInputVariableValue(0x2fbd960?, {{0xc006a71240, 0x2, 0x2}, {{}, {0xc008534618, 0x8}}})
	github.com/hashicorp/terraform/internal/namedvals/state.go:47 +0xe8
github.com/hashicorp/terraform/internal/terraform.(*evaluationStateData).GetInputVariable(0xc000e60fc0, {{}, {0xc008534618?, 0x20736569726f7469?}}, {{0xc005409b60, 0x27}, {0x1c, 0x10, 0x426}, {0x1c, ...}})
	github.com/hashicorp/terraform/internal/terraform/evaluate.go:284 +0x249
github.com/hashicorp/terraform/internal/lang.(*Scope).evalContext(0xc000e61050, {0xc00e6e9850, 0x2, 0x2}, {0x0, 0x0})
	github.com/hashicorp/terraform/internal/lang/eval.go:387 +0x1c72
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalContext(...)
	github.com/hashicorp/terraform/internal/lang/eval.go:246
github.com/hashicorp/terraform/internal/lang.(*Scope).ExpandBlock(0xc000e61050, {0x3e19b50, 0xc0017b71e0}, 0x0?)
	github.com/hashicorp/terraform/internal/lang/eval.go:38 +0xa5
github.com/hashicorp/terraform/internal/terraform.(*BuiltinEvalContext).EvaluateBlock(0x1?, {0x3e19b50, 0xc0017b71e0}, 0xc006ac5650, {0x0?, 0x0?}, {{{{0x0, 0x0}}, {0x0, 0x0}}, ...})
	github.com/hashicorp/terraform/internal/terraform/eval_context_builtin.go:316 +0xaa
github.com/hashicorp/terraform/internal/terraform.(*NodeAbstractResourceInstance).plan(0xc007262908, {0x3e3d608, 0xc00bfc54a0}, 0xc00cb2fd40, 0xc0078e78c0, 0x0, {0x0, 0x0, 0x0})
	github.com/hashicorp/terraform/internal/terraform/node_resource_abstract_instance.go:842 +0xbcb
github.com/hashicorp/terraform/internal/terraform.(*NodeApplyableResourceInstance).managedResourceExecute(0xc0026d7d40, {0x3e3d608, 0xc00bfc54a0})
	github.com/hashicorp/terraform/internal/terraform/node_resource_apply_instance.go:278 +0xafa
github.com/hashicorp/terraform/internal/terraform.(*NodeApplyableResourceInstance).Execute(0x18?, {0x3e3d608?, 0xc00bfc54a0?}, 0x80?)
	github.com/hashicorp/terraform/internal/terraform/node_resource_apply_instance.go:145 +0x9a
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc0007e1a40, {0x3e3d608, 0xc00bfc54a0}, {0x705ea6966600, 0xc0026d7d40})
	github.com/hashicorp/terraform/internal/terraform/graph_walk_context.go:153 +0xbb
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x3501ba0, 0xc0026d7d40})
	github.com/hashicorp/terraform/internal/terraform/graph.go:143 +0x83d
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc0077d4a80, {0x3501ba0, 0xc0026d7d40}, 0xc00b4d1500)
	github.com/hashicorp/terraform/internal/dag/walk.go:384 +0x2d7
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update in goroutine 103
	github.com/hashicorp/terraform/internal/dag/walk.go:307 +0xff3
Error: Terraform exited with code 11.
Error: Process completed with exit code 1.

Expected Behavior

No crash.

Actual Behavior

Crashes.

Steps to Reproduce

  1. terraform -chdir=root/github-lab/ init
  2. terraform -chdir=root/github-lab/ plan -out=tfplan-github.tfplan -refresh=false
  3. terraform -chdir=root/github-lab/ apply tfplan-github.tfplan

Additional Context

I do use one provider definition in root/github-lab/terraform.tf. The only provider, I use is integrations/github.

  • Providers.
Providers required by configuration:
.
├── provider[registry.terraform.io/integrations/github] 6.3.1
├── module.team_github_lab
│   ├── provider[registry.terraform.io/integrations/github] 6.3.1
│   ├── module.access
│   │   └── provider[registry.terraform.io/integrations/github] 6.3.1
│   └── module.repositories
│       └── provider[registry.terraform.io/integrations/github] 6.3.1
  • File tree.
.
├── README.md
├── modules
│   ├── github-repository
│   │   ├── main.tf
│   │   ├── terraform.tf
│   │   └── variables.tf
│   └── github-repository-access
│       ├── main.tf
│       ├── terraform.tf
│       └── variables.tf
└── root
    └── github-lab
        ├── import.tf
        ├── main.tf
        ├── provider.tf
        ├── repositories
        │   └── github_importer_lab.yaml
        ├── teams
        │   └── github_importer_lab
        │       ├── main.tf
        │       ├── terraform.tf
        │       └── variables.tf
        └── terraform.tf
  • Plan itself.
  # module.team_github_lab.module.repositories["repo"]
  # (imported from "repo")
  ~ resource "github_repository" "this" {
        allow_auto_merge            = false
      ~ allow_merge_commit          = true -> false
        allow_rebase_merge          = true
      ~ allow_squash_merge          = true -> false
      ~ allow_update_branch         = false -> true
      + archive_on_destroy          = true
        archived                    = false
      ~ auto_init                   = false -> true
        default_branch              = "master"
      ~ delete_branch_on_merge      = false -> true
        description                 = null
        etag                        = [redacted]
        full_name                   = "[redacted]"
        git_clone_url               = "[redacted]"
        has_discussions             = false
        has_downloads               = false
      ~ has_issues                  = true -> false
      ~ has_projects                = true -> false
        has_wiki                    = false
        homepage_url                = null
        html_url                    = "[redacted]"
        http_clone_url              = "[redacted]"
        id                          = "[redacted]"
        is_template                 = false
        merge_commit_message        = "PR_TITLE"
        merge_commit_title          = "MERGE_MESSAGE"
        name                        = "[redacted]"
        node_id                     = "[redacted]"
        primary_language            = "Groovy"
        private                     = true
        repo_id                     = 872958661
        squash_merge_commit_message = "COMMIT_MESSAGES"
        squash_merge_commit_title   = "COMMIT_OR_PR_TITLE"
        ssh_clone_url               = "[redacted]"
        svn_url                     = "[redacted]"
        topics                      = []
        visibility                  = "private"
      ~ vulnerability_alerts        = false -> true
        web_commit_signoff_required = false
    }

References

  • #35843
  • #35097

limakzi avatar Oct 15 '24 23:10 limakzi

Hi @limakzi, thanks for filing this!

Are you able to share the configuration around the var.template variable within module.team_github_lab.module.repositories["repo"]? The crash is happening when something is trying to read that variable. I'd be most interested in anything that reference that variable, the variable definition itself, and the module block that provides the value for the variable.

In addition, it would be great if you run this again with the debug logs enabled: export TF_LOG=trace. This will produce a lot of output, but it will help us narrow down the order of operations that could be causing something to read the variable before it has been initialised.

Without the above pieces of information it will be difficult for us to replicate and fix this. Thanks!

liamcervante avatar Oct 16 '24 08:10 liamcervante

@liamcervante, thank you for quick response. I have updated the issue with:

  • modules/github-repository/variables.tf -- variable definition;
  • modules/github-repository/main.tf -- variable usage;
  • root/github-lab/teams/github_importer_lab/main.tf -- module block that reference variable;
  • plan itself -- execution plan.

While the issue appears irregularly, (the exact same plan is executed many times during the day), I will try to put TRACE-log as well.

limakzi avatar Oct 16 '24 09:10 limakzi

Just to add some more context here, The definitions of the provider are not relevant to the panic, the error is related to the evaluation of the template variable. Since the provider behavior isn't relevant, TF_LOG_CORE=trace may produce substantially less output when you're trying to get logs.

The use of var.template[*] is unnecessary, so might be having an impact here. var.template is defined as a type of map, but the special [*] operator only applies to lists, sets, or tuples. This creates a special case where the terraform builds a list of a single object being the original map, which means there really is no reason for the dynamic block, and the map values could be assigned directly via a normal block.

The linked issue was related to overrides, are there any other special considerations like that which may come into play here?

The plan output also shows the resource as being imported during the plan, can you show how that is happening?

jbardin avatar Oct 17 '24 18:10 jbardin

@jbardin Thank you for the context.

The linked issue was related to overrides, are there any other special considerations like that which may come into play here?

Partially. Before the first error report, we had two providers in the state: integrations/github and hashicorp/github. We thought the main reason was the difficulty in navigating the directed-acyclic-graph with such defined dependencies. We noticed similar error-stacks then. That's main reason we put such subject.

The plan output also shows the resource as being imported during the plan, can you show how that is happening?

Yup.

import {
  id = "repo"
  to = module.team_github_lab.module.repositories["repo"].github_repository.this
}

Unfortunately, I cannot confirm or deny that this problem occurs only during import.

It seems, this crash does not occur when using version 1.8.1, but I can confirm that at the end of the week as the crash appears irregularly.

limakzi avatar Oct 20 '24 13:10 limakzi

@jbardin Another update.

a. Version constraint.

It seems, this crash does not occur when using version 1.8.1, but I can confirm that at the end of the week as the crash appears irregularly.

That certainly happened to be not true; it occurred many times since last update even in 1.8.1.

b. Star [*] operator.

This creates a special case where the terraform builds a list of a single object being the original map, which means there really is no reason for the dynamic block, and the map values could be assigned directly via a normal block.

I have changed the code a little bit since we want to use dynamic block there for purpose 9as the provider does not accept nullable owner and repository arguments)

resource "github_repository" "this" {
  name        = var.name
  description = var.description
  # [redacted]

  dynamic "template" {
    for_each = var.template != null ? ["0"] : []
    content {
      owner      = lookup(var.template, "owner", "owner")
      repository = lookup(var.template, "repository", "repo")
    }
  }
  
  # [redacted]
}

And it seems like similar stack appeared today again:

panic: value for module.team_team.module.repositories["repo"].var.template was requested before it was provided
goroutine 369 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.22.1/x64/src/runtime/debug/stack.go:24 +0x5e
github.com/hashicorp/terraform/internal/logging.PanicHandler()
	/home/runner/work/terraform/terraform/internal/logging/panic.go:84 +0x18b
panic({0x2e00120?, 0xc00e01f850?})
	/opt/hostedtoolcache/go/1.22.1/x64/src/runtime/panic.go:770 +0x132
github.com/hashicorp/terraform/internal/namedvals.(*values[...]).GetExactResult(0x3d956c0, {{0xc001339280, 0x2, 0x2}, {{}, {0xc00911be28, 0x8}}})
	/home/runner/work/terraform/terraform/internal/namedvals/values.go:88 +0x219
github.com/hashicorp/terraform/internal/namedvals.(*State).GetInputVariableValue(0x2f401c0?, {{0xc001339280, 0x2, 0x2}, {{}, {0xc00911be28, 0x8}}})
	/home/runner/work/terraform/terraform/internal/namedvals/state.go:47 +0xe8
github.com/hashicorp/terraform/internal/terraform.(*evaluationStateData).GetInputVariable(0xc003c5ae10, {{}, {0xc00911be28?, 0x6c6261796c707041?}}, {{0xc003d0d350, 0x27}, {0x1c, 0x10, 0x426}, {0x1c, ...}})
	/home/runner/work/terraform/terraform/internal/terraform/evaluate.go:286 +0x246
github.com/hashicorp/terraform/internal/lang.(*Scope).evalContext(0xc003c5aea0, {0xc00e01f6d0, 0x2, 0x2}, {0x0, 0x0})
	/home/runner/work/terraform/terraform/internal/lang/eval.go:386 +0x1c72
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalContext(...)
	/home/runner/work/terraform/terraform/internal/lang/eval.go:245
github.com/hashicorp/terraform/internal/lang.(*Scope).ExpandBlock(0xc003c5aea0, {0x3d75bf0, 0xc0036702c0}, 0x0?)
	/home/runner/work/terraform/terraform/internal/lang/eval.go:37 +0xa5
github.com/hashicorp/terraform/internal/terraform.(*BuiltinEvalContext).EvaluateBlock(0x1?, {0x3d75bf0, 0xc0036702c0}, 0xc00cd6b860, {0x0?, 0x0?}, {{{{0x0, 0x0}}, {0x0, 0x0}}, ...})
	/home/runner/work/terraform/terraform/internal/terraform/eval_context_builtin.go:313 +0xaa
github.com/hashicorp/terraform/internal/terraform.(*NodeAbstractResourceInstance).plan(0xc00d0686c8, {0x3d985e8, 0xc00df5a3c0}, 0xc00b9ad520, 0xc001fe1ec0, 0x0, {0x0, 0x0, 0x0?})
	/home/runner/work/terraform/terraform/internal/terraform/node_resource_abstract_instance.go:808 +0xb8b
github.com/hashicorp/terraform/internal/terraform.(*NodeApplyableResourceInstance).managedResourceExecute(0xc00d6e8740, {0x3d985e8, 0xc00df5a3c0})
	/home/runner/work/terraform/terraform/internal/terraform/node_resource_apply_instance.go:278 +0xafa
github.com/hashicorp/terraform/internal/terraform.(*NodeApplyableResourceInstance).Execute(0x18?, {0x3d985e8?, 0xc00df5a3c0?}, 0x70?)
	/home/runner/work/terraform/terraform/internal/terraform/node_resource_apply_instance.go:145 +0x9a
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc000648780, {0x3d985e8, 0xc00df5a3c0}, {0x7438a7f66c60, 0xc00d6e8740})
	/home/runner/work/terraform/terraform/internal/terraform/graph_walk_context.go:153 +0xbb
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x346ff80, 0xc00d6e8740})
	/home/runner/work/terraform/terraform/internal/terraform/graph.go:112 +0x62a
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc009b64a20, {0x346ff80, 0xc00d6e8740}, 0xc00825fe40)
	/home/runner/work/terraform/terraform/internal/dag/walk.go:384 +0x2d7
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update in goroutine 75
	/home/runner/work/terraform/terraform/internal/dag/walk.go:307 +0xff3

limakzi avatar Oct 28 '24 10:10 limakzi

Thanks for the extra info @limakzi, but I'm still not seeing how we get to the github_repository evaluation without first visiting var.template. I tried making a few different test cases, but the references are always hooked up correctly, and even when there's an unexpected error we can't reach the panic. If this can't be reduced to a minimal reproducible example, the trace logs (TF_LOG_CORE=trace) may shed some light on the problem.

jbardin avatar Oct 28 '24 20:10 jbardin

@jbardin I can confirm, this dynamic block is the source of all the problems. I will prepare minimal working example with TF_LOG_CORE=trace enabled.

limakzi avatar Oct 30 '24 07:10 limakzi

closing to use #36184 as the primary issue which has a more detailed example

jbardin avatar Dec 16 '24 20:12 jbardin

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

github-actions[bot] avatar Jan 16 '25 02:01 github-actions[bot]