.terragrunt-source-manifest breaks terraform helm_release resource

[dmorgan81]

Similar to #829

The file .terragrunt-source-manifest breaks a terraform helm_release resource with the error YAML parse error on .../.terragrunt-source-manifest: error converting YAML to JSON: yaml: invalid leading UTF-8 octet. Makes sense because the manifest is a binary file.

Now because terragrunt excludes dotfiles during the copy adding .terragrunt-source-manifest to a .helmignore doesn't work; the .helmignore doesn't exist in the terragrunt cache dir.

Working around this requires manually copying the .helmignore file into the terragrunt cache after a failed apply.

[yorinasub17]

Just to clarify, are you using a local path and the helm chart is embedded into the terraform module folder?

[yorinasub17]

https://github.com/gruntwork-io/terragrunt/issues/916 would probably resolve this.

[dmorgan81]

We're setting TERRAGRUNT_SOURCE to a directory containing a terraform module with multiple helm charts.

[jeffreymlewis]

In case it's not obvious to others...

One work-around is to temporarily move your helm chart outside the module directory, then set chart in the helm_release to the absolute path. (Works if you're setting TERRAGRUNT_SOURCE for testing purposes.)

[richardpen]

another workaround we found ugly but works is by copying the .helmignore file during apply and add a dependency to the helm_release resource, eg:

resource "helm_release" "access_control" {
  depends_on = [
    local_file.ns_helmignore
  ]
xxxx
}

resource local_file ns_helmignore {
  content     = file("./helmignore")
  filename = "./chart/.helmignore"
  file_permission = "0644"
}

[zetas]

The local_file fix isn't really feasible when you have a complex helm chart that's made up of other helm charts, I'd have to create a new local_file for each sub chart... Is there any way to get around this? Specifying an absolute path is also not really usable as we deploy from our local laptops as well as various CICD platforms so it always needs to be relative.

[FriedCircuits]

I ran into the same issue. I moved the chart outside of terrafrom but now I can't commit it as part of the module. This is a bitnami chart that uses other charts. It has a bug so I needed to us it locally with the fix.

[kravvcu]

I've stumbled upon this myself with Helm chart and Terraform code co-located:

$ find . -maxdepth 1
./variables.tf
./mitmproxy  # chart directory
./resources.tf
./versions.tf
./outputs.tf

What worked for me as a workaround is using an before_hook to remove the chart dir before running apply. The reason it works is that the helm chart is actually pulled from a Helm registry by helm_release.

I've added a script at scripts/remove_chart_dir.sh:

#!/usr/bin/sh

# remove the chart dir so that helm_release can run successfully
# https://github.com/gruntwork-io/terragrunt/issues/943
rm -rf mitmproxy

And adjusted my terragrunt configuration:

terraform {
  source = <path-to-terraform-module>

  before_hook "remove_chart_dir" {
    commands = ["apply"]
    execute = ["./scripts/remove_chart_dir.sh"]
  }
}

[gmoshiko]

if it helps anyone, It can be solved using terragrunt generator block like this:

generate "helmignore" {
  path      = "charts/fluentd/.helmignore"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
.terragrunt-source-manifest*
  EOF
}

[ddvdozuki]

@kravvcu interesting fix but I can't see why you would have a helm directory locally if your charts are pulled from a helm registry. Most of us are using local helm directories because we have local helm charts that are committed next to the terraform.

[ddvdozuki]

@gmoshiko Ah! Very interesting! I've had to use a ton of the local file provisioners to get around it for my 8 or so helm charts and it's been very messy. At least using your technique I can move the local file creation out of the terraform and have terragrunt manage it directly which makes it a lot more semantic. Thanks for the suggestion.

[kravvcu]

@kravvcu interesting fix but I can't see why you would have a helm directory locally if your charts are pulled from a helm registry. Most of us are using local helm directories because we have local helm charts that are committed next to the terraform.

I also co-locate Terraform module and Helm chart in the same repository. Both the module and chart are published to GitLab. My memory is vague but the answer to your question, I believe, is that the workaround is needed for local development when the chart is copied alongside the Terraform module during Terragrunt's work.

[mehighlow]

For those who are struggling with the Error: YAML parse error on <chart-name>/templates/<subfolder>/.terragrunt-source-manifest: error converting YAML to JSON: yaml: invalid leading UTF-8 octet issue in 2023...

In my scenario, I have a Terragrunt script that calls the Terraform service, which in turn calls several modules. One of these modules calls two Helm charts. While one of the charts was working fine, the other one was throwing the aforementioned error.

To summarize, the failing chart had the next structure with subfolders:

chart-name/
├── Chart.yaml
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── azure
│   │   ├── resource1.yaml
│   │   ├── resource2yaml
│   │   └── resource3.yaml
│   ├── certificate-sync
│   │   ├── object1.yaml
│   │   └── object2.yaml
│   ├── deployment.yaml
│   ├── sa.yaml
│   └── hpa.yaml
└── values.yaml

However, after moving all the files under the templates directory and deleting the subfolders, everything started to work. I hope this will save someone a ton of time.

[ddvdozuki]

@mehighlow Interesting solution. Unfortunately it's not always feasible to change the format of a helm chart, especially if it's maintained elsewhere. This is my solution in a common.hcl file:

locals {
  helmignore = <<EOF
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
.terragrunt-source-manifest
.terragrunt-source-manifest/
  EOF
}

generate "grafana_helmignore" {
  path      = "charts/grafana/.helmignore"
  if_exists = "overwrite_terragrunt"
  contents  = local.helmignore
}
generate "redis_helmignore" {
  path      = "charts/redis/.helmignore"
  if_exists = "overwrite_terragrunt"
  contents  = local.helmignore
}
...

And in the appropriate terragrunt.hcl file:

include "common" {
  path = find_in_parent_folders("common.hcl")
}

[denis256]

Hi, I was wondering if can be used include_in_copy

terraform {
  include_in_copy = [
    "**/.helmignore",
    ".helmignore",
  ]
}

https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/

[aaj-synth]

^ this works as well! You just need to add .helmignore files in your helm chart folders.