terraform-provider-google
terraform-provider-google copied to clipboard
hashicorp
Cloud Run env variables reordering causing resource to update in-place
Our cloud run resource needs to be updated every run as the environment variables are reordering. This seems to be completely on the Terraform side as our actual Cloud Run instance does not seem to be affected.
Terraform Version
v0.13.2
Affected Resource(s)
- google_cloud_run_service
Terraform Configuration Files
containers {
image = "XXXXX"
env {
name = "ENV_1"
value = data.google_secret_manager_secret_version.secrets["XXXX"].secret_data
}
env {
name = "ENV_2"
value = data.google_secret_manager_secret_version.secrets["XXXX"].secret_data
}
}
Expected Behavior
For the environment variables to be in the same order, leading to no change in the resource.
Actual Behavior
~ env {
~ name = "ENV_1" -> "ENV_2"
~ value = "XXX" -> "XXXX"
}
~ env {
~ name = "ENV_2" -> "ENV_1"
~ value = "XXXX" -> "XXX"
}
Steps to Reproduce
-
terraform applyand/orterraform plan
Important Factoids
- environment variables are coming from google_secret_manager_secret_version data object
References
b/272365080
I don't see that happening for the sample config below
name = "cloudrun-srv"
location = "us-central1"
template {
spec {
containers {
image = "gcr.io/cloudrun/hello"
env {
name = "SOURCE"
value = "remote"
}
env {
name = "TARGET"
value = "home"
}
}
}
}
traffic {
percent = 100
latest_revision = true
}
autogenerate_revision_name = true
}
Can you please attach the plan output and debug log for your apply
Hey! Thanks for your help, hopefully this is useful. I believe the issue is caused by the google_secret_manager_secret_version data object -- if the environment variables are hard-coded in plain text then we do not see the same behavior
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# google_cloud_run_service.default will be updated in-place
~ resource "google_cloud_run_service" "default" {
id = "REDACTED"
location = "us-central1"
name = "terraform"
project = "REDACTED"
status = [
{
conditions = [
{
message = ""
reason = ""
status = "True"
type = "Ready"
},
{
message = ""
reason = ""
status = "True"
type = "ConfigurationsReady"
},
{
message = ""
reason = ""
status = "True"
type = "RoutesReady"
},
]
latest_created_revision_name = "terraform-bdff6"
latest_ready_revision_name = "terraform-bdff6"
observed_generation = 4
url = "REDACTED"
},
]
metadata {
annotations = {
"serving.knative.dev/creator" = "REDACTED"
"serving.knative.dev/lastModifier" = "REDACTED"
}
generation = 4
labels = {
"cloud.googleapis.com/location" = "us-central1"
}
namespace = "REDACTED"
resource_version = "AAWw4zi+fg0"
self_link = "REDACTED"
uid = "77fe171e-b695-427c-9eb0-833643c1523e"
}
~ template {
metadata {
annotations = {
"autoscaling.knative.dev/maxScale" = "1000"
"run.googleapis.com/vpc-access-connector" = "REDACTED"
}
generation = 0
labels = {}
}
~ spec {
container_concurrency = 20
service_account_name = "REDACTED"
~ containers {
args = []
command = []
image = "REDACTED"
~ env {
~ name = "ENV" -> "EXISTING_ENV"
~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
}
~ env {
~ name = "ENV" -> "EXISTING_ENV"
~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
}
~ env {
~ name = "ENV" -> "EXISTING_ENV"
~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
}
~ env {
~ name = "ENV" -> "EXISTING_ENV"
~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
}
~ env {
~ name = "ENV" -> "EXISTING_ENV"
~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
}
~ env {
~ name = "ENV" -> "EXISTING_ENV"
~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
}
resources {
limits = {
"cpu" = "1000m"
"memory" = "256Mi"
}
requests = {}
}
}
}
}
traffic {
latest_revision = true
percent = 100
}
}
Debug Output
2020/10/08 17:12:42 [DEBUG] command: asking for input: "\nDo you want to perform these actions in workspace \"REDACTED"?"
I assume that it is using the data source, isn't it? (https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/secret_manager_secret_version)
If so, I could not verify this to be an issue, using:
Terraform v1.0.8
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v3.58.0
+ provider registry.terraform.io/hashicorp/google-beta v3.86.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
@skreinberg97 is it still an issue for you? If so, can you provide more information, please?
Hi @blueyed, I am experiencing the same with:
Terraform v1.0.9
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v3.89.0
+ provider registry.terraform.io/hashicorp/google-beta v3.89.0
+ provider registry.terraform.io/hashicorp/vault v2.24.1
In my case, it is google_secret_manager_secret_version resource, not data source.
Terraform will perform the following actions:
# google_cloud_run_service.service will be updated in-place
~ resource "google_cloud_run_service" "service" {
id = "locations/region/namespaces/project/services/service"
name = "service"
# (4 unchanged attributes hidden)
~ template {
~ spec {
# (3 unchanged attributes hidden)
~ containers {
# (2 unchanged attributes hidden)
- env {
- name = "ENV_VAR" -> null
- value_from {
- secret_key_ref {
- key = "1" -> null
- name = "secret_manager_secret_name" -> null
}
}
}
+ env {
+ name = "ENV_VAR"
+ value_from {
+ secret_key_ref {
+ key = "1"
+ name = "secret_manager_secret_name"
}
}
}
2021-10-19T12:59:44.812+0200 [INFO] provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2021-10-19T12:59:44.812+0200
2021-10-19T12:59:44.812+0200 [INFO] provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-10-19T12:59:44.812+0200
2021-10-19T12:59:44.816+0200 [WARN] Provider "registry.terraform.io/hashicorp/google-beta" produced an invalid plan for google_secret_manager_secret_version.vault_secrets["ENV_VAR"], but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .enabled: planned value cty.True for a non-computed attribute
2021-10-19T12:59:44.818+0200 [INFO] provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
2021-10-19T12:59:46.016+0200 [INFO] provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:46 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-10-19T12:59:46.016+0200
2021-10-19T12:59:46.046+0200 [WARN] Provider "registry.terraform.io/hashicorp/google-beta" produced an unexpected new value for google_cloud_run_service.service during refresh.
- .template[0].spec[0].containers[0].env: planned set element cty.ObjectVal(map[string]cty.Value{"name":cty.StringVal("ENV_VAR"), "value":cty.NullVal(cty.String), "value_from":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"secret_key_ref":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"key":cty.StringVal("2"), "name":cty.StringVal("secret_manager_secret_name")})})})})}) does not correlate with any element in actual
2021-10-19T12:59:46.128+0200 [WARN] Provider "registry.terraform.io/hashicorp/google-beta" produced an invalid plan for google_cloud_run_service.service, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .traffic: block count in plan (1) disagrees with count in config (0)
- .template[0].spec[0].containers[0].args: planned value cty.ListValEmpty(cty.String) for a non-computed attribute
- .template[0].spec[0].containers[0].working_dir: planned value cty.StringVal("") for a non-computed attribute
- .template[0].spec[0].containers[0].ports[0].protocol: planned value cty.StringVal("") for a non-computed attribute
2021-10-19T12:59:46.195+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-10-19T12:59:46.198+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/3.89.0/darwin_amd64/terraform-provider-google-beta_v3.89.0_x5 pid=43591
2021-10-19T12:59:46.198+0200 [DEBUG] provider: plugin exited
2021-10-19T12:59:46.199+0200 [INFO] backend/local: plan operation completed
Encountering the same thing. We also have some Secret versions being managed directly, not via a data source.
For those I have generated an UUID as the name prefixed with secret-, since that's the way the UI / cloud side of Cloud Run seems to handle things. Not sure if it has an effect on this.
In any case, even the non-secret environment variables are shuffled around.
I wonder if we could just sort the environment variables a certain way to keep Terraform state happy and have the order be the same during all runs.
It looks it is fixed by using https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_run_v2_service
This also reproduces with google_cloud_run_v2_service if you have a value_source of secret_key_ref.
A temporary workaround is to delete all env variables in the UI and then terraform apply again. But would really be nice to see this fixed.
This my terraform version. I have the same problem. Is there any solution for help?
I found this env change happend after import cloud_run_v2_service resourve,looks like import changed the env sort in terraform state ***.tfstate .
~ env {
~ name = "ENV_1" -> "ENV_2"
~ value = "XXX" -> "XXXX"
}
~ env {
~ name = "ENV_2" -> "ENV_1"
~ value = "XXXX" -> "XXX"
}
facing same problem with google_cloud_run_v2_service, was anyone able to figure out a solution?
Throwing my voice into the void here, same issue, spot checking ENV VAR changes gets really hard when you have more than a handful of them.
Same issue here on the latest version of the provider. We have about 20 services deployed and the output of terraform apply is constantly polluted by changes in env variables order (each env var references a secret from secret manager).
It is worth noting that I'm using google_secret_manager_secrets to load the secrets for a service and use a dynamic block in order to add them to the service template. From all that I've read so far in terms of documentation, I got the impression that order should be preserved at the terraform level during iteration, so I'm unsure where this re-ordering is caused by:
- the implementation of the google terraform provider using a data structure that doesn't preserve order
- the API response used by
google_secret_manager_secretsnot returning secrets in a consistent order
